Evaluation device, evaluation method, and computer program

Abstract

To provide an evaluation device, evaluation method, and computer program.SOLUTION: An evaluation device disclosed herein comprises a processing unit configured to acquire a first measurement value from a first measurement device for measuring a target space, acquire a second measurement value indicative of the behavior of the first measurement device, and evaluate the influence of the behavior on acquisition of the first measurement value on the basis of the acquired second measurement value.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an evaluation device, an evaluation method, and a computer program. [Background technology]

[0002] Patent Document 1 discloses a configuration in which a 3D map displaying a measurement target is generated using 3D point cloud data, and a measurement plan including information on the measurement target area to be acquired next is generated based on the evaluation results of the quality of the 3D map. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2020 / 121406 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, processed 3D measurement data is used to evaluate the quality of the obtained 3D map. In this case, unless the 3D point cloud data is processed, it is impossible to confirm whether appropriate data has been acquired to construct the 3D map.

[0005] An object of the present disclosure is to provide an evaluation device, an evaluation method, and a computer program that evaluate whether appropriate data for constructing a three-dimensional map is being acquired during data acquisition, which is the timing before the three-dimensional map is generated. [Means for solving the problem]

[0006] An evaluation device according to a first aspect of the present disclosure includes a processing unit that acquires a first measurement value from a first measuring instrument that measures a target space, acquires a second measurement value that indicates the behavior of the first measuring instrument, and evaluates the effect of the behavior on the acquisition of the first measurement value based on the acquired second measurement value.

[0007] An evaluation device according to a second aspect of the present disclosure is the evaluation device according to the first aspect, wherein the processing unit acquires the second measurement value from a second measuring instrument that measures behavior of the first measuring instrument.

[0008] An evaluation device according to a third aspect of the present disclosure is the evaluation device according to the first or second aspect, wherein the processing unit compares the second measurement value with a set value and outputs an alarm according to a comparison result.

[0009] An evaluation device according to a fourth aspect of the present disclosure is the evaluation device according to the third aspect, wherein the first measuring instrument includes multiple types of sensors, and the processing unit outputs an alarm according to a comparison result between one of the set values ​​set for each of the sensors and the second measurement value.

[0010] An evaluation device according to a fifth aspect of the present disclosure is the evaluation device according to the third or fourth aspect, wherein the alarm is constituted by at least one of sound, vibration, light, text, and symbols.

[0011] An evaluation device according to a sixth aspect of the present disclosure is the evaluation device according to any one of the first to fifth aspects, further comprising a display unit that displays information based on the second measurement value.

[0012] An evaluation device according to a seventh aspect of the present disclosure is an evaluation device according to any one of the first to sixth aspects, further comprising a reception unit that receives input of the setting value, and the processing unit evaluates the effect of the behavior on the acquisition of the first measurement value by comparing the second measurement value with the setting value received from the reception unit.

[0013] An evaluation device according to an eighth aspect of the present disclosure is an evaluation device according to any one of the first to seventh aspects, wherein the processing unit compares the second measurement value with a set value and stops measurement by the first measuring instrument depending on the comparison result.

[0014] An evaluation device according to a ninth aspect of the present disclosure is the evaluation device according to the eighth aspect, wherein the processing unit stops measurement by the first measuring device and then prompts the processing unit to measure again.

[0015] An evaluation device according to a tenth aspect of the present disclosure is the evaluation device according to the eighth aspect, wherein the processing unit stops measurement by the first measuring instrument and then guides the user to resume measurement from a point a set distance back.

[0016] An evaluation device according to an eleventh aspect of the present disclosure is the evaluation device according to any one of the first to tenth aspects, wherein the second measurement value includes at least one of the velocity, acceleration, and angular velocity of the first measuring instrument.

[0017] An evaluation method according to a twelfth aspect of the present disclosure includes a computer-implemented process of acquiring a first measurement value from a first measuring instrument that measures a target space, acquiring a second measurement value that indicates the behavior of the first measuring instrument, and evaluating the effect of the behavior on the acquisition of the first measurement value based on the acquired second measurement value.

[0018] A computer program according to a thirteenth aspect of the present disclosure causes a computer to perform a process of acquiring a first measurement value from a first measuring instrument that measures a target space, acquiring a second measurement value that indicates the behavior of the first measuring instrument, and evaluating the effect of the behavior on the acquisition of the first measurement value based on the acquired second measurement value. [Effects of the Invention]

[0019] According to the present disclosure, it is possible to evaluate whether appropriate data for constructing a three-dimensional map has been acquired during data acquisition, which is the timing before a three-dimensional map is generated. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a block diagram illustrating an example of a configuration of an information processing system according to an embodiment. [Figure 2] FIG. 2 is a schematic perspective view of a measuring device. [Figure 3] FIG. 2 is a schematic diagram illustrating an example of an interface screen provided by a terminal device. [Figure 4] FIG. 10 is a schematic diagram showing an example of a screen when an alarm is output. [Figure 5] 5 is a flowchart showing a procedure of processing executed by a terminal device in the first embodiment. [Figure 6] 10 is a flowchart showing a procedure of processing executed by a terminal device in the second embodiment. [Figure 7] FIG. 10 is a schematic diagram showing a guide screen when restarting measurement. [Figure 8] 11 is a flowchart showing a procedure of processing executed by a terminal device in the third embodiment. [Figure 9] FIG. 13 is a schematic diagram showing a remeasurement guide screen in the third embodiment. [Figure 10] 1 is a flowchart showing a procedure for creating a three-dimensional space model. [Figure 11] FIG. 1 is a schematic diagram illustrating an example of the configuration of a learning model. DETAILED DESCRIPTION OF THE INVENTION

[0021] An air conditioning-related device and an air conditioning system according to an embodiment of the present disclosure will be described below with reference to the drawings. When visiting a customer's property and conducting sales activities related to air conditioning, performing airflow analysis under the actual property conditions allows for the identification of air environment problems such as temperature unevenness and stagnation that may actually be occurring in the customer's property, before proposing equipment. However, when actually creating a fluid calculation model to perform airflow analysis, it takes a lot of time to measure and model the room shape and fixtures, and set conditions such as the heat source. To solve the above problems, we propose a method for automatically creating models for fluid calculations. In one example system, a user walks around a space using a handheld depth camera, a camera, a thermograph, and a smartphone-integrated module, and captures depth images, images, and thermography images of the space. The data stored on the smartphone is transferred to an information processing device 1, and 3D point cloud data is created using 3D reconstruction software. The floor, walls, and ceiling are detected from the 3D point cloud data, and a fluid calculation model, which is a spatial model of surfaces, is created. The captured images are then subjected to image recognition, previously trained through machine learning, to identify the type and position of objects (air conditioners, fixtures, etc.), and are reflected in the 3D spatial model. Instead of a person taking handheld photographs, a cleaning robot or drone can also be used.

[0022] (Embodiment 1) FIG. 1 is a block diagram showing an example of the configuration of an information processing system according to an embodiment, and FIG. 2 is a schematic perspective view of a measurement device. The information processing system according to this embodiment includes an information processing device 1, a measurement device 2, and a terminal device 3. The information processing system measures a space having at least a surface, and semi-automatically creates a three-dimensional space model for predicting the environment of the space based on the obtained data. Hereinafter, the space to be modeled will also be referred to as the target space.

[0023] The measurement device 2 includes an RGB sensor 21, a depth sensor 22, an IR projector 23, an infrared camera 24, a gripping unit 25, and a terminal holding unit 26. In this embodiment, the RGB sensor 21, the depth sensor 22, the IR projector 23, and the infrared camera 24 included in the measurement device 2 are examples of measuring instruments that measure the target space. Hereinafter, the measuring instrument that measures the target space will also be referred to as a first measuring instrument. Furthermore, the output of the first measuring instrument will also be referred to as a first measurement value.

[0024] The RGB sensor 21 is a camera that has an imaging element such as a CMOS sensor or a CCD sensor and captures color images of the target space. The RGB sensor 21 can capture moving images. The RGB sensor 21 outputs time-series image data (RGB data) obtained by capturing images of the target space. The time-series image data that constitutes the moving image includes time data indicating the date and time of capture.

[0025] The depth sensor 22 measures the distance (depth) from the measurement device 2 to the walls of the target space and various objects contained in the target space. The depth sensor 22 outputs depth data indicating the depth obtained by measuring the distance. The IR projector 23 is an element for projecting infrared rays onto the target space and objects in order to obtain more accurate depth data.

[0026] The infrared camera 24 has an infrared imaging element and is a camera that captures an image of the target space using infrared rays. The infrared camera 24 captures an image of the target space and outputs thermographic image data obtained.

[0027] The grip part 25 is a rod-shaped member that is gripped by a user (hereinafter also simply referred to as a user) of the measurement device 2. The terminal holding part 26 is a member for holding the terminal device 3.

[0028] The measuring device 2 configured in this manner outputs image data, depth data, and thermographic image data to the terminal device 3. The measuring device 2 may store the image data, depth data, and thermographic image data in an external portable storage device. The measuring device 2 may also transmit the image data, depth data, and thermographic image data directly to the information processing device 1.

[0029] The terminal device 3 is a portable information processing terminal (computer) such as a smartphone or tablet terminal, and includes a processing unit 31, a storage unit 32, an acquisition unit 33, a display unit , an operation unit 35, and an alarm unit .

[0030] The processing unit 31 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The CPU of the processing unit 31 loads various programs stored in advance in the ROM or the storage unit 32 into the RAM and executes them, thereby controlling the operation of the various hardware components described above and causing the entire device to function as the evaluation device of the present disclosure.

[0031] The processing unit 31 is not limited to the above configuration, and may be any processing circuit including, for example, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a quantum processor, a volatile or non-volatile memory, etc. The processing unit 31 may also have functions such as a clock that outputs date and time information, a timer that measures the elapsed time from when an instruction to start measurement is given until when an instruction to end measurement is given, and a counter that counts numbers.

[0032] The storage unit 32 includes a storage device such as a flash memory, a hard disk, etc. The storage unit 32 stores various computer programs executed by the processing unit 31, data necessary for executing the computer programs, etc. The computer programs stored in the storage unit 32 include an evaluation program PG for evaluating the effect of the behavior of the first measuring instrument on the acquisition of the first measurement value, etc.

[0033] The programs stored in the storage unit 32 are provided, for example, by a non-transitory storage medium RM on which the programs are readably recorded. The storage medium RM is a portable memory such as a CD-ROM, a USB memory, an SD (Secure Digital) card, a micro SD card, or a CompactFlash (registered trademark). The processing unit 31 reads the various programs from the storage medium RM using a reading device (not shown) and installs the read programs in the storage unit 32. The programs stored in the storage unit 32 may also be provided via communication. In this case, the processing unit 31 may acquire the various programs via communication and install the acquired programs in the storage unit 32.

[0034] The acquisition unit 33 includes an interface for acquiring various data from an external device. The acquisition unit 33 may include a communication interface for transmitting and receiving data to and from the external device via a wired or wireless connection, or may include a readout circuit for reading data from a portable storage device. The acquisition unit 33 can acquire image data, depth data, and thermographic image data output from the measurement device 2.

[0035] The display unit 34 includes a liquid crystal display device or the like, and displays information to be notified to the user. The operation unit 35 includes an input interface such as a touch panel, and accepts operations by the user.

[0036] The alarm unit 36 ​​includes an alarm device that outputs an alarm. The alarm device included in the alarm unit 36 ​​may be a buzzer that outputs an alarm by sound, a vibrator that outputs an alarm by vibration, or a light-emitting element that outputs an alarm by light. Alternatively, the alarm unit 36 ​​may be configured to output an alarm by displaying characters or symbols on the display unit 34. The alarm unit 36 ​​outputs an alarm in response to an instruction from the processing unit 31.

[0037] The terminal device 3 further includes a direction sensor S1, a speed sensor S2, an acceleration sensor S3, and an angular velocity sensor S4. The direction sensor S1 is a direction meter such as an electronic compass, and outputs direction data indicating the direction of the measurement device 2. The speed sensor S2 is a sensor that utilizes, for example, the Doppler effect or a spatial filter, and measures the speed of the measurement device 2 in three mutually orthogonal axis directions and outputs measurement value data. The acceleration sensor S3 is a sensor of a piezoelectric, piezo-resistive, or capacitance type, and measures the acceleration of the measurement device 2 in three mutually orthogonal axis directions and outputs measurement value data. The angular velocity sensor S4 is a sensor of a vibration type, capacitance type, or other type, and measures the angular velocity of the measurement device 2 in three mutually orthogonal axis directions and outputs measurement value data.

[0038] In this embodiment, the velocity sensor S2, the acceleration sensor S3, and the angular velocity sensor S4 are examples of measuring instruments that measure the behavior of the first measuring instrument. Hereinafter, the measuring instrument that measures the behavior of the first measuring instrument will also be referred to as the second measuring instrument. In addition, the output of the second measuring instrument will also be referred to as the second measurement value.

[0039] An angular velocity sensor S4 is preferably used as the second measuring instrument. Alternatively, a velocity sensor S2 or an acceleration sensor S3 may be used as the second measuring instrument. Furthermore, three sensors, namely, the velocity sensor S2, the acceleration sensor S3, and the angular velocity sensor S4, may be used as the second measuring instrument, or any two of these sensors may be used. Therefore, only the angle sensor S4 may be installed in the terminal device 3. Alternatively, only the velocity sensor S2 or only the acceleration sensor S3 may be installed in the terminal device 3. Furthermore, three sensors, namely, the velocity sensor S2, the acceleration sensor S3, and the angular velocity sensor S4, or any two of these sensors may be installed in the terminal device 3. These sensors may be external to the terminal device 3 or may be installed in the measurement device 2.

[0040] The terminal device 3 displays the image captured by the measuring device 2 on the display device based on the acquired image data, and evaluates the impact of the behavior of the first measuring instrument on the acquisition of the first measurement value based on the acquired second measurement value. If the terminal device 3 evaluates that the impact on the acquisition of the first measurement value is large, it outputs an alarm from the alarm unit 36 ​​and prompts the user to measure again. If the terminal device 3 evaluates that the impact on the acquisition of the first measurement value is small, it stores the acquired image data, depth data, and thermographic image, as well as the orientation data acquired by the orientation sensor S1, and transmits them to the information processing device 1 at an appropriate time. The terminal device 3 may be configured to store the acquired image data, depth data, and thermographic image, as well as the orientation data acquired by the orientation sensor S1 in a portable storage device.

[0041] With the measuring device 2 configured in this way, a user can capture images of and measure distances to the walls, floor, ceiling, and objects in the target space from multiple positions and directions by holding the measuring device 2, moving it around within the target space, and pointing the measuring device 2 in various directions. The measuring device 2 provides the information processing device 1 with target space data, thermographic image data, and orientation data obtained by measuring the target space from multiple positions.

[0042] The information processing device 1 includes a processing unit 11, a storage unit 12, an acquisition unit 13, a display unit 14, and an operation unit 15.

[0043] The processing unit 11 is a processor including a CPU, a ROM, a RAM, etc. The processing unit 11 may include one or more arithmetic circuits, such as a GPU, a TPU (Tensor Processing Unit), or an AI chip (semiconductor for AI), that are specialized for image processing related to object detection and image recognition. The processing unit 11 creates a three-dimensional space model of the target space by reading and executing a computer program stored in the storage unit 12. Each functional unit involved in creating the three-dimensional space model may be realized by software, or some or all of them may be realized by hardware.

[0044] The storage unit 12 is, for example, a storage such as a hard disk, an EEPROM (Electrically Erasable Programmable ROM), or a flash memory. The storage unit 12 stores various programs executed by the processing unit 11 and various data required for the processing of the processing unit 11. In this embodiment, the storage unit 12 stores at least the computer programs executed by the processing unit 11.

[0045] The computer program may be written to the storage unit 12 during the manufacturing stage of the information processing device 1, or may be distributed via a network from another information processing device 1, etc. The information processing device 1 acquires the computer program via communication and writes it to the storage unit 12. The computer program may be readably recorded on a recording medium such as a semiconductor memory such as a flash memory, an optical disk, a magneto-optical disk, or a magnetic disk. The information processing device 1 reads the computer program from the recording medium using a reading device (not shown) and stores the read computer program in the storage unit 12.

[0046] The acquisition unit 13 is a communication circuit that transmits and receives data to and from an external device via a wired or wireless connection, or a reading circuit that reads data from a portable storage device. The information processing device 1 acquires image data, depth data, thermographic image data, orientation data, etc. from the measurement device 2 via the acquisition unit 13.

[0047] The information processing device 1 may acquire duct system diagram data representing a duct system related to the inlets or outlets of the ventilation device in the target space from an external device via the acquisition unit 13. The duct system diagram data includes direction information representing the direction of the duct system and external shape information representing the external shape of the duct system. The external shape information includes information representing the position of the inlets or outlets of the air conditioner or ventilation device.

[0048] The information processing device 1 may acquire piping diagram data representing a piping diagram of a target space from an external device via the acquisition unit 13. The piping diagram data includes direction information representing the direction of the piping system and external shape information representing the external shape of the piping system. The piping diagram data includes information related to the positions of indoor units, thermal boundary conditions of air conditioners, etc.

[0049] The information processing device 1 may acquire, via the acquisition unit 13, time-series data on the outside air temperature in the target space and time-series data on the amount of sunlight in the target space from an external device.

[0050] The display unit 14 is a display device such as a liquid crystal display panel, an organic EL display panel, etc. The processing unit 11 displays an image related to the three-dimensional space model on the display unit 14.

[0051] The operation unit 15 is an input device such as a mouse, keyboard, or touch panel that accepts operations by the user of the information processing device 1. The processing unit 11 accepts, via the operation unit 15, editing of the three-dimensional space model by the user, etc.

[0052] The information processing device 1 may be a server device connected to a network. The information processing device 1 may be configured with a client device and a server device. The information processing device 1 may be configured to perform distributed processing using multiple computers, may be realized by multiple virtual machines provided in a single server, or may be realized using a cloud server. The information processing device 1 may also be a terminal device 3. In other words, the terminal device 3 may be configured to function as the information processing device 1.

[0053] 3 is a schematic diagram showing an example of an interface screen 340 provided by the terminal device 3. When measuring a target space using the measurement device 2, the processing unit 31 of the terminal device 3 displays the interface screen 340 as shown in FIG. 3 on the display unit 34. The interface screen 340 includes, for example, an image display field 341, a recording start button 342, a display switch button 344, a recording list display button 346, a recording time display field 348, and an indicator 350.

[0054] The image display field 341 displays an image obtained from the RGB sensor 21 of the measurement device 2 (a color image of three RGB colors) or an image obtained from the depth sensor 22 (a depth image).

[0055] The recording start button 342 is a software button that accepts a recording start instruction from the user. When the recording start button 342 is pressed, the processing unit 31 stores the first measurement values ​​acquired by the acquisition unit 33 (image data output from the RGB sensor 21, depth data output from the depth sensor 22, and thermographic image data output from the infrared camera 24) in the storage unit 32.

[0056] The display switching button 344 is a software button that accepts a display switching instruction from the user. Each time the display switching button 344 is pressed, the processing unit 31 switches the image display in the image display field 341 between a color image of three colors (RGB) and a depth image.

[0057] The recording list display button 346 is a software button for displaying a list of recorded files. When the recording list display button 346 is pressed, the processing unit 31 displays a list of recorded files on the display unit 34 instead of displaying an image. When a recorded file is selected from the list, the processing unit 31 reads out the selected file and displays the image (a color image in three colors, RGB, or a depth image) in the image display field 341.

[0058] The recording time display field 348 displays the elapsed time (recording time) since the recording start button 342 was pressed. The recording time display field 348 may display values ​​such as fps (frames per second). The fps value may be a fixed value or may change in real time.

[0059] Indicator 350 displays the second measurement value. Fig. 3 shows an example in which the magnitude of acceleration measured by acceleration sensor S3 is displayed by indicator 350. In this example, the length of bar 351 displayed on indicator 350 represents the magnitude of acceleration. The magnitude of acceleration (numerical value) itself may also be displayed on indicator 350.

[0060] Alternatively, the values ​​of acceleration in the three axial directions (three values) measured by the acceleration sensor S3 may be displayed by the indicator 350. Furthermore, the magnitude of the velocity measured by the velocity sensor S2 or the velocity value in each axial direction may be displayed by the indicator 350, or the magnitude of the angular velocity measured by the angular velocity sensor S4 or the angular velocity value in each axial direction may be displayed by the indicator 350.

[0061] The processing unit 31 of the terminal device 3 constantly monitors the second measurement value and determines whether the second measurement value exceeds the set value. If the second measurement value exceeds the set value, the processing unit 31 outputs an alarm. FIG. 4 is a schematic diagram showing an example of a screen when an alarm is output. FIG. 4 shows an interface screen 340 after recording has started. In the example screen of FIG. 4, a stop recording button 343 is displayed instead of the start recording button 342, and an end confirmation button 347 is displayed instead of the recording list display button 346.

[0062] The recording stop button 343 is a software button that accepts a recording stop instruction from the user. When the recording stop button 343 is pressed, the processing unit 31 temporarily stops recording the first measurement values ​​acquired by the acquisition unit 33 (image data output from the RGB sensor 21, depth data output from the depth sensor 22, and thermographic image data output from the infrared camera 24).

[0063] The end confirmation button 347 is a software button that accepts a user instruction to end recording. When the end confirmation button 347 is pressed, the processing unit 31 displays a screen on the display unit 34 that asks the user whether or not to end recording, and ends recording when the user confirms that recording should be ended. When recording has ended, the processing unit 31 stores the image data output from the RGB sensor 21, the depth data output from the depth sensor 22, and the thermographic image data output from the infrared camera 24 as files in the storage unit 32.

[0064] As described above, the processing unit 31 of the terminal device 3 constantly monitors the second measurement value and outputs an alarm if the second measurement value exceeds the set value. The example screen of FIG. 4 shows an example in which the second measurement value exceeds the set value and a message is displayed in the message field 352 indicating that the measurement may fail and that the measuring device 2 should be moved slowly. While the example of FIG. 4 shows a message consisting of only text, a message including symbols may also be displayed. Furthermore, the processing unit 31 may change the color of the bar 351 of the indicator 350 to a more noticeable color (e.g., from green to yellow or red) to notify the user that the second measurement value has exceeded the set value. Furthermore, the processing unit 31 may output an alarm from the alarm unit 36.

[0065] 5 is a flowchart showing the procedure of processing executed by the terminal device 3 in the first embodiment. The processing unit 31 of the terminal device 3 acquires first measurement values ​​through the acquisition unit 33 (step S101). That is, the processing unit 31 acquires image data output from the RGB sensor 21, depth data output from the depth sensor 22, and thermography image data output from the infrared camera 24. The processing unit 31 may acquire the first measurement values ​​when an interface screen 340 is displayed on the display unit 34, or when an acquisition instruction is given by the user. The processing unit 31 displays the interface screen 340 on the display unit 34, and displays an image based on the acquired first measurement values ​​in the image display field 341 (step S102).

[0066] The processing unit 31 acquires a second measurement value (step S103). That is, the processing unit 31 may acquire at least one of the speed data output from the speed sensor S2, the acceleration data output from the acceleration sensor S3, and the angular velocity data output from the angular velocity sensor S4. In this flowchart, the second measurement value is acquired after the first measurement value is acquired, but these steps may be performed in reverse order or simultaneously in parallel.

[0067] The processing unit 31 compares the acquired second measurement value with a set value (step S104) and outputs an alarm according to the comparison result (step S105). In this embodiment, the set value for the second measurement value is stored in the storage unit 32. Alternatively, the processing unit 31 may accept a set value through the operation unit 35 and store the accepted set value in the storage unit 32. Alternatively, the processing unit 31 may accept a set value calculated based on evaluation results of past measurements via the acquisition unit 13. The set value calculated based on evaluation results of past measurements represents, for example, a value calculated by storing the second measurement values ​​in chronological order in the storage unit 32 during photography, and calculating the density of the 3D point cloud and model validity information obtained during creation of the 3D space model, and the second measurement values ​​at each time, so that the second measurement values ​​do not affect the 3D space model. The set value may be set for each of the velocity, acceleration, and angular velocity. In step S105, the processing unit 31 outputs an alarm if it determines that the second measurement value is greater than the set value. 4, or may output an alarm by activating the alarm unit 36. The set value may differ for each item due to factors such as different frame rates for each of the acquired items of the first measurement values ​​(image data output from the RGB sensor 21, depth data output from the depth sensor 22, and thermographic image data output from the infrared camera 24). In this case, a set value may be set for each acquired item, and the magnitudes of the set values ​​may be compared to output an alarm based on the condition with the lowest value.

[0068] The processing unit 31 evaluates the influence of the behavior of the first measuring device on the first measurement value based on the second measurement value (step S106). If the processing unit 31 determines that the second measurement value is equal to or less than the set value as a result of comparing the second measurement value with the set value in step S104, it evaluates that the influence on the first measurement value is small, and if the processing unit 31 determines that the second measurement value is greater than the set value, it evaluates that the influence on the first measurement value is large.

[0069] If it is determined that the effect on the first measurement value is small (S107: YES), the processing unit 31 determines whether or not the measurement has ended (step S108). If the end confirmation button 347 on the interface screen 340 is pressed and the end is confirmed by the user, the processing unit 31 determines that the measurement has ended. If it is determined that the measurement has not ended (S108: NO), the processing unit 31 returns the process to step S101 and continues the measurement.

[0070] If it is determined that the measurement has ended (S108: YES), the processing unit 31 transmits the first measurement value acquired in step S101 (image data output from the RGB sensor 21, depth data output from the depth sensor 22, and thermographic image data output from the infrared camera 24) and the orientation data obtained from the orientation sensor S1 to the information processing device 1 (step S109).

[0071] If it is evaluated in step S107 that the impact on the first measurement value is large (S107: NO), the processing unit 31 terminates the processing according to this flowchart without transmitting the first measurement value acquired in step S101 or the orientation data obtained from the orientation sensor S1 to the information processing device 1.

[0072] As described above, if it is evaluated that the impact on the acquisition of the first measurement values ​​is small, the first measurement values ​​(image data output from the RGB sensor 21, depth data output from the depth sensor 22, and thermography image data output from the infrared camera 24) and the orientation data obtained from the orientation sensor S1 are transmitted to the information processing device 1. Based on the data acquired from the terminal device 3, the information processing device 1 semi-automatically creates a three-dimensional space model for predicting the environment of the target space. An existing method is used to create the three-dimensional space model.

[0073] As described above, in embodiment 1, the impact of the behavior of the first measuring instrument on the acquisition of the first measurement value is evaluated based on the second measurement value indicating the behavior of the first measuring instrument, so that the user can be notified that there is a measurement defect without creating a three-dimensional space model in the information processing device 1.

[0074] (Embodiment 2) In the second embodiment, a configuration will be described in which, when the behavior of the first measuring device has a large effect on the acquisition of the first measurement value, the measurement is stopped and then a prompt to measure again is provided. The configuration of the information processing system and the configuration of the terminal device 3 are the same as those in the first embodiment, and therefore a description thereof will be omitted.

[0075] 6 is a flowchart showing the procedure of processing executed by terminal device 3 in embodiment 2. Processing unit 31 of terminal device 3 acquires the first measurement value and the second measurement value in the same procedure as in embodiment 1, and compares the second measurement value with a set value (steps S201 to S204). Processing unit 31 stops measurement by the first measuring instrument according to the comparison result between the second measurement value and the set value (step S205). Terminal device 3 stops measurement by measuring device 2 by issuing a measurement stop instruction to measuring device 2. If measurement is not to be stopped, processing unit 31 executes the processing from step S208 onwards.

[0076] After stopping the measurement by the first measuring instrument, the processing unit 31 determines whether to resume the measurement (step S206). Even after stopping the measurement by the first measuring instrument, the processing unit 31 continues to acquire the second measurement value as needed, and determines whether the second measurement value is less than the set value, thereby determining whether to resume the measurement. If the second measurement value is greater than the set value (S206: NO), the processing unit 31 waits without resuming the measurement.

[0077] If the second measurement value becomes less than the set value, it is determined that measurement should be resumed (S206: YES), and the processing unit 31 prompts the user to re-measure and resumes measurement (step S207). FIG. 7 is a schematic diagram showing a guidance screen when measurement is resumed. FIG. 7 shows an example in which a message indicating that measurement will be resumed is displayed superimposed on the interface screen 340. Alternatively, the processing unit 31 may prompt the user to resume measurement by outputting a sound. If measurement is resumed, the processing unit 31 executes the processes of steps S201 to S204 again.

[0078] Next, the processing unit 31 determines whether or not to end the measurement (step S208). When the end confirmation button 347 on the interface screen 340 is pressed and the end is confirmed by the user, the processing unit 31 determines that the measurement has ended. When it is determined that the measurement has not ended (S208: NO), the processing unit 31 returns the process to step S201 and continues the measurement.

[0079] If it is determined that the measurement has ended (S208: YES), the processing unit 31 transmits the first measurement value acquired in step S201 (image data output from the RGB sensor 21, depth data output from the depth sensor 22, and thermographic image data output from the infrared camera 24) and the orientation data obtained from the orientation sensor S1 to the information processing device 1 (step S209).

[0080] The information processing device 1 semi-automatically creates a three-dimensional space model for predicting the environment of the target space based on data acquired from the terminal device 3. An existing method is used to create the three-dimensional space model.

[0081] As described above, in the second embodiment, if the behavior of the first measuring device has a large effect on the acquisition of the first measurement value, the measurement can be temporarily stopped, and then a message can be sent to restart the measurement when the measurement is successful.

[0082] In the first embodiment, a warning is output in response to the comparison result between the second measurement value and the set value, and in the second embodiment, measurement is temporarily stopped in response to the comparison result between the second measurement value and the set value. Alternatively, two set values ​​(hereinafter referred to as a first set value and a second set value, and the first set value is < the second set value) may be prepared, and a warning may be output when the second measurement value > the first set value and the second measurement value < the second set value, and measurement may be temporarily stopped when the second measurement value > the second set value, and then measurement may be resumed when the measurement is successful (for example, when the second measurement value < the first set value).

[0083] (Embodiment 3) In the third embodiment, a configuration will be described in which, when the behavior of the first measuring device has a large effect on the acquisition of the first measurement value, the measurement is stopped and then re-measurement is prompted from a point a set distance back. The configuration of the information processing system and the configuration of the terminal device 3 are the same as those in the first embodiment, and therefore a description thereof will be omitted.

[0084] 8 is a flowchart showing the procedure of processing executed by terminal device 3 in embodiment 3. Processing unit 31 of terminal device 3 acquires the first measurement value and the second measurement value in the same procedure as in embodiment 1, and compares the second measurement value with a set value (steps S301 to S304). Processing unit 31 stops measurement by the first measuring instrument according to the comparison result between the second measurement value and the set value (step S305). Terminal device 3 stops measurement by measuring device 2 by issuing a measurement stop instruction to measuring device 2. If measurement is not to be stopped, processing unit 31 executes the processing from step S309 onwards.

[0085] After stopping the measurement by the first measuring device, the processing unit 31 prompts the user to remeasure from a point that is a set distance back (step S306). FIG. 9 is a schematic diagram showing a remeasurement prompt screen in the third embodiment. FIG. 9 shows an example in which a message prompting the user to remeasure from a point that is one meter back is superimposed on the interface screen 340. Alternatively, the processing unit 31 may prompt the user to remeasure by outputting a voice. Note that the distance that the user goes back is arbitrary and is set in advance and stored in the storage unit 32. Alternatively, the processing unit 31 may calculate the distance to go back based on the elapsed time since the second measurement value began to deteriorate and the speed measured by the speed sensor S2.

[0086] The processing unit 31 determines whether or not to resume measurement (step S307). When the user has gone back the set distance and the second measurement value is less than the set value, the processing unit 31 determines to resume measurement. Whether or not the user has gone back the set distance may be determined by the user's self-declaration. For example, when the preparation complete button 353 is pressed on the guidance screen shown in FIG. 9, it may be determined that the user has gone back the set distance. When it is determined not to resume measurement (S307: NO), the processing unit 31 waits until it is possible to resume measurement.

[0087] When it is determined that the measurement should be resumed (S307: YES), the processing unit 31 resumes the measurement (step S308). When the measurement is resumed, the processing unit 31 executes the processes of steps S301 to S304 again.

[0088] Next, the processing unit 31 determines whether or not to end the measurement (step S309). If the end confirmation button 347 on the interface screen 340 is pressed and the end is confirmed by the user, the processing unit 31 determines that the measurement has ended. If it is determined that the measurement has not ended (S309: NO), the processing unit 31 returns the process to step S301 and continues the measurement.

[0089] If it is determined that the measurement has ended (S309: YES), the processing unit 31 transmits the first measurement value acquired in step S301 (image data output from the RGB sensor 21, depth data output from the depth sensor 22, and thermographic image data output from the infrared camera 24) and the orientation data obtained from the orientation sensor S1 to the information processing device 1 (step S310).

[0090] The information processing device 1 semi-automatically creates a three-dimensional space model for predicting the environment of the target space based on data acquired from the terminal device 3. An existing method is used to create the three-dimensional space model.

[0091] As described above, in embodiment 3, if the behavior of the first measuring instrument has a large impact on the acquisition of the first measurement value, the measurement is temporarily stopped and resumed from a position a set distance back, thereby making it possible to collect data suitable for creating a three-dimensional space model.

[0092] In the third embodiment, if the behavior of the first measuring instrument has a large impact on the acquisition of the first measurement value, the measurement is temporarily stopped and the user is guided to go back a set distance, but it is also possible to not stop the measurement and instead continue measuring and be guided to go back a set distance.

[0093] In the first embodiment, a warning is output in accordance with the comparison result between the second measurement value and the set value, and in the third embodiment, a guide is given to remeasure from a point a set distance back in time in accordance with the comparison result between the second measurement value and the set value. Alternatively, two set values ​​(hereinafter referred to as a first set value and a second set value, and the first set value < the second set value) may be prepared, and a warning may be output if the second measurement value > the first set value and the second measurement value < the second set value, and measurement may be temporarily stopped (or not stopped) and a guide is given to remeasure from a point a set distance back in time in accordance with the comparison result between the second measurement value and the set value.

[0094] (Fourth embodiment) In the fourth embodiment, a configuration for creating a three-dimensional space model in the terminal device 3 will be described. The configuration of the information processing system and the configuration of the terminal device 3 are the same as those in the first embodiment, and therefore a description thereof will be omitted.

[0095] 10 is a flowchart showing the procedure for creating a three-dimensional space model. The processing unit 31 of the terminal device 3 reconstructs a three-dimensional point cloud of the target space based on the image data and depth data acquired from the measurement device 2 (step S401). The processing unit 31 can reconstruct the three-dimensional point cloud using, for example, SfM (Structure from Motion), which geometrically estimates a sparse point cloud from an input image, and MVS (Multi-View Stereo), which generates a dense point cloud based on the SfM point cloud. The method for reconstructing the three-dimensional point cloud is not limited to SfM and MVS, and any existing method can be used.

[0096] The processing unit 31 generates a three-dimensional space model for fluid calculation based on the generated three-dimensional point cloud data (step S402). The processing unit 11 can identify the position of each plane, such as planes corresponding to the wall, floor, and ceiling surfaces of the target space, and planes corresponding to the outer surfaces of objects such as various devices and fixtures, using, for example, a self-position estimation technique and a line-of-sight estimation technique, and generates a three-dimensional space model composed of multiple planes.

[0097] The processing unit 31 recognizes the positions and classes of objects included in the target space (step S403). In this embodiment, the objects to be recognized are things that affect the environment of the target space, and include, for example, furniture, fittings, people, industrial machinery, plants, air conditioners, ventilation equipment, etc. Furniture includes at least one of a computer, a monitor, lighting equipment, a heater, and a chair. Fittings includes at least one of a window, a blind, a partition, and a door. The processing unit 31 recognizes the positions and classes (attributes) of objects included in the target space, for example, using a machine learning learning model MD (see FIG. 11).

[0098] FIG. 11 is a schematic diagram showing an example of the configuration of a learning model MD. The learning model MD includes, for example, a convolutional neural network (CNN) that has been trained using deep learning. The learning model MD has an input layer L1 to which image data of a target space is input, an intermediate layer L2 that extracts features of the image data, and an output layer L3 that outputs an inference result related to a detected object. The learning model MD is, for example, a model based on YOLO.

[0099] The input layer L1 of the learning model MD has multiple nodes that accept input of image data, that is, the pixel values ​​of each pixel that constitutes the image of the target space and the surface of the object, and passes the input pixel values ​​to the intermediate layer L2.

[0100] The intermediate layer L2 has multiple sets of convolutional layers (CONV layers), pooling layers, and a fully connected layer. The convolutional layers filter the values ​​output from the nodes in the previous layer to extract feature maps. The pooling layer shrinks the feature maps output from the convolutional layers to obtain new feature maps.

[0101] The output layer L3 has nodes that output the final inference results for objects detected from the image data. The inference results include the center coordinate position and horizontal and vertical sizes of a bounding box surrounding the object, an object detection score indicating the likelihood that the image surrounded by the bounding box is an image of the object, and a class score indicating the likelihood that the object belongs to a specific class.

[0102] Such a learning model MD is learned using an existing method, and the learned learning model MD is stored in the memory unit 32 of the terminal device 3. The processing unit 11 inputs the acquired image data into the learned learning model MD and executes calculations using the learning model MD to recognize the position and class (attribute) of the object included in the image data.

[0103] In this embodiment, the terminal device 3 is configured to include the learning model MD, and the processing unit 31 of the terminal device 3 executes calculations using the learning model MD, but the information processing device 1 or the external server may be configured to include the learning model MD, and to cause the information processing device 1 or the external server to execute calculations using the learning model MD in response to instructions from the terminal device 3. In this case, the terminal device 3 may transmit image data acquired from the measurement device 2 to the information processing device 1 or the external server, and acquire the results of calculations using the learning model MD from the information processing device 1 or the external server.

[0104] After recognizing the objects in step S403, the processing unit 11 reflects the temperature distribution data in the three-dimensional space model (step S404). The processing unit 11 identifies a temperature distribution for each recognized object and associates the recognized object with the identified temperature distribution, thereby reflecting the temperature distribution data in the three-dimensional space model. The processing unit 11 also adds information based on the acquired orientation data to the three-dimensional space model (step S405). The three-dimensional space model to which the temperature distribution and orientation information have been added is stored in the storage unit 12.

[0105] In this embodiment, the terminal device 3 is configured to perform the restoration of the 3D point cloud, the generation of the 3D space model, and the recognition of the object within the terminal device 3, but these processes may be performed by the information processing device 1 or an external server. In this case, the calculation program and the learning model MD are stored in the information processing device 1 or the external server. The terminal device 3 transmits the image data and depth data acquired from the measurement device 2 to the information processing device 1 or the external server, and causes the information processing device 1 or the external server to perform calculations, thereby acquiring the 3D space model and the object recognition results.

[0106] As described above, in the fourth embodiment, a three-dimensional space model that reflects the attributes and positions of recognized objects can be created.

[0107] The embodiments disclosed herein should be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0108] 2. Measuring equipment 3 Terminal Devices 21 RGB sensor 22 Depth sensor 23 IR projector 24 Infrared Camera 31 Processing section 32 Storage section 33 Acquisition Department 34 Display section 35 Control section S1 orientation sensor S2 Speed ​​Sensor S3 Accelerometer S4 Angular Rate Sensor

Claims

1. A first measurement value is acquired from a first measuring device that measures the target space; obtaining a second measurement value indicative of a behavior of the first measuring device; evaluating the effect of the behavior on the acquisition of the first measurement by comparing the acquired second measurement with a set value; stopping the measurement by the first measuring device in accordance with a comparison result between the second measurement value and the set value; After stopping the measurement by the first measuring device, the device guides the user to resume the measurement from a point a set distance back. Processing section An evaluation device comprising:

2. The processing unit The second measurement value is obtained from a second measuring instrument that measures the behavior of the first measuring instrument. The evaluation device according to claim 1 .

3. The processing unit An alarm is output depending on a result of comparison between the second measurement value and the set value. The evaluation device according to claim 1 .

4. the first measuring instrument includes a plurality of types of sensors; The processing unit An alarm is output in accordance with a comparison result between one of the set values ​​set for each of the sensors and the second measurement value. The evaluation device according to claim 3 .

5. The warning may include: Composed of at least one of sound, vibration, light, letters and symbols The evaluation device according to claim 3 .

6. a display unit that displays information based on the second measurement value; The evaluation device according to claim 1 .

7. a receiving unit that receives an input of the setting value, The processing unit The second measurement value is compared with the set value received from the receiving unit to evaluate the influence of the behavior on the acquisition of the first measurement value. The evaluation device according to claim 1 .

8. The second measurement value includes at least one of a velocity, an acceleration, and an angular velocity of the first measuring instrument. The evaluation device according to claim 1 .

9. A first measurement value is acquired from a first measuring device that measures the target space; obtaining a second measurement value indicative of a behavior of the first measuring device; evaluating the effect of the behavior on the acquisition of the first measurement by comparing the acquired second measurement with a set value; stopping the measurement by the first measuring device in accordance with a comparison result between the second measurement value and the set value; After stopping the measurement by the first measuring device, the device guides the user to resume the measurement from a point a set distance back. An evaluation method in which processing is performed by a computer.

10. A first measurement value is acquired from a first measuring device that measures the target space; obtaining a second measurement value indicative of a behavior of the first measuring device; evaluating the effect of the behavior on the acquisition of the first measurement by comparing the acquired second measurement with a set value; stopping the measurement by the first measuring device in accordance with a comparison result between the second measurement value and the set value; After stopping the measurement by the first measuring device, the device guides the user to resume the measurement from a point a set distance back. A computer program that causes a computer to execute a process.

Images