Information processing device, information processing system, information processing method, and computer program

Abstract

The information processing device includes: an acquisition unit that acquires object space data obtained by measuring an inside of an object space; and a processing unit. The processing unit recognizes the attribute and position of the hiding target on the basis of the object spatial data acquired by the acquisition unit, associates and outputs the object spatial data with the attribute and position of the hiding target, and processes the hiding information of the hiding target.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing system, an information processing method, and a computer program. Background Art

[0002] Patent document 1 discloses a technique for creating model data representing the shape of an object based on three-dimensional point cloud data and using the created model data to perform fluid calculations. In Patent document 1, the shape information of a three-dimensional space can be acquired using the three-dimensional point cloud data and an indoor appearance model can be created. Prior art literature Patent Literature

[0003] Patent Document 1: Japanese Patent No. 6286902 Summary of the invention Technical problem to be solved by the invention

[0004] However, when confidential objects are included in the 3D point cloud data, there is a problem that the confidential objects are directly modeled. Although it is also possible to simply desensitize the objects that require confidential information, there is a problem that the boundary conditions required for fluid calculation cannot be set, thereby reducing the accuracy of fluid calculation.

[0005] An object of the present disclosure is to provide an information processing device and the like that can conceal a confidential object included in a target space and output the attribute and position of the concealed object in association with each other. Technical solutions adopted to solve technical problems

[0006] The information processing device of the first viewpoint of the present disclosure includes an acquisition unit and a processing unit, wherein the acquisition unit acquires object space data obtained by measuring within an object space, and the processing unit identifies the attributes and position of a hidden object target based on the object space data acquired by the acquisition unit, and outputs the attributes and position of the hidden object target relative to the environment of the object space in an associated manner based on the object space data acquired by the acquisition unit and the attributes and position of the hidden object target in the object space, thereby processing the hidden information of the hidden object target.

[0007] That is, the information processing device of the first viewpoint of the present disclosure includes an acquisition unit and a processing unit, wherein the acquisition unit acquires object space data obtained by measuring within the object space, and the processing unit identifies the attributes and position of a hidden object target based on the object space data acquired by the acquisition unit, outputs the object space data in association with the attributes and position of the hidden object target, and processes the hidden information of the hidden object target.

[0008] The information processing device of the second viewpoint of the present disclosure is based on the information processing device of the first viewpoint, and produces a three-dimensional space model of the object space based on the object space data acquired by the acquisition unit and the attributes and position of the hidden object target in the object space. The three-dimensional space model of the object space reflects the attributes and position of the hidden object target relative to the environment of the object space, and processes the hidden information of the hidden object target in the three-dimensional space model.

[0009] The information processing device of the third viewpoint of the present disclosure is based on the information processing device of the first viewpoint or the second viewpoint, and the hidden object target includes at least one of a display device, a person, an industrial machinery, a document, a destination display board, a notice, a whiteboard, a blackboard, a note, and an email.

[0010] An information processing device according to a fourth aspect of the present disclosure is the information processing device according to any one of the first to third aspects, wherein the processing unit acquires thermal imaging data of the target space.

[0011] An information processing device according to a fifth aspect of the present disclosure is the information processing device according to any one of the first to fourth aspects, wherein the processing unit creates the three-dimensional space model including the heating conditions of the hidden target based on the acquired thermal imaging data.

[0012] The information processing device of the sixth viewpoint of the present disclosure is based on the information processing device of any one of the first viewpoints to the fifth viewpoints, wherein the processing unit obtains the impact regulation data of the hidden object target from the storage unit, and produces the three-dimensional space model based on the acquired object space data and the impact regulation data, and the storage unit stores the attributes of the hidden object target and the impact regulation data in association with each other, and the impact regulation data stipulates the impact of the hidden object target relative to the environment of the object space.

[0013] An information processing device according to a seventh aspect of the present disclosure is the information processing device according to any one of the first to sixth aspects, wherein the influence regulation data includes a thermal boundary condition.

[0014] An information processing device according to an eighth aspect of the present disclosure is the information processing device according to any one of the first to seventh aspects, wherein the processing unit determines whether the target space data repeatedly includes the hidden target object.

[0015] An information processing device according to a ninth aspect of the present disclosure is the information processing device according to any one of the first to eighth aspects, wherein when the processing unit repeatedly recognizes the hidden object included in the object space, the processing unit removes the repeated hidden object.

[0016] An information processing device according to a tenth aspect of the present disclosure is the information processing device according to any one of the first to ninth aspects, wherein the processing unit replaces the concealed object with a model object not containing concealed information and creates the three-dimensional space model.

[0017] An information processing system according to an eleventh aspect of the present disclosure includes: an information processing device according to any one of the first to tenth aspects; and a measuring device, wherein the measuring device measures a physical quantity of a shape in the object space, and the information processing device acquires the object space data from the measuring device.

[0018] The information processing system of the twelfth aspect of the present disclosure includes an information processing device, which includes an acquisition unit and a processing unit, wherein the acquisition unit acquires object space data obtained in a measured object space, and the processing unit identifies the attributes and position of a hidden object target based on the object space data acquired by the acquisition unit, and associates and outputs the attributes and position of the hidden object target relative to the environment of the object space based on the object space data acquired by the acquisition unit and the attributes and position of the hidden object target in the object space, thereby processing the hidden information of the hidden object target.

[0019] That is, the information processing system of the twelfth aspect of the present disclosure includes an information processing device, wherein the information processing device includes an acquisition unit and a processing unit, wherein the acquisition unit acquires object space data obtained by measuring within the object space, and the processing unit identifies the attributes and position of a hidden object target based on the object space data acquired by the acquisition unit, outputs the object space data in association with the attributes and position of the hidden object target, and processes the hidden information of the hidden object target.

[0020] The information processing method of the thirteenth aspect of the present disclosure obtains object space data obtained in the measured object space, identifies the attributes and position of the hidden object target based on the acquired object space data, associates and outputs the attributes and position of the hidden object target relative to the environment of the object space based on the acquired object space data and the attributes and position of the hidden object target in the object space, and processes the hidden information of the hidden object target.

[0021] That is, the information processing method of the thirteenth aspect of the present disclosure acquires object space data obtained in the measurement object space, identifies the attributes and position of the hidden object target based on the acquired object space data, outputs the object space data in association with the attributes and position of the hidden object target, and processes the hidden information of the hidden object target.

[0022] The computer program of the fourteenth aspect of the present disclosure enables the computer to perform the following processing: obtaining object space data obtained in the measured object space, identifying the attributes and position of the hidden object target based on the acquired object space data, and associating and outputting the attributes and position of the hidden object target relative to the environment of the object space based on the acquired object space data and the attributes and position of the hidden object target in the object space, and processing the hidden information of the hidden object target.

[0023] That is, the computer program of the fourteenth aspect of the present disclosure enables the computer to perform the following processing: obtain object space data obtained in the measurement object space, identify the attributes and position of the hidden object target based on the acquired object space data, output the object space data in association with the attributes and position of the hidden object target, and process the hidden information of the hidden object target. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a block diagram showing a configuration example of an information processing system according to the present embodiment. Figure 2 It is a perspective view showing a configuration example and a usage example of the measuring device according to the present embodiment. Figure 3 This is a conceptual diagram of the learning model of this embodiment. Figure 4 This is a conceptual diagram of the target DB in this embodiment. Figure 5 This is a flowchart showing the processing procedure of the information processing device system according to the present embodiment. Figure 6 This is a flowchart showing the steps of creating a three-dimensional space model. Figure 7 This is a flowchart showing the steps of creating a three-dimensional space model. Figure 8 This is an explanatory diagram showing an example of a three-dimensional point cloud and a captured image. Fig. 9 This is an explanatory diagram showing an example of a three-dimensional space model. Fig.10 This is a conceptual diagram showing the concealment process of the concealment target object. Fig.11 This is a conceptual diagram showing a process of removing duplicate objects. Fig.12 This is a flowchart showing the editing process steps of an object. Fig.13A This is a conceptual diagram showing how to change a target. Fig. 13B This is a conceptual diagram showing how to change a target. Fig. 13C This is a conceptual diagram showing how to change a target. Fig.14A This is a conceptual diagram showing how to add targets. Fig. 14B This is a conceptual diagram showing how to add targets. Fig. 14C This is a conceptual diagram showing how to add targets. Fig.15A This is a conceptual diagram showing a method of editing a plane constituting an object space. Fig. 15B This is a conceptual diagram showing a method of editing a plane constituting an object space. Fig. 15C This is a conceptual diagram showing a method of editing a plane constituting an object space. DETAILED DESCRIPTION

[0025] Hereinafter, an air conditioning-related device and an air conditioning system according to this embodiment of the present disclosure will be described with reference to the drawings. The outline of this embodiment is as follows. By performing airflow analysis under actual property conditions when visiting customers' properties and conducting air conditioning-related sales activities, we can propose equipment based on clarifying the air environment problems that are actually occurring at customers' properties, such as temperature unevenness and stagnation. However, when creating a fluid calculation model for actual airflow analysis, it takes a lot of time to measure and model the room shape and dimensions of objects, and set conditions such as heat sources. As a method for solving the above-mentioned technical problems, a method for automatically creating a model for fluid calculation is proposed. As a system example, a handheld module that integrates a depth camera, a camera, a thermal imager, and a smart phone is used to take pictures while walking in the space, and the depth image and the image in the space are saved in the smart phone. The data saved in the smart phone is transmitted to the information processing device 1, and three-dimensional point cloud data is created using software for three-dimensional reconstruction. The floor, wall, and ceiling are detected from the three-dimensional point cloud data, and a model for fluid calculation is created as a spatial model of the surface. Here, the captured image is subjected to image recognition that has been learned in advance through machine learning, so as to determine the type (air conditioner, miscellaneous objects, etc.) and position of the target, and reflect it in the three-dimensional space model. As an alternative to shooting by handheld by a person, a cleaning robot, a drone, etc. can also be used for shooting.

[0026] Figure 1 is a block diagram showing a configuration example of an information processing system according to the present embodiment. Figure 22 is a perspective view showing a configuration example and a usage example of a measuring device 2 of the present embodiment. The information processing system of the present embodiment includes an information processing device 1, a measuring device 2, and a terminal device 3. The information processing system photographs and measures the distance of a space having a surface at least in part, and semi-automatically creates a three-dimensional space model for predicting the environment of the space based on the measured data. Hereinafter, the space of the modeled object is referred to as the object space.

[0027] The object space includes one or more targets that may affect the environment of the object space. The target includes at least one of the following targets: a target that affects the airflow of the object space; a target that affects the thermal environment of the object space; a target that affects the air quality of the object space; and a target that controls the environment of the object space. Specifically, the target includes at least one of miscellaneous objects, building accessories, people, industrial machinery, plants, air conditioners, and ventilation devices. Miscellaneous objects include at least one of computers, displays, lighting fixtures, heaters, and chairs. Building accessories include at least one of windows, blinds, partitions, and doors. Air conditioners include air purifiers.

[0028] The target space has a duct system for ventilation. The duct system includes an air inlet or an outlet of an air conditioner or a ventilator. The target space has a piping system of an air conditioner.

[0029] On the other hand, the target includes a confidentiality target and a non-confidentiality target. The confidentiality target includes at least one of a display device, a person, an industrial machine, a document, a destination display board, a notice, a whiteboard, a blackboard, a note, and an email.

[0030] The measuring device 2 includes an RGB sensor 21, a depth sensor 22, an IR transmitter 23, an infrared camera 24, a gripping unit, and a terminal holding unit, and is a device for measuring the physical quantity of the shape in the target space. The gripping unit is a member gripped by a user of the measuring device 2. The gripping unit is, for example, a rod-shaped member. The terminal holding unit is a member for holding the terminal device 3.

[0031] The RGB sensor 21 is a camera that has an imaging element such as a CMOS sensor or a CCD sensor and performs color photography of the object space. The RGB sensor 21 can capture video. The RGB sensor 21 outputs time-series image data (RGB data) obtained by capturing the object space. The time-series image data constituting the video includes time data indicating the shooting date.

[0032] The depth sensor 22 measures the distance (depth) from the measuring device 2 to the wall of the target space and various targets included in the target space. The depth sensor 22 outputs depth data indicating the depth obtained by the distance measurement. The IR transmitter 23 is a component for emitting infrared rays to the target space and the target to obtain more accurate depth data.

[0033] The infrared camera 24 is a camera that has an infrared imaging element and images a target space using infrared rays. The infrared camera 24 outputs thermal imaging image data obtained by imaging the target space.

[0034] The measuring device 2 of the above structure outputs the image data, the depth data and the thermal imaging image data to the terminal device 3. The measuring device 2 can also be configured to output the image data, the depth data and the thermal imaging image data to an external portable storage device and store them. The measuring device 2 can also be configured to send the image data, the depth data and the thermal imaging image data directly to the information processing device 1. Hereinafter, the image data and the depth data are referred to as object space data.

[0035] The terminal device 3 is, for example, a portable information processing terminal such as a smart phone, a mobile phone, a tablet terminal, a PDA (personal digital assistant), etc. The terminal device 3 includes an input interface for inputting image data, depth data, thermal imaging images, and orientation data. In addition, the terminal device 3 includes an orientation sensor 31.

[0036] The orientation sensor 31 is an orientation meter such as an electronic compass, for example, and outputs orientation data indicating the orientation of the measurement device 2 .

[0037] In addition, the content of the orientation data is not particularly limited as long as it is information that can determine the orientation of the object space. For example, the orientation data can also be information based on the latitude, longitude, time and position of a celestial body in the object space. A celestial body is an object whose position in the celestial sphere is known and exists in the universe that can be photographed. Examples of celestial bodies are the sun, the moon, and artificial satellites. If the captured image contains a celestial body whose position on the celestial sphere is known and the latitude, longitude and shooting time of the captured object space are known, the orientation of the object space can be determined. The orientation data may also be information based on the latitude, longitude, time, direction of the shadow in the room, and length of the shadow. The position of the sun is determined based on the direction and length of the shadow, so the orientation of the object space can be determined in the same way as above. The processing unit 11 of the information processing device 1 can identify the orientation of the target space based on the above information.

[0038] The terminal device 3 displays the image captured by the measuring device 2 on the display device based on the image data input through the input interface. The terminal device 3 stores the input image data, depth data, thermal imaging image, and orientation data obtained by the orientation sensor 31, and transmits them to the information processing device 1 at an appropriate time. The terminal device 3 may also be configured to store the input image data, depth data, thermal imaging image, and orientation data obtained by the orientation sensor 31 in a portable storage device.

[0039] According to the measuring device 2 of the above structure, the user can capture and measure the distance of the wall surface, floor surface, ceiling surface and target of the target space from multiple positions and directions by holding the measuring device 2 and moving it in the target space or pointing the measuring device 2 in various directions. The measuring device 2 provides the target space data, thermal imaging image data and orientation data obtained by measuring the target space from multiple positions to the information processing device 1.

[0040] <Information Processing Device 1> The information processing device 1 includes a processing unit 11 , a storage unit 12 , an acquisition unit 13 , a display unit 14 , and an operation unit 15 . The processing unit 11 is a processor having arithmetic circuits such as a CPU (central processing unit), an MPU (microprocessor), an ASIC (application-specific integrated circuit), an FPGA (field programmable gate array), an internal storage device such as a ROM (read-only memory), a RAM (random access memory), and an I / O terminal. The processing unit 11 may also include one or more arithmetic circuits such as a GPU (graphics processing unit), a TPU (tensor processing unit), or an AI chip (semiconductor for AI) dedicated to image processing related to object detection and image recognition. The processing unit 11 reads and executes a computer program ( Program Products )P, and create a three-dimensional space model of the object space. Each functional unit related to the creation of the three-dimensional space model can be implemented by software, or part or all of it can be implemented by hardware.

[0041] The storage unit 12 is, for example, a hard disk, an EEPROM (electrically erasable programmable read-only memory), a flash memory, or other memory. The storage unit 12 stores various programs executed by the processing unit 11 and various data required for the processing of the processing unit 11. In the present embodiment, the storage unit 12 stores at least a computer program P executed by the processing unit 11, a learning model 16, a vent discrimination model 17, and a target DB 18. The details of the learning model 16, the vent discrimination model 17, and the target DB 18 will be described later.

[0042] The computer program P may be written into the storage unit 12 at the manufacturing stage of the information processing device 1, or may be distributed from other information processing devices 1 via a network. The information processing device 1 acquires the computer program P through communication and writes it into the storage unit 12. The computer program P may also be stored in a storage medium 10 such as a semiconductor memory such as a flash memory, an optical disk, a magneto-optical disk, or a magnetic disk in a readable manner. The information processing device 1 reads the computer program P and stores it in the storage unit 12.

[0043] The acquisition unit 13 is a communication circuit for transmitting / receiving data to / from an external device by wire or wirelessly, or a reading circuit for reading data from a portable storage device. The information processing device 1 acquires image data, depth data, thermal imaging image data, orientation data, etc. from the measuring device 2 via the acquisition unit 13. The information processing device 1 acquires duct system diagram data from an external device via the acquisition unit 13, wherein the duct system diagram data indicates a duct system related to an air inlet or an air outlet of a ventilation device in a target space. The duct system diagram data includes direction information indicating a direction of the duct system and shape information indicating a shape of the duct system. The shape information includes information indicating a position of an air inlet or an air outlet of an air conditioner or a ventilation device. The information processing device 1 acquires piping system diagram data representing a piping system diagram of a target space from an external device via the acquisition unit 13. The piping system diagram data includes direction information representing the direction of the piping system and shape information representing the shape of the piping system. The piping system diagram data includes information related to the position of the indoor unit and the thermal boundary conditions of the air conditioner, etc. The information processing device 1 acquires time-series data of the outside temperature of the target space and time-series data of the amount of sunshine in the target space from an external device via the acquisition unit 13 .

[0044] The display unit 14 is a display device such as a liquid crystal display panel, an organic EL display panel, etc. The processing unit 11 displays an image related to the three-dimensional space model on the display unit 14 .

[0045] The operation unit 15 is an input device such as a mouse, a keyboard, or a touch panel that receives operations from a user of the information processing device 1. The processing unit 11 receives, through the operation unit 15, editing of the three-dimensional space model by the user.

[0046] In addition, the information processing device 1 may also be a server device connected to the network. The information processing device 1 may also be composed of a terminal computer and a server device. The information processing device 1 may also be composed of multiple computers and perform distributed processing, or may be implemented by multiple virtual machines set in one server, or may be implemented using a cloud server. In addition, the information processing device 1 may also be a terminal device 3. That is, the terminal device 3 may also be configured to function as the information processing device 1.

[0047] <Learning Model 16> Figure 3 : is a conceptual diagram of the learning model 16 of the present embodiment. The learning model 16 includes, for example, a convolutional neural network (CNN) that has completed learning through deep learning. The learning model 16 includes: an input layer 16a, the input layer 16a is input with image data captured by the measuring device 2; an intermediate layer 16b, the intermediate layer 16b extracts feature quantities of the image data; and an output layer 16c, the output layer 16c outputs an inference result related to the detected target. The learning model 16 is, for example, a YOLO model.

[0048] Each layer of the learning model 16 has a plurality of nodes. The nodes of each layer are connected by edges. Each layer has an activation function (response function), and the edges have weights. The values ​​output from the nodes of each layer are calculated based on the values ​​of the nodes of the previous layer, the weights of the edges, and the activation functions of the layers. The weights of the edges can be changed by learning.

[0049] The input layer 16 a of the learning model 16 has a plurality of nodes that receive input of image data, ie, pixel values ​​of pixels of an image constituting the object space and the surface of the target, and transfers the input pixel values ​​to the intermediate layer 16 b . The middle layer 16b has multiple groups of convolutional layers (CONV layers), pooling layers, and fully connected layers. The convolutional layer performs filtering processing on the values ​​output from the previous layer and extracts a feature map. The pooling layer reduces the feature map output from the convolutional layer to obtain a new feature map.

[0050] The output layer 16c has nodes that output the final inference results related to the target detected from the image data. The inference results include: the center coordinate position and the vertical and horizontal dimensions of the bounding box surrounding the target; the object detection score, which indicates the accuracy of the image surrounded by the bounding box being the image of the target; and the category score, which indicates the accuracy of the target belonging to a specific category.

[0051] In addition, the output obtained from the output layer 16c includes multiple bounding boxes that are repeated with each other. Through post-processing to remove the repeated bounding boxes, such as NMS (non-maximum suppression) processing, the most accurate position and size of the bounding boxes that respectively surround one or more targets included in the image of the object space, the object detection score, and the category score are obtained.

[0052] The method for generating the learning model 16 is described. First, a plurality of image data of an object space including a target and training data including an annotation file of each image data are prepared. The annotation file is teacher data indicating the correct value assigned to the corresponding image. Specifically, the annotation file is data indicating the center coordinate position, vertical and horizontal dimensions, and category of a bounding box surrounding the image of the target included in the corresponding image. The category indicates a group classified according to the attribute of the target.

[0053] When an image of training data is input to CNN, the weight coefficients of the neural network are optimized using error back propagation methods, error gradient descent methods, etc. so as to reduce the error (the value of a specified loss function or error function) between the data output from the CNN and the data represented by the teacher data, thereby generating a learning model 16.

[0054] In addition, YOLO is described as an example of the learning model 16, but R-CNN, FastR-CNN, FasterR-CNN, other CNNs, vision transformers, etc. can also be used to constitute the learning model 16. In addition, a learning model 16 using algorithms such as decision trees, random forests, and SVMs (support vector machines) can also be used. In addition, the learning model 16 can also be composed of a combination of the above-mentioned multiple algorithms.

[0055] The vent discrimination model 17 is an image recognition learning model that determines whether the vent is an inlet or an outlet based on the image data of the vent of the air conditioner or the ventilation device. The vent discrimination model 17 includes, for example, a convolutional neural network (CNN) that has completed learning through deep learning. The structure of the neural network itself is the same as that of the learning model 16. The vent discrimination model 17 has: an input layer, which is input with vent image data that is data of an image portion of the vent of the air conditioner or the ventilation device (hereinafter referred to as the vent image); an intermediate layer, which extracts feature quantities of the vent image data; and an output layer, which outputs accuracy data indicating the accuracy of the vent image conforming to the inlet or the outlet.

[0056] The method for generating the vent discrimination model 17 is described. First, a plurality of vent images and training data including teacher data are prepared, wherein the teacher data indicates whether each vent image corresponds to an inlet or an outlet. Then, when the vent image of the training data is input to the CNN, the processing unit 11 uses an error back propagation method, an error gradient descent method, etc. to optimize the weight coefficient of the neural network so as to reduce the error (the value of a specified loss function or error function) between the data output from the CNN and the data represented by the teacher data, thereby enabling the vent discrimination model 17 to be generated.

[0057] In addition, as long as sufficient discrimination accuracy can be obtained, the learning model 16 may be used to discriminate between the inlet and the outlet.

[0058] <Target DB 18> Figure 4 : is a conceptual diagram of the target DB 18 of the present embodiment. The target DB 18, for example, establishes correspondence and stores the target category, target type, confidentiality, boundary condition type, and boundary condition parameter. The target type includes the above-mentioned objects, building accessories, people, industrial machinery, plants, air conditioners, ventilation devices, etc. Confidentiality is data indicating whether the target is a hidden target or a non-hidden target. The boundary condition parameter is, for example, data indicating the temperature, pressure, and flow rate of the target surface, Dirichlet boundary condition, Neumann boundary condition, etc. The parameter includes, for example, sensible heat, latent heat, wind volume, wind direction, wind speed, etc.

[0059] The parameter of the environmental condition is an example of influence defining data that defines the influence of the concealment target object on the environment of the target space.

[0060] <Information Processing Method> Figure 5 1 is a flowchart showing the processing steps of the information processing device 1 system of this embodiment. The user moves or rotates the measuring device 2 in the target space, and photographs and measures the distance of the target space (step S111). The infrared camera 24 photographs the target space by infrared rays (step S112). The orientation sensor 31 detects the orientation of the measuring device 2 (step S113).

[0061] The terminal device 3 transmits the target space data, the thermal imaging image data, and the orientation data to the information processing device 1 (step S114 ).

[0062] The processing unit 11 of the information processing device 1 acquires the target space data, the thermal imaging image data, and the orientation data output from the terminal device 3 (step S115 ).

[0063] When the orientation data is not orientation data directly indicating the orientation of the object space, the processing unit 11 preferably converts the orientation data into data directly indicating the orientation through calculation processing. For example, the processing unit 11 calculates data indicating the direction based on the latitude, longitude, time, and position of a celestial body of the target space. For example, the processing unit 11 calculates data indicating the direction based on the latitude, longitude, and time of the target space, and the direction and length of the shadow in the room. Furthermore, the processing unit 11 may calculate data indicating the orientation in the three-dimensional space based on the image data and the time data.

[0064] The processing unit 11 obtains piping system diagram data from the external device (step S116) and obtains piping system diagram data (step S117). The processing unit 11 obtains external temperature time series data from the external device (step S118). The processing unit 11 obtains sunlight amount time series data from the external device (step S119). For example, a sensor installed around the target space detects the outside temperature and sunshine amount, and sends the outside temperature time series data and sunshine amount time series data to the information processing device 1. The processing unit 11 obtains the outside temperature time series data and sunshine amount time series data from the sensor. In addition, the processing unit 11 can also be configured to obtain these data from an external server that distributes meteorological information such as the outside temperature time series data and sunshine amount time series data.

[0065] The processing unit 11 creates a three-dimensional space model reflecting the influence of the object on the object space based on the acquired object space data and other data (step S120 ).

[0066] <3D space modeling> Figure 6 and Figure 7 is a flowchart showing the steps of creating a three-dimensional space model. Figure 8 is an explanatory diagram showing an example of a three-dimensional point cloud and a captured image. Fig. 9 is an explanatory diagram showing an example of a three-dimensional space model. Fig.10 This is a conceptual diagram showing the concealment process of the concealment target object.

[0067] The processing is summarized as follows. When creating a three-dimensional space model for fluid analysis, three-dimensional point cloud data of the object space is created by analyzing the image data and the depth data, and the type and position of the elements such as heat sources that need to be considered when performing fluid analysis are determined from the captured image by using the learning model 16 that has been learned in advance, and reflected in the three-dimensional space model.

[0068] like Figure 8As shown, the processing unit 11 of the information processing device 1 restores the three-dimensional point cloud of the object space based on the image data and the depth data (step S131). Figure 8 In the figure, the upper figure shows image data and depth data, the lower left figure shows 3D point cloud data, and the lower right figure shows image data corresponding to the 3D point cloud data.

[0069] like Fig. 9 As shown, the processing unit 11 uses the self-position estimation technology and the line of sight direction estimation technology based on the three-dimensional point cloud data to create a three-dimensional space model for fluid calculation (step S132). The processing unit 11 can use the self-position estimation technology and the line of sight direction estimation technology to determine the planes corresponding to the wall surface, floor surface, and ceiling surface of the target space, the planes corresponding to the outer surfaces of various equipment, objects, etc., and the positions of each plane, and create a three-dimensional space model composed of multiple planes.

[0070] like Figure 8 As shown in the lower right figure of , the processing unit 11 detects the target (non-hidden target and hidden target) by inputting the image data into the learning model 16, and identifies the position and category of the target (step S133). In addition, the processing unit 11 identifies the size of the target. By referring to the target DB 18 with the category as a key, the processing unit 11 identifies whether the detected target is a hidden target. In the case where the image data of the target contains information such as characters and symbols related to thermal boundary conditions, the processing unit 11 can also be configured to read the information such as characters and obtain information related to the attributes of the target and the thermal boundary conditions.

[0071] The processing unit 11 determines the boundary conditions of the non-hidden target and the hidden target based on the target recognition result (step S134). Specifically, the processing unit 11 determines the values ​​of the boundary conditions by reading the parameters of the boundary conditions corresponding to the categories of the non-hidden target and the hidden target.

[0072] The processing unit 11 adds boundary conditions of the non-hidden target object and the hidden target object to the three-dimensional space model (step S135). The position where the boundary condition is set in the three-dimensional space is determined based on the position and size of the object recognized in step S132 and the three-dimensional point cloud.

[0073] In addition, when the processing unit 11 recognizes the object of a chair in the object space, it can also be configured to give the thermal boundary conditions of a person to the object of the chair. The storage unit 12 of the information processing device 1 stores typical prescribed thermal boundary conditions of the object of a person, such as sensible heat and latent heat, and the processing unit 11 preferably gives the thermal boundary conditions to the object of the chair.

[0074] The processing unit 11 determines the temperature, heat or temperature characteristics of the non-concealed object and the concealed object based on the thermal imaging image data, and assigns a heating condition to the object (step S136 ).

[0075] The heat of an object or other target can be used to analyze the environment of the target space in more detail. To do this, thermal imaging is used to measure, calculate the temperature or heat of the target, and reflect it in the fluid calculation model.

[0076] The hidden object included in the three-dimensional space model is replaced with a model object that does not contain hidden information (step S137). The shape of the model object can be a roughly rectangular object surrounding the hidden object, or a target with a shape corresponding to the hidden object. For example, the storage unit 12 can also be configured to store the model objects in a manner corresponding to the category of the hidden object. The processing unit 11 reads the model object corresponding to the category of the hidden object, and replaces the hidden object with the read model object.

[0077] The method of replacing the hidden object with a model object is an example of a method of hiding the hidden object. As long as the hidden object can be hidden, the hidden object can be processed by any method. The computer and time point for making the hidden target anonymous are not particularly limited. When the information processing device 1 is composed of a terminal computer and a server device, the process can be executed by the terminal computer or the server device. It can also be configured so that the process of making the hidden target anonymous is executed by the terminal device 3.

[0078] Other boundary condition setting methods are described below. The processing unit 11 inputs the image data of the vent of the air conditioner or the ventilator into the vent identification model 17 to thereby identify whether the vent is a suction port or a blow-out port (step S138 ).

[0079] In step S138, an example of using machine learning to distinguish between the suction port and the blow-out port is described, but it can also be configured to calculate the feature value related to the suction port and the blow-out port based on the image data of the vent, and distinguish whether the vent is a suction port or a blow-out port based on the calculated feature value.

[0080] When the inlet and outlet can be distinguished with sufficient accuracy by the image recognition process of step S133 using the learning model 16 , the processing unit 11 may omit the processes of steps S138 to S141 .

[0081] The processing unit 11 performs position alignment between the object space represented by the object space data and the piping system represented by the piping system data based on the acquired orientation data related to the orientation of the object space, the information representing the outer shape of the object space based on the object space data, and the piping system diagram data (step S139). Alternatively, the object space and the piping system may be aligned based on the position of an object such as an elevator, a staircase, or a pillar whose accurate position is objectively known and unchanged.

[0082] The processing unit 11 determines whether the vent is a suction port or an outlet based on the acquired duct system diagram data (step S140). In step S140, the duct system diagram is used to determine whether the duct-type suction port or outlet is difficult to determine by image recognition. Regarding the inlet and outlet of a duct-type air conditioner or ventilation device, the results of fluid calculation differ greatly depending on whether it is an inlet or an outlet, so it is necessary to correctly distinguish them. However, since there is no significant difference in their appearance, it is difficult to distinguish whether it is an inlet or an outlet only through image recognition. For this reason, the information of the inlet and outlet is obtained from the duct system diagram of the target space and reflected in the three-dimensional space model.

[0083] According to step S140, by using the piping system diagram when creating the three-dimensional space model, the piping system of the air conditioner or the ventilator can be identified, and the setting of the thermal boundary conditions of the suction port and the blow-out port can be assisted.

[0084] The processing unit 11 determines whether the vent is a suction port or a blow-out port based on the thermal imaging image data containing the vent (step S141). In step S141, the temperature of the suction port and the blow-out port of the air conditioner or ventilation device and the temperature of the ceiling are measured using the thermal imaging image to determine the suction port and the blow-out port, and the thermal boundary conditions of the suction port and the blow-out port are reflected in the three-dimensional space model.

[0085] The processing unit 11 obtains orientation data related to the orientation of the object space, and based on the acquired orientation data, information representing the outer shape of the object space based on the object space data, and the piping system diagram data, performs position alignment between the object space represented by the object space data and the piping system diagram data (step S142).

[0086] The thermal boundary conditions of the air conditioner based on the acquired piping system diagram data are assigned to the three-dimensional space model (step S143). Since the piping system of the air conditioner is aligned with the target space, the processing unit 11 can determine the relationship between the target and the piping system of the air conditioner included in the target space. The processing unit 11 can determine information related to the thermal boundary conditions of the piping system corresponding to the target of the air conditioner, and can assign the thermal boundary conditions of the target to the three-dimensional space model.

[0087] In step S142 and step S143, when performing airflow analysis, the system information of the air conditioner becomes important. However, sufficient information about the air conditioner cannot be obtained only through image recognition of the target. For this reason, by using the piping system diagram data, the system information of the indoor unit can be reflected in the three-dimensional space model. In addition, the piping system diagram can also be displayed in an overlapping manner in the three-dimensional space model.

[0088] The processing unit 11 calculates the heat permeability of the building accessories based on the thermal imaging image data, the time series data of the external temperature of the target space, the time series data of the amount of sunlight, and the thickness data of the building accessories (step S144). The processing unit 11 assigns the calculated heat permeability as the thermal boundary condition of the building accessories (step S145). The thickness of the building accessories is obtained by the image recognition processing of step S133. The processing unit 11 can also be configured to combine the image recognition processing result of step S133 with the data of the three-dimensional point cloud to calculate the thickness of the building accessories.

[0089] The heat permeability of building components such as walls and windows can be used to analyze the environment of the target space in more detail. Therefore, in steps S144 and S145, thermal imaging is used to perform measurements, thereby calculating the heat permeability of building components and reflecting it in the three-dimensional space model. According to step S144 and step S145, the heat permeability of building accessories such as walls and windows can be calculated by using thermal imaging images, external air temperature time series data, and sunlight amount time series data, and the input of thermal boundary conditions can be assisted.

[0090] The processing unit 11 determines the usage scene of the object space based on the image data (step S146), and assigns the usage scene data representing the determined usage scene of the object space to the three-dimensional space data (step S147). For example, attributes are assigned to the three-dimensional space data matching scenes such as hospitals, schools, factories, and offices.

[0091] The processing unit 11 may specify the underfloor socket included in the target space based on the image data, and provide data indicating the position of the underfloor socket to the three-dimensional space model.

[0092] The processing unit 11 adds the orientation data or information based on the orientation data to the three-dimensional space model (step S148), and ends the three-dimensional space model creation process. For example, data indicating the orientation of the three-dimensional space model is added to the three-dimensional space model.

[0093] Back to Figure 5 The processing unit 11 determines whether the object space data includes objects repeatedly (step S121). For example, when the similarity of shapes and attributes of multiple objects is greater than a specified value and the distance between the multiple objects is less than a specified distance, the processing unit 11 determines that the multiple objects are repeated objects.

[0094] Fig.11 is a conceptual diagram showing the removal process of duplicate targets. In the case where the target is duplicated (step S121: Yes), Fig.11 As shown, the processing unit 11 removes duplicate targets (step S122).

[0095] When it is determined that the objects are not duplicated at the end of the process of step S122 (step S121 : No), the processing unit 11 executes the editing process of the three-dimensional space model (step S123 ).

[0096] In addition to the automatic creation of the three-dimensional space model, arbitrary editing is also possible. The fluid calculation model can be edited while the three-dimensional point cloud data is displayed in an overlapping manner, so that an accurate fluid calculation model can be created. Special surfaces such as glass and mirrors that allow infrared rays to pass through and reflect are sometimes not correctly identified as surfaces. Therefore, in this case, the surfaces of the objects in the three-dimensional space model are manually corrected and created. In the case where the type or position of the determined element does not match the actual one, corrections and creation are also manually performed. At this time, the three-dimensional point cloud data is displayed in an overlapping manner while being corrected and created, so that a more accurate fluid calculation model can be created.

[0097] Fig.12 1 is a flowchart showing the steps of editing the object. The processing unit 11 superimposes the three-dimensional point cloud on the created three-dimensional space model and displays it on the display unit 14 (step S151). The processing unit 11 receives the change of the object through the operation unit 15 (step S152), and changes the object according to the received operation (step S153). For example, the processing unit 11 receives the change of the parameters of the shape, type, and boundary conditions of the object and changes them.

[0098] When people move or gather in the object space, the three-dimensional space model changes. The processing unit 11 receives the increase or decrease of human objects in the object space and the change of their positions, and changes the number and positions of human objects.

[0099] The processing unit 11 may also obtain data related to a seating configuration diagram of the object space, and assign a human thermal boundary condition to a target in a seat based on the obtained data and information indicating the outer shape of the object space based on the object space data. The seating configuration diagram is data indicating the position of a target in a seat in the object space. The processing unit 11 assigns a human thermal boundary condition to a target in a seat based on the object space represented by the object space data and the position of the target in the seat represented by the data related to the seating configuration diagram. The storage unit 12 of the information processing device 1 stores typical prescribed thermal boundary conditions for a human target, and the processing unit 11 preferably assigns the thermal boundary condition to a target in a seat.

[0100] Fig.13A , Fig. 13B and Fig. 13C This is a conceptual diagram showing how to change a target. Fig.13A A three-dimensional spatial model representing the object space containing the recognized target. Fig. 13B Indicates the state of the edited object after the target is selected. Fig. 13C Indicates the state after the shape of the object has been changed.

[0101] The processing unit 11 accepts the addition of the target via the operation unit 15 (step S154 ), and adds the target according to the accepted operation (step S155 ).

[0102] Fig.14A , Fig. 14B and Fig. 14C This is a conceptual diagram showing how to add targets. Fig.14A A three-dimensional spatial model representing the object space where actual objects are not recognized. Fig. 14B Indicates the status of the target append operation in progress. Fig. 14C Indicates the state of the target after it is added.

[0103] The processing unit 11 receives the change and addition of the plane of the three-dimensional space through the operation unit 15 (step S156), and changes or adds the plane of the three-dimensional space to the target according to the received operation (step S157). The plane is a part equivalent to the wall surface in the object space. The user can change the position and size of the wall surface by operating the operation unit 15. In addition, the user can add a wall surface that is not reflected in the three-dimensional space model by operating the operation unit 15.

[0104] Fig.15A , Fig. 15B and Fig. 15C This is a conceptual diagram showing a method of editing a plane constituting an object space. Fig.15A A three-dimensional space model representing an object space in which the actual wall planes are not recognized. Fig. 15B Indicates the status of the wall plane addition operation in progress. Fig. 15C Indicates the state after the wall plane is added.

[0105] <Effects> According to the information processing device 1 and the like of the present embodiment, it is possible to create a three-dimensional space model reflecting the influence of objects included in the object space on the environment of the object space based on the object space data obtained by measuring the object space from a plurality of positions. It can reduce the workload of creating three-dimensional space models for fluid calculations (models of indoor shapes, objects, heat sources, and other elements). It is no longer necessary to collect the dimensions of indoor shapes and objects, eliminating the error in dimension collection.

[0106] The duct system diagram data can be used to create a three-dimensional space model to which boundary conditions are given in consideration of the difference between the suction port and the discharge port.

[0107] The thermal imaging image can be used to create a three-dimensional space model to which boundary conditions are given in consideration of the difference between the air inlet and the air outlet.

[0108] By image recognition processing, a three-dimensional space model to which boundary conditions are given in consideration of the difference between the air inlet and the air outlet can be created.

[0109] By using the piping system diagram data, a three-dimensional space model provided with system information of indoor units can be created.

[0110] A three-dimensional space model with heat permeability of building components such as walls and windows can be created based on thermal imaging images.

[0111] The object space and the target included in the three-dimensional space model can be edited and changed by the user's operation.

[0112] Through user operation, it is possible to add a wall plane or an object of the target space to the three-dimensional space model.

[0113] Able to create three-dimensional space models including orientation information.

[0114] A three-dimensional space model including data representing a utilization scenario of the target space can be created.

[0115] Able to create a three-dimensional space model without duplicate objects.

[0116] It is possible to create a three-dimensional space model that conceals a confidential object included in the target space and reflects the influence of the object on the target space.

[0117] A three-dimensional space model can be created by replacing hidden object targets with model targets.

[0118] Although the embodiments have been described above, it should be understood that various changes in form and details may be made without departing from the spirit and scope of the claims. At least a part of the above embodiments may be arbitrarily combined. Explanation of symbols

[0119] 1. Information processing device; 2. Measuring device; 3. Terminal device; 11. Processing Department; 12 storage unit; 13. Acquisition Department; 14 display unit; 15. Operation unit; 16. Learning model; 17. Ventilation outlet discrimination model; 18 TargetDB; 21RGB sensor; 22 depth sensor; 23IR transmitter; 24 infrared cameras; 31 azimuth sensor; PComputer Program.

Claims

1. An information processing device, characterized in that: The information processing device includes an acquisition unit and a processing unit. The acquisition unit acquires object space data obtained in the measurement object space. The processing unit Based on the object space data acquired by the acquisition unit, identifying the attributes and position of the hidden object target, Associating the object space data with the attributes and location of the hidden object target and outputting them, The hidden information of the hidden object target is processed.

2. The information processing device according to claim 1, characterized in that A three-dimensional space model of the object space is created based on the object space data acquired by the acquisition unit and the attributes and position of the hidden object target in the object space, wherein the three-dimensional space model of the object space reflects the attributes and position of the hidden object target relative to the environment of the object space, The hidden information of the hidden object target in the three-dimensional space model is processed.

3. The information processing device according to claim 1 or 2, characterized in that: The hidden object target includes at least one of a display device, a person, an industrial machine, a document, a destination display board, a notice, a whiteboard, a blackboard, a note, and an email.

4. The information processing device according to any one of claims 1 to 3, characterized in that: The processing unit acquires thermal imaging data of the target space.

5. The information processing device according to any one of claims 1 to 4, characterized in that: The processing unit creates the three-dimensional space model including the heating conditions of the concealed target object based on the acquired thermal imaging data.

6. The information processing device according to any one of claims 1 to 5, characterized in that: The processing unit obtains the impact regulation data of the hidden object target from the storage unit, and creates the three-dimensional space model based on the acquired object space data and the impact regulation data. The storage unit stores the attributes of the hidden object target and the impact regulation data in association with each other, and the impact regulation data stipulates the impact of the hidden object target on the environment of the object space.

7. The information processing device according to any one of claims 1 to 6, characterized in that: The impact specification data includes thermal boundary conditions.

8. The information processing device according to any one of claims 1 to 7, characterized in that: The processing unit determines whether the target space data repeatedly includes the hidden target object.

9. The information processing device according to any one of claims 1 to 8, characterized in that: The processing unit removes the duplicated hidden object when the hidden object included in the object space is repeatedly recognized.

10. The information processing device according to any one of claims 1 to 9, characterized in that: The processing unit replaces the concealed object with a model object that does not contain concealed information and creates the three-dimensional space model.

11. An information processing system, characterized in that: include: The information processing device according to any one of claims 1 to 10; as well as a measuring device for measuring a physical quantity of a shape in the object space, The information processing device acquires the object space data from the measuring device.

12. An information processing system, comprising an information processing device, characterized in that: The information processing device includes an acquisition unit and a processing unit. The acquisition unit acquires object space data obtained in the measurement object space. The processing unit Based on the object space data acquired by the acquisition unit, identifying the attributes and position of the hidden object target, Associating the object space data with the attributes and location of the hidden object target and outputting them, The hidden information of the hidden object target is processed.

13. An information processing method, characterized in that: Acquire the object space data obtained in the measurement object space, Based on the acquired object space data, identify the attributes and location of the hidden object target, Associating the object space data with the attributes and location of the hidden object target and outputting them, The hidden information of the hidden object target is processed.

14. A computer program, characterized in that The computer program causes the computer to execute the following processing: Acquire the object space data obtained in the measurement object space, Based on the acquired object space data, identify the attributes and location of the hidden object target, Associating the object space data with the attributes and location of the hidden object target and outputting them, The hidden information of the hidden object target is processed.

Images