Distortion identification device, distortion identification method, and recording medium

Abstract

A distortion identification device comprises: a three-dimensional data acquisition unit that acquires three-dimensional data representing an object; a distortion candidate identification unit that identifies a candidate for distortion of the object represented by the three-dimensional data with respect to the reference line; and an output unit that outputs information on the candidate for the distortion of the object.

Description

Distortion identification device, distortion identification method, and recording medium

[0001] The present disclosure relates to a distortion identification device and the like.

[0002] There are techniques for generating three-dimensional data that models an object. For example, Patent Literature 1 describes a technique for generating a three-dimensional model using Neural Radiance Fields (NeRF) based on a captured image that shows all or part of the object.

[0003] JP 2024-043792 A

[0004] However, identifying distortions in an object can still be difficult.

[0005] An example of an object of the present disclosure is to provide a distortion identification device or the like that facilitates identification of distortion in an object.

[0006] A distortion identification device according to one aspect of the present disclosure includes a three-dimensional data acquisition means for acquiring three-dimensional data representing an object, a distortion candidate identification means for identifying a candidate distortion of the object represented by the three-dimensional data relative to a reference line, and an output means for outputting information relating to the candidate distortion of the object.

[0007] In one aspect of the present disclosure, a distortion identification method includes a computer performing the following processes: acquiring three-dimensional data representing an object; identifying potential distortions of the object represented by the three-dimensional data relative to a reference line; and outputting information about the potential distortions of the object.

[0008] A program in one aspect of the present disclosure causes a computer to perform the following processes: acquire three-dimensional data representing an object; identify potential distortions of the object represented by the three-dimensional data relative to a reference line; and output information regarding the potential distortions of the object.

[0009] Each program may be stored in a non-transitory computer-readable recording medium.

[0010] According to the present disclosure, it is possible to facilitate identification of distortions in an object.

[0011] FIG. 1 is a block diagram showing an example of the configuration of a distortion identifying device. FIG. 1 is an explanatory diagram showing an example of an object being parallel projected and a reference line being superimposed and displayed. FIG. 1 is an explanatory diagram showing an example of an object being displayed so that a portion with a difference in gloss and a portion with an inclination relative to the reference line can be distinguished. FIG. 2 is a flowchart showing an example of the operation of a distortion identifying device. FIG. 2 is an explanatory diagram showing an example of the configuration of a distortion identifying system including a distortion identifying device. FIG. 3 is a block diagram showing an example of the configuration of a distortion identifying device. FIG. 4 is an explanatory diagram showing an example of an object having temperature information superimposed and displayed at each position. FIG. 5 is an explanatory diagram showing an example of the position of a tube included in an object. FIG. 6 is a flowchart showing an example of the operation of a distortion identifying device. FIG. 7 is an explanatory diagram showing an example of the hardware configuration of a computer.

[0012] Hereinafter, with reference to the drawings, embodiments of a distortion identification device, a distortion identification method, a program, and a non-transitory recording medium for recording the program according to the present disclosure will be described in detail. The disclosed technology is not limited to these embodiments.

[0013] First Embodiment A first embodiment will be described in detail with reference to the drawings.

[0014] 1 is a block diagram showing an example of the configuration of a distortion identification device 10. The distortion identification device 10 presents information on candidates for distortion of an object. Here, the object is not particularly limited to a building, a structure, a moving object, or the like.

[0015] In FIG. 1, the distortion identification device 10 includes a three-dimensional data acquisition unit 101 , a distortion candidate identification unit 103 , and an output unit 105 .

[0016] The three-dimensional data acquisition unit 101 acquires three-dimensional data representing an object. The three-dimensional data may be three-dimensional data created using NeRF, three-dimensional mesh data representing a general three-dimensional mesh model, or point cloud data, and is not particularly limited. The three-dimensional data acquisition unit 101 may acquire three-dimensional data prepared in advance from a database or the like, may acquire three-dimensional data by accepting input of three-dimensional data prepared in advance, or may acquire three-dimensional data by generating three-dimensional data, and is not particularly limited.

[0017] Here, an example of the three-dimensional data acquisition unit 101 acquiring three-dimensional data using NeRF will be described. For example, the three-dimensional data acquisition unit 101 may acquire three-dimensional data by using NeRF to generate three-dimensional data representing an object based on multiple images of the object captured from multiple different positions. Features of modeling using three-dimensional data using NeRF will be described later.

[0018] Next, the distortion candidate identification unit 103 identifies a distortion candidate of the object represented by the three-dimensional data relative to the reference line. Furthermore, for example, the reference line may be a line parallel to the ground or a line perpendicular to the ground. Taking the example of a case where the object is a building, the reference line may be a line parallel to or perpendicular to the ground on which the building stands. For example, if past three-dimensional data of the object can be acquired, the reference line may be a line representing the shape of the object in the past. Furthermore, for example, if an imaging device that captures an object is equipped with an inertial sensor such as a gyro sensor or an acceleration sensor, the reference line is a reference line obtained from the sensor. The imaging device may be, for example, an imaging device equipped in a terminal device or an imaging device such as a compact camera. Generally, at least one of the horizontal direction and the vertical direction may be identified by sensing using a sensor equipped in the imaging device. The reference line obtained from the sensor may be a line representing the horizontal direction or the vertical direction. As an example of a line representing the shape of an object in the past, a case where the object is a house will be described. If three-dimensional data of the house as constructed can be acquired, the shape of the house represented by the three-dimensional data of the house as constructed is the reference line.

[0019] As an example of the process in which the distortion candidate identifying unit 103 identifies distortion candidates using a reference line, if the tilt of the object represented by the three-dimensional data with respect to the reference line is equal to or greater than a predetermined value, the distortion candidate identifying unit 103 identifies the portion with the tilt equal to or greater than the predetermined value as a distortion candidate. For example, the distortion candidate identifying unit 103 may determine whether the tilt of a line representing the appearance of each element included in the object represented by the three-dimensional data is equal to or greater than a predetermined value. The lines representing the appearance of each element are one example, and are not particularly limited as long as a portion with an identifiable tilt with respect to the reference line can be identified.

[0020] As another example of the process in which the distortion candidate identification unit 103 identifies distortion candidates using a reference line, the distortion candidate identification unit 103 identifies, as a distortion candidate, a deviation in the shape of an object represented by three-dimensional data relative to the reference line. For example, there is an object that includes a portion where the same shape is repeated multiple times. An example of a portion where the same shape is repeated multiple times is a block wall. Taking a block wall as an example, the reference line is a line that represents the shape of each block, and the distortion candidate identification unit 103 identifies whether the shape of each block of the block wall represented by three-dimensional data is deformed relative to the line that represents the shape of each block.

[0021] Furthermore, when the three-dimensional data is generated using NeRF, the distortion candidate identification unit 103 may identify portions of the object represented by the three-dimensional data that have differences in gloss. Here, the characteristics of modeling using three-dimensional data using NeRF will be briefly described. For example, NeRF is a method for representing three-dimensional (3D) objects using deep learning. When an object is modeled using NeRF, the object's shape, surface gloss, etc. are expressed using mathematical formulas. Unlike general three-dimensional mesh models, NeRF can reproduce different appearances depending on the viewpoint. For example, in modeling that creates three-dimensional mesh data, objects of the same color are modeled identically. In contrast, NeRF can reproduce differences in the appearance of an object due to the way light hits it, which is caused by differences in material. Differences in appearance include, for example, gloss. For example, if an object has cracks, fissures, scratches, etc., the area or its surroundings may be distorted. Alternatively, the object itself may be distorted, causing the cracks or fissures. It is expected that unevenness caused by cracks, fissures, scratches, etc. on at least a part of an object will result in differences in gloss. It is also expected that peeling paint on at least a part of an object will result in differences in gloss. Gloss is reproduced in 3D data using NeRF.

[0022] The portion where there is a difference in gloss may be, for example, a portion where there is a change in gloss. Specifically, the distortion candidate identification unit 103 identifies a portion where there is a change in gloss from a reproduction image representing an object reproduced based on three-dimensional data as a candidate for distortion of the object. Furthermore, a portion where gloss ends may be a portion where the shape of the object changes. Therefore, the distortion candidate identification unit 103 identifies a portion where gloss ends from the reproduction image as a portion where gloss changes, as a candidate for distortion of the object.

[0023] In particular, the higher the degree of reproduction of an object using 3D data using NeRF, the more likely it is that parts of the object containing hidden cracks, etc., will be identified. Note that when using NeRF, the greater the number of viewpoints, the higher the degree of reproduction of the object when modeled.

[0024] Next, the output unit 105 outputs information regarding the object distortion candidates. The output method is not particularly limited to audio output, display, projection, storage, etc. The audio output destination, display destination, projection destination, and storage destination are not particularly limited. The display destination may be, for example, a terminal device owned by the user. The information regarding the object distortion candidates is not particularly limited to position information indicating the position of the object distortion candidates, information indicating the number of object distortion candidates, information indicating the degree of inclination with respect to the reference line, information indicating the degree of deformation of the object's shape with respect to the reference line, information regarding the degree of gloss difference, information regarding the size of the distortion candidates, etc.

[0025] Furthermore, for example, the output unit 105 may output whether or not there are candidates for distortion of the object. Furthermore, for example, the output unit 105 may output the number of candidates for distortion of the object.

[0026] For example, the output unit 105 may output position information representing the position of a candidate for distortion of the object. The position information may be information that can identify the position of the object. For example, if the object is a house, the output unit 105 may output information such as "below right side of the entrance" as position information representing the candidate for distortion of the object.

[0027] As an example of displaying information about distortion candidates, when the output unit 105 displays an object represented by three-dimensional data, it may display distortion candidate portions of the object represented by the three-dimensional data in an identifiable manner.

[0028] Furthermore, when displaying an object represented by three-dimensional data, the output unit 105 may superimpose a reference line on the object represented by the three-dimensional data. For example, if the object appears curved in the captured image, the reference line may not be displayed properly. Therefore, the output unit 105 parallel projects the object represented by the three-dimensional data based on the three-dimensional data, and superimposes the reference line on the object represented by the three-dimensional data. In this way, if the object represented by the three-dimensional data is parallel projected, the reference line can be more accurately superimposed and displayed.

[0029] FIG. 2 is an explanatory diagram showing an example in which an object is parallel projected and a reference line is superimposed on the object. For ease of explanation, a simple object is used as an example in FIG. 2. In FIG. 2, the output unit 105 parallel projects and displays the object represented by three-dimensional data. The output unit 105 then superimposes and displays the reference line on the object represented by the three-dimensional data. This makes it easier for the user to check for possible distortions in the object.

[0030] Furthermore, when the distortion candidate identification unit 103 identifies a portion where there is a difference in gloss, the output unit 105 outputs the portion where there is a difference in gloss identified by the distortion candidate identification unit 103 as a candidate for distortion of the object. Note that the output format and output examples are not particularly limited, as described above.

[0031] In addition, when a portion with a difference in gloss is identified as a candidate for distortion of the object, and when a candidate for distortion of the object represented by the three-dimensional data relative to the reference line is identified, the output unit 105 may output the candidate for distortion of the object that is the portion with a difference in gloss in a manner that allows it to be distinguished from the candidate for distortion of the object represented by the three-dimensional data relative to the reference line.

[0032] 3 is an explanatory diagram showing an example in which portions with differences in gloss and portions with inclination relative to a reference line are displayed in a distinguishable manner. For example, in FIG. 3, the output unit 105 displays portions with differences in gloss on an object represented by three-dimensional data in a distinguishable manner using star marks. For example, in FIG. 3, the output unit 105 displays portions with inclination relative to a reference line on an object represented by three-dimensional data in a distinguishable manner using triangular marks.

[0033] Furthermore, the output unit 105 may display information regarding candidate object distortions on a display device using augmented reality (AR). The output unit 105 displays information regarding candidate object distortions on the object, for example, on a display device. The display device may be a display device provided in a terminal device. For example, the output unit 105 may display marks indicating candidate object distortions on an actual object using AR. The marks are not particularly limited to symbols, numbers, letters, etc. The marks may be capable of distinguishing between areas with differences in gloss and areas that are inclined relative to a reference line.

[0034] As an example of projecting a distortion candidate, the output unit 105 may project information related to the distortion candidate of the object onto the actual object. For example, the output unit 105 may project a mark representing the distortion candidate of the object onto the actual object. As in the case described in the example of AR display, the mark is not particularly limited.

[0035] The output format and the output method may be combined as appropriate. The output unit 105 may also output the date and time when the captured image of the object used to generate the three-dimensional data was captured.

[0036] 4 is a flowchart showing an example of the operation of the distortion identification device 10. First, the acquisition unit acquires three-dimensional data representing an object (step S101).

[0037] Next, the distortion candidate identification unit 103 identifies candidate distortions of the object based on the three-dimensional data (step S102). In step S102, the distortion candidate identification unit 103 identifies candidate distortions of the object represented by the three-dimensional data relative to the reference line. Furthermore, if the three-dimensional data is three-dimensional data generated using Nerf, in step S102 the distortion candidate identification unit 103 may identify areas of the object represented by the three-dimensional data that have differences in gloss. Then, the output unit 105 outputs information regarding the candidate distortions of the object (step S103).

[0038] As described above, in the first embodiment, the distortion identification device 10 acquires three-dimensional data representing an object, identifies candidate distortions of the object represented by the three-dimensional data relative to a reference line based on the acquired three-dimensional data, and outputs information about the candidate distortions of the object. This allows the user to check the information about the candidate distortions of the object. This makes it easier to identify distortions of the object. For example, the user can use the candidate distortions to check distortions in the actual object or perform detailed work to address the distortions. In this way, it is possible to reduce the amount of investigation work required by the user.

[0039] The distortion identification device 10 may also acquire 3D data by generating 3D data representing an object using NeRF based on multiple images of the object captured from multiple different positions. This allows for more accurate identification of potential distortions by using 3D data that can more accurately reproduce the object. For example, 3D data using NeRF more accurately reproduces gloss and other characteristics. Gloss may appear different not only depending on the material but also on unevenness. Irregularities may be caused by cracks or fissures, for example, rather than the original unevenness of the object. Therefore, when the 3D data is generated using NeRF, the distortion identification device 10 identifies areas of the object represented by the 3D data that have differences in gloss and outputs the identified areas as potential distortions of the object. This makes it easier for users to identify potential distortions of the object that may be caused by cracks or fissures.

[0040] Second Embodiment In the second embodiment, an example of identifying the wall thickness when the object is a building or the like will be described. Also, in the second embodiment, an example of identifying distortion candidates by further using sensor data detected by a sensor will be described. Also, in the second embodiment, an example of superimposing and displaying sensor data detected by a sensor and three-dimensional data will be described. The second embodiment will be described in detail with reference to the drawings. Below, to the extent that the description of the second embodiment is not unclear, description of content that overlaps with the above description will be omitted.

[0041] 5 is an explanatory diagram showing an example of the configuration of a distortion identification system including a distortion identification device. The distortion identification system 1 includes a distortion identification device 20 and a terminal device 21.

[0042] The distortion identification device 20 is connected to a terminal device 21 via a communication network. The terminal device 21 may have, for example, an imaging device capable of capturing an image of an object. For example, the terminal device 21 may be an output device that outputs information from the distortion identification device 20, or an input device that inputs information to the distortion identification device 20. For example, the terminal device 21 may be pre-installed with an application program that can output information from the distortion identification device 20 or transmit information to the distortion identification device 20. For example, the terminal device 21 may access the website of the distortion identification device 20 via the communication network NT. The type of the terminal device 21 is not particularly limited, and may be a personal computer (PC), a smartphone, a tablet device, a head-mounted display (HMD), or the like. The number of terminal devices 21 may be provided for each user, and is not particularly limited.

[0043] The distortion identification device 20 may also be connected to a sensor 22 via a communication network. The sensor 22 is a device capable of sensing an object. The type of the sensor 22 is not particularly limited, and may be an infrared sensor, a depth sensor, a polarization camera, or the like. For example, the depth sensor may be a LiDAR (Light Detection And Ranging) sensor, a stereo camera, or the like, and is not particularly limited. The terminal device 21 may also have the sensor 22.

[0044] The type of communication network is not particularly limited, and may be configured by a plurality of communication networks.

[0045] Fig. 6 is a block diagram showing an example of the configuration of the distortion identification device 20. In Fig. 6, the distortion identification device 20 includes a three-dimensional data acquisition unit 201, a distortion candidate identification unit 203, an output unit 205, a structure identification unit 207, and a sensor data acquisition unit 209.

[0046] The three-dimensional data acquisition unit 201 may have, as a basic function, the function of the three-dimensional data acquisition unit 101 shown in Fig. 1. The distortion candidate identification unit 203 may have, as a basic function, the function of the distortion candidate identification unit 103 shown in Fig. 1. The output unit 205 may have, as a basic function, the function of the output unit 105 shown in Fig. 1.

[0047] Here, when displaying an object represented by three-dimensional data, the output unit 205 may perform parallel projection as described in the first embodiment. The output unit 205 may output various information to the terminal device 21. The output method and output destination are not particularly limited as described in the first embodiment.

[0048] First, an example will be described in which an image capturing device provided in the terminal device 21 captures images of an object from multiple positions in response to a user's operation. For example, the three-dimensional data acquisition unit 201 acquires images of an object captured from multiple positions by the image capturing device provided in the terminal device 21 in response to a user's operation.

[0049] Here, when creating three-dimensional data using NeRF, if the positional relationship between the inside and outside of an object is unknown, it may not be possible to connect and reproduce the inside and outside of the object. For example, if the object is a house, if the positional relationship between an image captured from a room in the entrance of the house and an image captured from outside the entrance of the house is unknown, it may not be possible to connect and reproduce the room in the entrance of the house and the part outside the entrance of the house.

[0050] For example, if the position of the viewpoint relative to the global origin, i.e., the relative positions of the inside and outside of an object, are known, the inside and outside of the object can be connected in 3D data reproduced using NeRF. For example, markers are placed on at least one of the inside and outside of an actual object. The markers may be paper, and are not particularly limited as long as their positional relationship can be identified. The 3D data acquisition unit 201 acquires images of the inside and outside of the object taken from multiple different positions.

[0051] Then, the three-dimensional data acquisition unit 201 generates three-dimensional data representing the object using NeRF based on the relative viewpoint positions when each of the multiple images identified based on the size and orientation of the same fixed marker when imaged from inside the object and the size and orientation of the marker when imaged from outside the object, and based on the multiple images.

[0052] In addition, in areas where the image capture is likely to be interrupted, it is possible to align the image using markers, etc., in the same way as for the inside and outside of the object. Note that the inside and outside of the object may be expressed as the interior view and exterior view of the object.

[0053] Furthermore, if the interior and exterior of an object can be reproduced as a single piece of three-dimensional data, it may be possible to identify the structure of the object. For example, the structure identification unit 207 may identify the thickness of a wall included in the object represented by the three-dimensional data.

[0054] Note that the wall is just one example, and the thickness between the inside and outside of the object may simply be specified, or the thickness of a partition separating two different compartments in the object may be specified.

[0055] Then, the output unit 205 outputs information about the wall thickness identified by the structure identification unit 207. For example, the information about the wall thickness may be information in which the wall thickness is expressed numerically, information indicating the degree of wall thickness, or information on a relative comparison between the identified wall thickness and other thicknesses.

[0056] Furthermore, the distortion candidate identifying unit 203 identifies, as a distortion candidate, a portion where there is a difference of a predetermined thickness or more between the wall thickness identified by the structure identifying unit 207 and the wall thickness in the design data of the object. Then, the output unit 205 outputs information on the distortion candidates identified by the distortion candidate identifying unit 203.

[0057] Next, an example will be described in which information relating to detected values ​​contained in sensor data is superimposed on an object represented by three-dimensional data.

[0058] The sensor data acquisition unit 209 acquires sensor data detected at a plurality of different positions on the object by the sensors 22. As described above, the type of the sensors 22 is not particularly limited.

[0059] The output unit 205 then superimposes information about the detection values ​​at each position of the object, which is represented by the sensor data, on the object represented by the three-dimensional data. The information about the detection values ​​may be the numerical values ​​of the detection values ​​themselves, or may be information indicating the degree of the detection values, and is not particularly limited.

[0060] For example, the output unit 205 may superimpose information related to the detection values ​​on the object represented by the three-dimensional data like a heat map. As an example of a heat map, the output unit 205 may superimpose a color corresponding to the detection values ​​on the object represented by the three-dimensional data. As an example of a heat map, the output unit 205 may superimpose a pattern corresponding to the detection values ​​on the object represented by the three-dimensional data.

[0061] For example, a case will be described in which the sensor 22 is an infrared sensor. The sensor data includes temperature data measured by the infrared sensor at multiple different positions on the object. The output unit 205 superimposes information about the temperature at each position on the object, represented by the temperature data, on the object represented by the three-dimensional data. The temperature information may be a numerical value of the temperature, information indicating the degree of temperature, such as high or low, or information relative to a measured temperature compared with a predetermined temperature. The predetermined temperature may be a uniquely determined temperature or may vary depending on the position. Furthermore, the predetermined temperature may be determined based on a temperature measured in the past, or may be a temperature set by the user via the terminal device 21, etc., but is not particularly limited.

[0062] FIG. 7 is an explanatory diagram showing an example in which temperature information is superimposed on each position of an object. In FIG. 7 , the output unit 205 displays an object represented by three-dimensional data on the terminal device 21. In FIG. 7 , the output unit 205 superimposes temperature information at each position of the object on the object represented by the three-dimensional data. In FIG. 7 , the temperature information is displayed like a heat map. Specifically, in FIG. 7 , the temperature information is displayed so that temperature zones can be distinguished by three levels of color shading: high, medium, and low. Note that the temperature display in FIG. 7 is an example. For example, while temperatures are distinguished by color shading in FIG. 7 , temperatures may also be distinguished by color. As shown in FIG. 7 , the output unit 205 may display a triangular mark on a portion that is inclined relative to a reference line as described in the first embodiment. The output unit 205 may display a star mark on a portion that has a difference in gloss as described in the first embodiment.

[0063] The sensor 22 includes a depth sensor. The sensor data includes distance data measured by the depth sensor at different positions on the object. For example, the output unit 205 superimposes information about the distance to each position of the object, represented by the distance data, on the object represented by the three-dimensional data. The distance to each position of the object is, for example, the distance from the depth sensor to each position of the object.

[0064] Furthermore, the output unit 205 may superimpose information about the detection values ​​at each position of the object onto the actual object using AR. Furthermore, the output unit 205 may project information about the detection values ​​at each position of the object onto the actual object.

[0065] Furthermore, for example, the output unit 205 may identifiably display each detection value represented by a plurality of pieces of sensor data measured by different types of sensors 22 on the object represented by the three-dimensional data.

[0066] Furthermore, for example, the distortion candidate identifying unit 203 may identify distortion candidates of an object based on detection values ​​included in the sensor data. Specifically, for example, the distortion candidate identifying unit 203 may identify distortion candidates by at least one of a relative comparison and an absolute comparison of detection values ​​included in the sensor data. As an example of a relative comparison, the distortion candidate identifying unit 203 may identify distortion candidates by comparing detection values ​​included in the sensor data with detection values ​​at other positions on the object. The detection values ​​at other positions may be detection values ​​at representative positions or statistical values ​​such as an average, median, or mode. As an example of an absolute comparison, the distortion candidate identifying unit 203 may identify positions of an object where detection values ​​included in the sensor data are equal to or greater than a threshold as distortion candidates. The threshold may be a uniquely determined threshold or may vary depending on the position. The threshold may be a value determined based on detection values ​​measured in the past, a value determined according to the weather, or a value that the user can set appropriately via the terminal device 21, etc., and is not particularly limited. As another example of absolute comparison, the distortion candidate identifying unit 203 may identify, as a distortion candidate, the position of an object whose detection value included in the sensor data is within a predetermined range. As another example of absolute comparison, the distortion candidate identifying unit 203 may identify, as a distortion candidate, the position of an object whose detection value included in the sensor data is outside a predetermined range.

[0067] Next, an example of identifying the distortion of a pipe included in an object will be described. The pipe included in the object may be a drainage pipe, a water distribution pipe, or the like, but is not particularly limited.

[0068] For example, if the object is a house, pipes such as water pipes may be hidden inside the walls and invisible. This can make it difficult to check for distortions in the pipes. Therefore, an infrared sensor measures the temperature at each position on the object while a liquid at a predetermined temperature is passed through the pipes included in the object. An example will be described in which the position of the pipe included in the object is identified based on whether there is a part whose temperature is close to the predetermined temperature. The predetermined temperature may be any temperature that can be distinguished from the temperatures of other objects. First, the sensor data acquisition unit 209 acquires temperature data measured by the infrared sensor at multiple different positions on the object while a liquid at a predetermined temperature is passed through the pipes included in the object.

[0069] Next, the distortion candidate identifying unit 203 identifies a portion where there is a difference between the position of a pipe included in the object in the design data for the object and the position of the object through which the liquid is estimated to have passed based on the temperature data. The design data may be information sufficient to identify, for example, the position of the pipe, and the method of expressing the design data is not particularly limited. For example, the distortion candidate identifying unit 203 may identify, based on the temperature data, each position of the object that is within a predetermined range of difference from a predetermined temperature as the position of the pipe.

[0070] The output unit 205 outputs information about the candidate distortion of the pipe, which is the part identified by the distortion candidate identifying unit 203, to the object represented by the three-dimensional data.

[0071] Furthermore, the output unit 205 outputs information about the pipe whose position has been identified based on the temperature data. For example, the information about the pipe may be position information indicating the position of the pipe identified based on the temperature data, information about the temperature at the identified position of the pipe, or a combination thereof.

[0072] FIG. 8 is an explanatory diagram showing an example of the position of a pipe included in an object. The output unit 205 highlights information about the temperature at the position of the pipe as information about the pipe whose position is identified based on the temperature data on the object represented by the three-dimensional data. The strain identification device 20 can visualize the position of a pipe in an invisible location. Note that output formats and output methods may be combined as appropriate. The output unit 205 may also output information such as the date and time of sensing. The output unit 205 may also output information about identified candidates for strain, sensing data, etc. in chronological order.

[0073] 9 is a flowchart showing an example of the operation of the distortion identification device 20. First, the three-dimensional data acquisition unit 201 acquires three-dimensional data representing an object (step S201).

[0074] The sensor data acquisition unit 209 acquires sensor data detected by the sensor 22 at multiple different positions on the object (step S202). Next, the distortion candidate identification unit 203 identifies candidate distortions of the object (step S203). In step S203, the distortion candidate identification unit 203 may identify candidate distortions of the object based on the 3D data, as described in step S102. In step S203, the distortion candidate identification unit 203 may identify candidate distortions of the object based on the sensor data. Then, the output unit 205 displays information about candidate distortions of the object while superimposing the sensor data on each position of the object represented by the 3D data (step S204). This concludes the description of the processing performed by the distortion identification device 20 shown in FIG. 9.

[0075] As described above, in the second embodiment, markers are fixedly installed at least inside or outside the actual object so that the interior and exterior of the object are configured to form continuous data in the three-dimensional data. The distortion identification device 20 acquires images of the exterior and interior of the object from multiple different positions, and generates three-dimensional data using NeRF based on the relative viewpoint positions when each of the multiple images was captured, which are determined based on the size and orientation of the marker when captured from the interior of the object and the size and orientation of the marker when captured from the exterior of the object, to acquire the three-dimensional data. The distortion identification device 20 then identifies the thickness of walls included in the object represented by the three-dimensional data. The distortion identification device 20 outputs information regarding the wall thickness. Therefore, the distortion identification device 20 can reproduce the connection between the interior and exterior, such as wall thickness, in the three-dimensional data and can identify the thickness of walls included in the object represented by the three-dimensional data. Furthermore, if the object is a house, markers may be similarly installed inside the object to connect different rooms in the three-dimensional data, and the viewpoint position of each image relative to the global viewpoint may be determined.

[0076] The distortion identification device 20 also acquires sensor data detected by sensors at multiple different positions on the object. The distortion identification device 20 superimposes information about the detected values ​​at each position of the object, represented by the sensor data, on the object represented by the three-dimensional data. The user can easily grasp the information about the detected values ​​by the sensors.

[0077] The distortion identification device 20 also acquires temperature data measured by an infrared sensor at multiple different positions on the object while a liquid at a predetermined temperature is being passed through a tube included in the object. The distortion identification device 20 then identifies a portion where there is a difference between the position of the tube included in the object in the design data for the object and the position on the object through which the liquid passed, as identified by the temperature data. The distortion identification device 20 then outputs the identified portion as a candidate for distortion in the tube included in the object. The user can confirm the candidate for distortion in the tube included in the object.

[0078] The above is the description of each embodiment. Each embodiment may be modified. Modifications will be described below.

[0079] (Modification) In the second embodiment, an infrared sensor, a depth sensor, and a polarization camera have been described as examples of the sensor 22. For example, when the distortion of an object is measured using a stripe pattern image analysis method, the distortion identification system 1 may include a light projection device and a sensor 22 that receives reflected light in the case of a grid projection method. The sensor data acquisition unit 209 may acquire the results of the stripe pattern image analysis, and the output unit 205 may output the image analysis results. Examples of the output may be similar to those of the sensor 22 described above.

[0080] Furthermore, when an X-ray inspection is performed, the strain identification system 1 may include an X-ray analysis device. The X-ray analysis device irradiates each position on the object with X-rays to create an image of the object. The sensor data acquisition unit 209 may acquire the results of the X-ray inspection, and the output unit 205 may output the results of the X-ray inspection. Examples of the output may be similar to those of the sensor 22 described above.

[0081] This concludes the description of the modified examples. The embodiments and modified examples may be combined as appropriate. There are no particular limitations on how they may be combined.

[0082] In each embodiment, the distortion identification devices 10 and 20 may present information about candidate distortions of an object in various scenes. As described above, the type of object is not particularly limited.

[0083] For example, if the object is a building or the like, the distortion identification devices 10 and 20 may be used as part of a function such as property appraisal. The appraisal value of a property is determined using the building, land, surrounding environment, etc. For example, it is known that whether a building's exterior or interior is tilted or distorted affects the appraisal. For example, the output unit 205 may output information related to property appraisal along with information about potential distortions in the object. The distortion identification devices 10 and 20 can facilitate property appraisal.

[0084] Furthermore, in the case of a building, the distortion identification devices 10 and 20 may be used as part of a function for determining the status of the building during a disaster. Determining the status of a house during a disaster may be used, for example, for disaster relief applications. For example, the output unit 205 may output information related to disaster relief along with information about candidate distortions of the object. The distortion identification devices 10 and 20 can facilitate disaster relief applications.

[0085] Furthermore, if the object is a structure, the distortion identification devices 10 and 20 may be used as part of a function for performing periodic inspections, such as deterioration diagnosis of the structure. The object may also be a moving body or vehicle, such as a vehicle, a ship, an airplane, or a drone, and the distortion identification devices 10 and 20 may be used as part of a function for vehicle inspection, ship inspection, airplane inspection, drone inspection, or the like. The vehicle is not particularly limited to a road vehicle or a railroad car. For example, the output unit 205 may output information related to the inspection along with information about possible distortions in the object. Furthermore, the imaging device may be an inspection camera attached to a warehouse that stores vehicles, ships, airplanes, drones, or the like. The processing by the distortion identification devices 10 and 20 may replace the original inspection for each inspection, or may be used to reduce the frequency of the original inspection, or may be used to reduce the number of original inspections. The distortion identification devices 10 and 20 can facilitate inspections. Alternatively, the distortion identification devices 10 and 20 can simplify inspections.

[0086] The distortion identification devices 10 and 20 may also be configured to include some of the functional units and information.

[0087] Furthermore, the embodiments are not limited to the examples described above and can be modified in various ways. Furthermore, the configuration of the distortion identification devices 10 and 20 is not particularly limited. For example, the functional units of the distortion identification devices 10 and 20 may be implemented by a single device. Alternatively, for example, each functional unit or database of the distortion identification devices 10 and 20 may be implemented by a different device, and configured as a system. For example, each functional unit of the distortion identification devices 10 and 20 may be implemented by a plurality of servers, and configured as a system. For example, a system may be realized that includes a database server including each database and a server having each functional unit. A system may be realized that includes a server having some of the functional units of the distortion identification device and another server having some of the functional units of the distortion identification device. The number of servers is not particularly limited.

[0088] Furthermore, the terminal device 21 may have each functional unit of the distortion identification devices 10 and 20. That is, the terminal device 21 may have installed thereon an application program in which each functional unit of the distortion identification devices 10 and 20 is coded.

[0089] Furthermore, the process of generating information to be displayed on the terminal device 21 may be performed by a functional unit included in the distortion identification devices 10 and 20, such as the output units 105 and 205. This process may also be performed by the terminal device 21. That is, the terminal device 21 may generate information for a screen to be displayed on the terminal device 21 based on data received from the distortion identification devices 10 and 20, and display the screen. Furthermore, the user interface in each embodiment is an example, and various modifications are possible.

[0090] (Example of Computer Hardware Configuration) Next, an example of a hardware configuration in which each device, such as the distortion identification devices 10 and 20 and the terminal device 21, is realized by a computer will be described.

[0091] 10 is an explanatory diagram showing an example of the hardware configuration of a computer. For example, some or all of the devices can be realized using any combination of a computer 80 and a program as shown in FIG.

[0092] The computer 80 includes, for example, a processor 801, a ROM (Read Only Memory) 802, a RAM (Random Access Memory) 803, and a storage device 804. The computer 80 also includes a communication interface 805 and an input / output interface 806. The components are connected to each other, for example, via a bus 807. The number of each component is not particularly limited, and there may be one or more of each component.

[0093] The processor 801 controls the entire computer 80. The processor 801 may be, for example, a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, or a combination thereof, and is not particularly limited.

[0094] The computer 80 also includes a ROM 802, a RAM 803, and a storage device 804. Examples of the storage device 804 include semiconductor memory such as flash memory, a hard disk drive (HDD), and a solid state drive (SSD). For example, the storage device 804 stores an operating system (OS) program, application programs, and programs according to the embodiments. Alternatively, the ROM 802 stores application programs and programs according to the embodiments. The RAM 803 is used as a work area for the processor 801.

[0095] The processor 801 also loads programs stored in the storage device 804, ROM 802, etc. The processor 801 then executes each process coded in the program. The processor 801 may also download various programs via the communication network NT. The processor 801 also functions as a part or all of the computer 80. The processor 801 may then execute the processes or instructions in the illustrated flowchart based on the program.

[0096] The communication interface 805 is connected to a communication network NT such as a LAN (Local Area Network) or a WAN (Wide Area Network) via a wireless or wired communication line. The communication network NT may be composed of multiple communication networks NT. As a result, the computer 80 is connected to external devices and external computers 80 via the communication networks NT. The communication interface 805 serves as an interface between the communication network NT and the inside of the computer 80. The communication interface 805 also controls the input and output of data from external devices and external computers 80.

[0097] Furthermore, the input / output interface 806 is connected to at least one of an input device, an output device, and an input / output device. The connection method may be wireless or wired. Examples of the input device include a keyboard, a mouse, and a microphone. Examples of the output device include a display device, a lighting device, and an audio output device that outputs audio. Examples of the input / output device include a touch panel display. Note that the input device, output device, and input / output device may be built into the computer 80 or may be external.

[0098] The hardware configuration of the computer 80 is an example. The computer 80 may have some of the components shown in FIG. 10 . The computer 80 may have components other than those shown in FIG. 10 . For example, the computer 80 may have a drive device or the like. The processor 801 may then read programs and data stored on a recording medium attached to the drive device or the like into the RAM 803. Examples of non-transitory tangible recording media include optical disks, flexible disks, magneto-optical disks, and USB (Universal Serial Bus) memories. As described above, the computer 80 may have input devices such as a keyboard and a mouse. The computer 80 may have an output device such as a display. The computer 80 may also have an input device, an output device, and an input / output device.

[0099] The computer 80 may also include various sensors (not shown). The types of sensors are not particularly limited. The computer 80 may also include an imaging device capable of capturing images or videos.

[0100] This concludes the description of the hardware configuration of each device. There are various variations in the method of realizing each device. For example, each device may be realized by any combination of a different computer and program for each component. Furthermore, multiple components of each device may be realized by any combination of a single computer and program.

[0101] Furthermore, some or all of the components of each device may be realized by circuits for specific applications. Furthermore, some or all of the components of each device may be realized by general-purpose circuits such as FPGAs (Field Programmable Gate Arrays). Furthermore, some or all of the components of each device may be realized by a combination of circuits for specific applications and general-purpose circuits. These circuits may be a single integrated circuit. Alternatively, these circuits may be divided into multiple integrated circuits. The multiple integrated circuits may be connected via a bus or the like.

[0102] Furthermore, when some or all of the components of each device are realized by a plurality of computers, circuits, etc., the plurality of computers, circuits, etc. may be centrally located or distributed.

[0103] The distortion identification method described in each embodiment may be realized by being executed by a computer such as the distortion identification device 10 or 20.

[0104] Each program described in each embodiment is recorded on a computer-readable recording medium such as a HDD, SSD, flexible disk, optical disk, magneto-optical disk, or USB memory. Each program is executed by being read from the recording medium by a computer. Each program may also be distributed via a communication network NT.

[0105] The functions of each of the components of the distortion identification device 10 and the distortion identification device 20 described above may be realized by dedicated hardware such as a computer. Alternatively, each component may be realized by software. Alternatively, each component may be realized by a combination of hardware and software.

[0106] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. The configuration and details of the present disclosure may include embodiments to which various modifications that would be apparent to those skilled in the art are applied within the scope of the present disclosure. The present disclosure may include embodiments in which the features described herein are appropriately combined or substituted as necessary. For example, features described using a particular embodiment may also be applied to other embodiments to the extent that no contradiction occurs. For example, although multiple operations are described in sequence in the form of a flowchart, the order of description does not limit the order in which the multiple operations are performed. Therefore, when implementing the embodiments, the order of the multiple operations may be changed as long as the content is not affected.

[0107] Some or all of the above-described embodiments can be described as follows: However, some or all of the above-described embodiments are not limited to the following.

[0108] (Supplementary Note 1) A distortion identification device comprising: a three-dimensional data acquisition means for acquiring three-dimensional data representing an object; a distortion candidate identification means for identifying candidates for distortion of the object represented by the three-dimensional data relative to a reference line; and an output means for outputting information regarding the candidates for distortion of the object.

[0109] (Supplementary Note 2) The distortion identification device according to Supplementary Note 1, wherein the distortion candidate identification means identifies an object represented by the three-dimensional data as a distortion candidate if the inclination of the object relative to the reference line is equal to or greater than a predetermined value.

[0110] (Supplementary Note 3) The distortion identifying device according to Supplementary Note 1 or 2, wherein the distortion candidate identifying means identifies, as the distortion candidate, a deviation in the shape of the object represented by the three-dimensional data relative to the reference line.

[0111] (Supplementary Note 4) The distortion identification device according to any one of Supplementary Notes 1 to 3, wherein the three-dimensional data acquisition means acquires the three-dimensional data by generating three-dimensional data representing the object using NeRF based on a plurality of images of the object captured from a plurality of different positions.

[0112] (Supplementary Note 5) The distortion identification device according to any one of Supplementary Notes 1 to 4, wherein the three-dimensional data is three-dimensional data generated using NeRF, and the distortion candidate identification means identifies a portion of the object represented by the three-dimensional data that has a difference in gloss as a candidate for distortion of the object.

[0113] (Supplementary Note 6) The distortion identifying device according to any one of Supplementary Notes 1 to 5, wherein the output means parallel projects the object represented by the three-dimensional data based on the three-dimensional data, and displays the reference line superimposed on the object represented by the three-dimensional data.

[0114] (Supplementary Note 7) The distortion identification device according to any one of Supplementary Notes 1 to 6, wherein the output means displays the identified distortion candidate portion in an identifiable manner in the object represented by the three-dimensional data.

[0115] (Supplementary Note 8) A distortion identification device according to any of Supplementary Notes 1 to 7, comprising: a structure identification means for identifying a thickness of a wall included in an object represented by the three-dimensional data; markers are fixedly installed at least either inside or outside the object; the three-dimensional data acquisition means acquires images of the outside and inside of the object from a plurality of different positions, and acquires the three-dimensional data by generating the three-dimensional data using NeRF based on the plurality of images and a relative viewpoint position when each of the plurality of images was taken, the relative viewpoint position being identified based on the size and orientation of the marker when taken from inside the object and the size and orientation of the marker when taken from outside the object; the structure identification means identifies a thickness of a wall included in the object represented by the three-dimensional data; and the output means outputs information regarding the thickness of the wall.

[0116] (Supplementary Note 9) The distortion identification device according to any one of Supplementary Notes 1 to 7, further comprising: a sensor data acquisition means for acquiring sensor data detected by sensors at a plurality of different positions of the object; and the output means for superimposing and displaying information relating to the detection values ​​at each position of the object, which is represented by the sensor data, on the object represented by the three-dimensional data.

[0117] (Supplementary Note 10) The strain identification device according to Supplementary Note 9, wherein the sensor includes an infrared sensor, the sensor data includes temperature data measured by the infrared sensor at a plurality of different positions on the object, and the information on the detected values ​​at each position on the object is information on the temperature at each position on the object.

[0118] (Supplementary Note 11) The distortion identification device according to Supplementary Note 9 or 10, wherein the sensor includes a depth sensor, the sensor data includes distance data measured by the depth sensor at a plurality of different positions on the object, and the information on the detection values ​​at each position on the object is information on the distance to each position on the object.

[0119] (Supplementary Note 12) A distortion identification device according to any one of Supplementary Notes 1 to 11, comprising: a sensor data acquisition means for acquiring temperature data measured by an infrared sensor at a plurality of different positions on the object while a liquid at a predetermined temperature is passed through a tube included in the object; wherein the distortion candidate identification means identifies a part where there is a difference between a position of the tube included in the object in design data for the object and a position on the object through which the liquid has passed as identified by the temperature data; and the output means outputs the identified part as a candidate for distortion in the tube included in the object.

[0120] (Supplementary Note 13) The distortion identifying device according to any one of Supplementary Notes 1 to 12, wherein the output means projects information relating to candidates for distortion of the object onto the object.

[0121] (Supplementary Note 14) The distortion identifying device according to any one of Supplementary Notes 1 to 13, wherein the output means displays information about candidates for distortion of the object on a display device.

[0122] (Supplementary Note 15) The distortion identifying device according to any one of Supplementary Notes 1 to 14, wherein the output means displays information about candidates for distortion of the object on a display device.

[0123] (Supplementary Note 16) The distortion identifying device according to any one of Supplementary Notes 1 to 15, wherein the object is a building.

[0124] (Supplementary Note 17) The distortion identifying device according to any one of Supplementary Notes 1 to 16, wherein the object is a moving object.

[0125] (Supplementary Note 18) The distortion identifying device according to Supplementary Note 4, wherein the reference line is a reference line obtained from an inertial sensor provided in an imaging device that captures the plurality of images.

[0126] (Supplementary Note 19) A distortion identification method, wherein a computer performs the following processes: acquire three-dimensional data representing an object; identify candidate distortions of the object represented by the three-dimensional data relative to a reference line; and output information about the candidate distortions of the object.

[0127] (Supplementary Note 20) A non-transitory computer-readable recording medium having recorded thereon a program for causing a computer to execute the following processes: acquire three-dimensional data representing an object; identify candidates for distortion of the object represented by the three-dimensional data relative to a reference line; and output information regarding the candidates for distortion of the object.

[0128] (Supplementary Note 21) A program that causes a computer to execute the following processes: acquire three-dimensional data representing an object; identify candidates for distortion of the object represented by the three-dimensional data relative to a reference line; and output information regarding the candidates for distortion of the object.

[0129] Furthermore, some or all of the configurations described in Supplementary Notes 2 to 18, which are dependent on Supplementary Note 1, may also be dependent on Supplementary Notes 19, 20, and 21 in the same dependent relationship as Supplementary Notes 2 to 18. Furthermore, not limited to Supplementary Notes 1, 19, 20, and 21, some or all of the configurations described as Supplements may be made dependent on various hardware, software, various recording means for recording software, or systems, within the scope of each of the above-mentioned embodiments.

[0130] REFERENCE SIGNS LIST 1 Distortion identification system 10, 20 Distortion identification device 21 Terminal device 22 Sensor 80 Computer 101, 201 Three-dimensional data acquisition unit 103, 203 Distortion candidate identification unit 105, 205 Output unit 207 Structure identification unit 209 Sensor data acquisition unit 801 Processor 804 Storage device 805 Communication interface 806 Input / output interface 807 Bus NT Communication network

Claims

1. A distortion identification device comprising: a three-dimensional data acquisition means for acquiring three-dimensional data representing an object; a distortion candidate identification means for identifying candidates for distortion of the object represented by the three-dimensional data relative to a reference line; and an output means for outputting information regarding the candidate distortion of the object.

2. The distortion identification device according to claim 1, wherein said distortion candidate identification means identifies an object represented by said three-dimensional data as a distortion candidate if the inclination of the object relative to said reference line is equal to or greater than a predetermined value.

3. The distortion identifying device according to claim 1 or 2, wherein the distortion candidate identifying means identifies, as the distortion candidate, a deviation in the shape of the object represented by the three-dimensional data relative to the reference line.

4. A distortion identification device according to any one of claims 1 to 3, wherein the three-dimensional data acquisition means acquires the three-dimensional data by generating three-dimensional data representing the object using Neural Radiance Fields (NeRF) based on a plurality of images of the object captured from a plurality of different positions.

5. A distortion identification device according to any one of claims 1 to 4, wherein the three-dimensional data is three-dimensional data generated using NeRF, and the distortion candidate identification means identifies areas of the object represented by the three-dimensional data that have differences in gloss as candidates for distortion of the object.

6. A distortion identification device according to any one of claims 1 to 5, wherein the output means parallel projects the object represented by the three-dimensional data based on the three-dimensional data, and displays the reference line superimposed on the object represented by the three-dimensional data.

7. The distortion identification device according to any one of claims 1 to 6, wherein said output means displays the identified distortion candidate portions in an identifiable manner in the object represented by said three-dimensional data.

8. A distortion identification device as claimed in any one of claims 1 to 7, comprising: a structure identification means for identifying a thickness of a wall included in an object represented by the three-dimensional data; markers are fixedly installed at least either inside or outside the object; the three-dimensional data acquisition means acquires images of the outside and inside of the object from a plurality of different positions, and acquires the three-dimensional data by generating the three-dimensional data using NeRF based on the plurality of images and a relative viewpoint position when each of the plurality of images was taken, the relative viewpoint position being identified based on the size and orientation of the marker when taken from inside the object and the size and orientation of the marker when taken from outside the object; the structure identification means identifies the thickness of a wall included in the object represented by the three-dimensional data; and the output means outputs information regarding the thickness of the wall.

9. A distortion identification device according to any one of claims 1 to 7, comprising: a sensor data acquisition means for acquiring sensor data detected by a sensor at a plurality of different positions on the object; and wherein the output means superimposes information relating to the detected values ​​at each position of the object, represented by the sensor data, on the object represented by the three-dimensional data.

10. The strain identification device according to claim 9, wherein the sensor includes an infrared sensor, the sensor data includes temperature data measured by the infrared sensor at a plurality of different positions on the object, and the information relating to the detected values ​​at each position on the object is information relating to the temperature at each position on the object.

11. The distortion identification device according to claim 9 or 10, wherein the sensor includes a depth sensor, the sensor data includes distance data measured by the depth sensor at a plurality of different positions on the object, and the information relating to the detection values ​​at each position on the object is information relating to the distance to each position on the object.

12. A distortion identification device as described in any one of claims 1 to 11, comprising: a sensor data acquisition means for acquiring temperature data measured by an infrared sensor at a plurality of different positions on the object while a liquid at a predetermined temperature is passed through a tube included in the object; wherein the distortion candidate identification means identifies a portion where there is a difference between the position of the tube included in the object in design data for the object and the position on the object through which the liquid passed as identified by the temperature data; and the output means outputs the identified portion as a candidate for distortion in the tube included in the object.

13. The distortion identifying device according to any one of claims 1 to 12, wherein the output means projects information relating to candidates for distortion of the object onto the object.

14. The distortion identifying device according to any one of claims 1 to 13, wherein the output means displays information about candidates for distortion of the object on a display device.

15. A distortion identifying device according to any one of claims 1 to 14, wherein the output means displays information about candidates for distortion of the object on a display device.

16. The distortion identification device according to any one of claims 1 to 15, wherein the object is a building.

17. The distortion identifying device according to any one of claims 1 to 16, wherein the object is a moving object.

18. The distortion identifying device according to claim 4, wherein the reference line is obtained from an inertial sensor provided in an imaging device that captures the plurality of images.

19. A distortion identification method in which a computer performs the following processing: acquires three-dimensional data representing an object; identifies potential distortions of the object represented by the three-dimensional data relative to a reference line; and outputs information about the potential distortions of the object.

20. A non-transitory computer-readable recording medium having recorded thereon a program for causing a computer to execute the following processes: acquire three-dimensional data representing an object; identify candidates for distortion of the object represented by the three-dimensional data relative to a reference line; and output information regarding the candidate distortion of the object.

Images