Information provision device, information provision method, and program

Abstract

To provide accurate information regarding the deterioration and wear and tear of buildings. [Solution] The server 10 has a calculation unit 132 and a provision unit 133. The calculation unit 132 calculates the repair locations, repair details, and repair costs for parts of a building included in a photographed image of the building. The provision unit 133 provides information for placing labels at the locations corresponding to the repair locations on the image.

Description

Technical Field

[0001] The present invention relates to an information providing apparatus, an information providing method, and a program.

Background Art

[0002] Buildings and facilities installed in buildings deteriorate and wear out over time. For example, in the real estate rental business, it is necessary to determine whether repairs are required due to deterioration and wear of the rented property (such as a room).

[0003] Conventionally, a system for diagnosing a building based on an image of the building taken and the sound generated from the building is known (see, for example, Patent Document 1). Also, a method of taking construction record photos with a 360° camera at a construction site and a real estate property is known (see, for example, Non-Patent Document 1). Further, a technique for calculating the degree of deterioration and wear of a building by comparing photos of the building taken at different times is known (see, for example, Patent Document 2).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of the Invention

[0006] However, conventional technology has the problem that it may not provide accurate information regarding the deterioration and wear and tear of buildings.

[0007] For example, in the technology described in Patent Document 2, the marks indicating damage and other defects and their explanations are displayed in separate locations from each other, making it difficult for users to intuitively grasp information about repairs, and potentially resulting in inaccurate information being conveyed.

[0008] This invention has been made in view of the above, and aims to provide accurate information regarding the deterioration and wear and tear of buildings. [Means for solving the problem]

[0009] To solve the above-mentioned problems and achieve the objective, the information providing device of the present invention is characterized by comprising: a calculation unit that calculates the repair locations, repair details, and repair costs of parts of a building included in a photographed image of the building based on the photographed image of the building; and a providing unit that provides information for placing a label on which text can be entered at a position corresponding to the repair location on the photographed image if the wall or floor of the building is visible in the photographed image. [Effects of the Invention]

[0010] According to the present invention, accurate information regarding the deterioration and wear of buildings can be provided. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 shows an example of the configuration of an information provision system. [Figure 2] Figure 2 shows an example of a server configuration. [Figure 3] Figure 3 shows an example of a property table. [Figure 4] Figure 4 shows an example of a terminal configuration. [Figure 5]FIG. 5 is a diagram for explaining the information provided. [Figure 6] FIG. 6 is a flowchart showing the processing flow of the terminal. [Figure 7] FIG. 7 is a flowchart showing the processing flow of the server. [Figure 8] FIG. 8 is a diagram showing a configuration example of a computer that executes a program.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, an information providing apparatus, an information providing method, and a program according to the present application will be described in detail based on the drawings. Note that the present invention is not limited to the embodiments described below.

[0013] [First Embodiment] The configuration of the information providing system 1 will be described using FIG. 1. FIG. 1 is a diagram showing a configuration example of the information providing system.

[0014] As an example, the information providing system 1 provides information on the deterioration of a building (rental property) rented out as a residence. The information provided by the information providing system 1 is used by real estate management companies, real estate information providers, construction contractors, brokers, landlords, tenants, etc.

[0015] In particular, the information providing system 1 provides information on the repair of buildings and facilities. Buildings and facilities may need to be repaired due to aging deterioration, normal wear and tear, and special wear and tear. Aging deterioration is natural deterioration and wear and tear, etc. Normal wear and tear is wear and tear caused by the normal use of the tenant, etc. Special wear and tear is wear and tear caused by the tenant's intention / negligence, violation of the duty of due care, or other use beyond normal use, etc.

[0016] For example, the information providing system 1 provides, as information, the location to be repaired (repair location), the content of the repair, and the cost required for the repair.

[0017] The information providing system 1 of the present embodiment may have the same configuration as a system including the information processing apparatus described in Patent Document 2. In particular, the information providing system 1 can acquire an image and provide a screen in the same manner as a system including the information processing apparatus described in Patent Document 2. For example, the information providing system 1 is realized by adding functions to a system including the information processing apparatus described in Patent Document 2.

[0018] As shown in FIG. 1, the information providing system 1 includes a server 10 and a terminal 20. The server 10 is connected to the terminal 20 via a network N. The network N is, for example, the Internet. The server 10 is an example of an information providing apparatus.

[0019] The user uses the camera provided in the terminal 20 to photograph the building 50. In the present embodiment, a case where the interior of the building is photographed by the terminal 20 will be described as an example. However, the object photographed by the terminal 20 is not limited to the interior, and may be the exterior of the building or the like.

[0020] The server 10 acquires an image from the terminal 20. The server 10 calculates the locations that need building repair, the repair content, and the repair cost based on the acquired image; the calculation results by the server 10 are provided to the terminal 20 or other apparatuses.

[0021] The configuration of the server 10 will be described with reference to FIG. 2. FIG. 2 is a diagram showing a configuration example of the server. As shown in FIG. 2, the server 10 includes a communication unit 11, a storage unit 12, and a control unit 13.

[0022] The communication unit 11 is an interface for communicating with other apparatuses including the terminal 20. For example, the communication unit 11 is a NIC (Network Interface Card).

[0023] The storage unit 12 is a storage device such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or optical disc. Alternatively, the storage unit 12 may be a rewritable semiconductor memory such as RAM (Random Access Memory), flash memory, or NVSRAM (Non-Volatile Static Random Access Memory). The storage unit 12 stores the OS (Operating System) and various programs executed on the server 10. The storage unit 12 also stores model information 121 and property tables 122.

[0024] Model information 121 is information such as parameters for building a machine learning model. For example, model information 121 may include the weights and biases of a neural network. Model information 121 may also include information for multiple machine learning models with different purposes.

[0025] The property table 122 is a table that stores data related to properties. In this embodiment, each record in the property table 122 corresponds to each room that makes up a multi-unit building (apartment, condominium, etc.). In the following description, each room may be referred to as a property.

[0026] Figure 3 shows an example of a property table. As shown in Figure 3, the property table 122 has columns such as "Property Name", "Address", "Room Number", "Floor Plan", and "Image".

[0027] The "Property Name" column stores the name of the property. The "Address" column stores the address of the property. The "Room Number" column stores the room number of the property. The "Floor Plan" column stores the floor plan of the property. The "Image" column stores information about the property's image (e.g., the path to the image file, or binary data).

[0028] For example, the first row of the property table 122 in Figure 3 shows that the floor plan of "Room 101" in "Apartment X" located at "1-2-3, City B, Prefecture A" is "1LDK". The first row of the property table 122 also stores information about the image of "Room 101".

[0029] Returning to Figure 2, the control unit 13 controls the entire server 10. The control unit 13 is, for example, an electronic circuit such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0030] The control unit 13 has internal memory for storing programs and control data that define various processing procedures, and executes each process using the internal memory. In addition, the control unit 13 functions as various processing units when various programs are run. For example, the control unit 13 has a specific unit 131, a calculation unit 132, and a supply unit 133.

[0031] The identification unit 131 identifies the location where the image was taken. For example, the identification unit 131 identifies the location where the image was taken based on the orientation and other information acquired along with the image.

[0032] The identification unit 131 can identify the shooting location using a machine learning model. Alternatively, the identification unit 131 may identify the shooting location by template matching using images stored in the property table 122 as templates.

[0033] The calculation unit 132 calculates the location requiring repair, the repair content, and the repair cost based on the image. The calculation unit 132 uses a machine learning model that takes the image as input and outputs the location requiring repair, the repair content, and the repair cost. The calculation unit 132 may also perform the calculation by referring to the shooting location identified by the identification unit 131.

[0034] The providing unit 133 provides the calculation results from the calculation unit 132 to the terminal 20 or another device. For example, the providing unit 133 causes the terminal 20 to display a screen showing the calculation results.

[0035] The configuration of terminal 20 will be explained using Figure 4. Figure 4 is a diagram showing an example of terminal configuration. As shown in Figure 4, terminal 20 has a communication unit 21, a sensor unit 22, a display unit 23, a storage unit 24, and a control unit 25. Terminal 20 can be a smartphone, a tablet computer, a personal computer, etc.

[0036] The communication unit 21 is an interface for communicating with other devices, including the terminal 20. For example, the communication unit 21 is a network interface card (NIC).

[0037] The sensor unit 22 consists of multiple sensors. The camera 221 captures images. The depth sensor 222 measures the distance to an object using infrared light or the like. For example, the depth sensor 222 measures the distance to an object that is in the same direction as the camera 221's shooting direction. The compass sensor 223 detects the Earth's magnetic field and measures the orientation of the terminal 20. For example, the compass sensor 223 outputs the camera 221's shooting direction as a 3-axis orientation (e.g., east-west axis, north-south axis, up-down axis).

[0038] Furthermore, the control unit 25 may generate images such as 3D images, panoramic images, and 360-degree images based on the measurement results from the sensor unit 22.

[0039] The display unit 23 is a display that shows images. The display unit 23 may also be a touch panel display.

[0040] The storage unit 24 is a storage device such as an HDD, SSD, or optical disc. Alternatively, the storage unit 24 may be a rewritable semiconductor memory such as RAM, flash memory, or NVSRAM. The storage unit 24 stores the OS and various programs executed on the terminal 20.

[0041] The control unit 25 controls the entire terminal 20. The control unit 25 is, for example, an electronic circuit such as a CPU, MPU, or GPU, or an integrated circuit such as an ASIC or FPGA.

[0042] The control unit 25 has internal memory for storing programs and control data that define various processing procedures, and executes each process using the internal memory. In addition, the control unit 25 functions as various processing units when various programs are run. For example, the control unit 25 has an acquisition unit 251, a supply unit 252, and a display control unit 253.

[0043] The acquisition unit 251 acquires measurement results from the sensor unit 22. The acquisition unit 251 acquires images (captured images) taken by the camera 221, depth measured by the depth sensor 222, and direction measured by the direction sensor 223.

[0044] The providing unit 252 provides the server 10 with the information acquired by the acquisition unit 251. For example, the providing unit 252 provides images, depth, and orientation.

[0045] The display control unit 253 causes the display unit 23 to display a screen. The display control unit 253 displays the screen based on data received from the server 10.

[0046] Figure 5 illustrates the details of the process by which server 10 provides information. Figure 5 is a diagram illustrating the information provided. Screen 70 in Figure 5 is displayed on terminal 20 based on the information provided by server 10.

[0047] As shown in Figure 5, the screen 70 includes areas 71, 72, 73, 74, and 75. For example, the screen 70 may be generated by the providing unit 133. The providing unit 133 places images, objects, and text on the screen.

[0048] Area 71 displays the name and address of the property (in this case, an apartment building including rooms). Area 72 displays the room number, which can be selected. Area 73 displays the room floor plan 731, a compass object 732, and a three-axis object 733. For example, the information displayed in areas 71, 72, and 73 is provided by the provisioning unit 133 in response to a specification from the terminal 20, by referring to the property table 122, etc., stored in the storage unit 12.

[0049] The compass object 732 corresponds to the floor plan 731 and represents a direction. The top of the compass object 732 points north. That is, the direction from bottom to top on the floor plan 731 is north, and the direction from top to bottom on the floor plan 731 is south.

[0050] The 3-axis object 733 is an object that indicates the orientation of three axes to represent the location of a room. Here, the x-axis is the axis from east to west, the y-axis is the axis from south to north, and the z-axis is the vertically upward axis.

[0051] Furthermore, a shooting location object 734 is displayed in the floor plan 731. The shooting location object 734 indicates the shooting location and direction of the image, as identified by the identification unit 131. For example, the shooting location object 734 indicates that the image was taken facing south from near the center of the room.

[0052] Area 74 displays an image to which information has been added by terminal 20. Area 74 also displays a three-axis object 741. The three-axis object 741 corresponds to the three-axis object 733 in the floor plan 731. For example, in the three-axis object 741, the x-axis is the axis from east to west, the y-axis is the axis from south to north, and the z-axis is the vertically upward axis.

[0053] Labels 742, 743, and 744 are attached to the areas requiring repair, as calculated by the calculation unit 132. Labels 742, 743, and 744 also display the repair details and costs in text or other format, corresponding to the areas requiring repair.

[0054] For example, label 742 is attached to a windowpane that needs repair. Label 742 indicates that the repair required is "South-facing living room windowpane: Cleaning required due to stains," and the repair cost is "¥4,000."

[0055] Furthermore, area 75 displays a list of the repair details and repair costs. Each item in area 75 corresponds to a label.

[0056] Here, the calculation process performed by the calculation unit 132 will be described. The calculation unit 132 inputs the necessary data into the trained machine learning model constructed based on the model information 121 and obtains the output. The calculation unit 132 may also perform further calculations based on the output of the machine learning model.

[0057] The calculation unit 132 inputs the image to the repair information calculation model. The repair information calculation model is a model that outputs the repair locations, repair details, and repair costs based on the image. The repair information calculation model may be based on a model used for image recognition, such as a convolutional neural network.

[0058] For example, a restoration information calculation model is trained using data that combines an image with information about the areas to be restored, the details of the restoration, and the cost of the restoration. For instance, the restoration information calculation model detects discoloration, scratches, stains, etc., on the walls, floors, ceilings, openings, and fixtures of a room in an image, and identifies them as rectangular regions. The restoration information calculation model then calculates the restoration method and cost for each rectangular region.

[0059] Furthermore, if there are images taken from the same position and orientation as the captured image, Server 10 can improve the accuracy of calculating the repair locations, repair details, and repair costs by comparing the images. Server 10 may also calculate the repair locations, repair details, and repair costs based on the rate of change obtained by comparing an image taken at a first date and time (for example, before the tenant moves in) with an image taken at a second date and time (for example, when the tenant moves out), using a method similar to that described in Patent Document 2.

[0060] The information provider 133 provides information for placing labels at positions corresponding to the repair areas on the image. This allows the information provider 133 to display the labeled image on the terminal 20 or the like. For example, the information provider 133 provides the position corresponding to the repair area on the image, along with text indicating the repair details and repair costs to be displayed on the label.

[0061] Furthermore, the providing unit 133 can provide the position corresponding to the repair location as coordinates in the coordinate space shown in the 3-axis object 733 and the 3-axis object 741. Also, the label is an object for displaying text. The shape of the label is not limited to a callout format as shown in Figure 5, but may be an object that displays only text.

[0062] This makes it possible to display labels near the repair location, as shown in Figure 5. As a result, users can more easily understand the repair location, the repair details, and the repair cost intuitively.

[0063] The display control unit 253 of terminal 20 places a label indicating the detected rectangular area on the captured image based on the information provided by the supply unit 133. Then, based on the information provided by the supply unit 133, the display control unit 253 writes the repair details and repair costs on the label.

[0064] The providing unit 133 provides information indicating that text can be entered on the label, along with the location corresponding to the repair area on the image. In this case, the display control unit 253, based on the information provided by the providing unit 133, leaves the label in a state where text can be entered, without including the repair details and repair costs. For example, the providing unit 133 places a label where text can be entered when the captured image only shows a wall or floor and the detection accuracy is low.

[0065] Figure 6 is a flowchart showing the processing flow of the terminal. First, as shown in Figure 6, terminal 20 acquires an image of the room taken by camera 221 (step S101). Terminal 20 also acquires its orientation from orientation sensor 223 (step S102).

[0066] Next, terminal 20 transmits the acquired image and orientation to server 10 (step S103).

[0067] Subsequently, terminal 20 receives information from server 10 regarding the location requiring repair, the repair details, and the cost (step S104). Based on the information received from server 10, terminal 20 displays the image on its screen (step S105) and displays the repair details and cost at the corresponding location on the image (step S106).

[0068] Figure 7 is a flowchart showing the server's processing flow. First, as shown in Figure 7, the server 10 receives an image and orientation from the terminal 20 (step S201).

[0069] Next, server 10 identifies the shooting location based on the image and orientation (step S202). Then, server 10 calculates the location requiring repair, the repair details, and the cost based on the image (step S203). Server 10 can perform calculations using a machine learning model.

[0070] Server 10 sends the calculation result to terminal 20 (step S204). Server 10 may send the calculation result as is, or it may send the data of a screen created based on the calculation result.

[0071] [System configuration, etc.] Each component of the illustrated device is a functional concept and does not necessarily have to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed or integrated in any unit according to various loads and usage conditions. Furthermore, each processing function performed by each device can be implemented, all or any part of it, by a CPU and a program that is analyzed and executed by that CPU, or by hardware using wired logic. Note that the program may be executed not only by the CPU but also by other processors such as a GPU.

[0072] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.

[0073] [program] In one embodiment, the server 10 can be implemented by installing a program that performs the above processing as packaged software or online software on a desired computer. For example, by having the information processing device execute the above program, the information processing device can function as the server 10. The information processing device referred to here includes desktop or notebook personal computers. In addition, the category of information processing device also includes tablet terminals, smartphones, mobile communication terminals such as mobile phones and PHS (Personal Handyphone System), and slate terminals such as PDA (Personal Digital Assistant).

[0074] Furthermore, server 10 may be implemented as a web server, or it may be implemented as a cloud service that provides the above processing services through outsourcing.

[0075] Figure 8 shows an example configuration of a computer running a program. Computer 1000 has, for example, memory 1010 and a CPU 1020. Computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0076] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM (Random Access Memory) 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0077] The hard disk drive 1090 stores, for example, the OS 1091, application programs 1092, program modules 1093, and program data 1094. That is, the programs that define each process of the server 10 are implemented as program modules 1093 in which executable code for the computer is written. The program modules 1093 are stored, for example, on the hard disk drive 1090. For example, a program module 1093 for performing the same processes as the functional configuration of the server 10 is stored on the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0078] Furthermore, the configuration data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes the processing of the above-described embodiment.

[0079] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070. [Explanation of symbols]

[0080] N Network 10 servers 11, 21 Communications Department 12, 24 Storage section 13, 25 Control Unit 20 devices 22 Recovery Unit 23 Display section 70 screens 71, 72, 73, 74, 75 area 122 Property Table 221 Camera 222 Depth Sensor 223 Directional sensor 251 Acquisition Department 252 Provision Department 253 Display Control Unit 732 Directional Objects 733 3-axis object 734 Shooting location object 741 3-axis object Labels 742, 743, 744

Claims

1. A calculation unit that calculates the repair locations, repair details, and repair costs for parts of the building included in the photographed images of the building, based on the photographed images of the building. If the aforementioned captured image shows the wall or floor of a building, the providing unit provides information for placing a label on which text can be entered at the position corresponding to the repair area on the aforementioned captured image. An information providing device characterized by having the following features.

2. The providing unit provides the location on the captured image corresponding to the repair area, along with the text of the repair details and repair costs to be displayed on the label. The information providing device according to feature 1.

3. The providing unit provides information indicating that text can be entered into the label, along with the position corresponding to the repair area on the captured image. The information providing device according to feature 1.

4. An information provision method performed by an information provision device, A calculation process that calculates the repair locations, repair details, and repair costs for parts of the building included in the photographed images, based on the images of the building. If the aforementioned captured image shows the wall or floor of a building, the process of providing information for placing a label on which text can be entered at the position corresponding to the repair area on the aforementioned captured image, A method of providing information characterized by including the following.

5. A calculation step that calculates the repair locations, repair details, and repair costs for the parts of the building included in the photographed images of the building, based on the photographed images of the building. If the aforementioned captured image shows the wall or floor of a building, the providing step provides information for placing a label on which text can be entered at the position corresponding to the repair area on the aforementioned captured image. A program characterized by causing a computer to execute something.

Images