Display method, program, and terminal device.

Abstract

This invention provides a display method that can assist users in capturing the data necessary to create ceiling plan data. [Solution] The display method performed by the terminal device 50, which includes an imaging unit 56 and a display unit 55, includes a first display step in which the imaging unit 56 captures RGB data and depth data targeting the ceiling surface of an indoor space used to create a ceiling plan of the indoor space, and displays at least one of the RGB data and depth data being captured by the imaging unit 56 on the display unit 55, and a second display step in which guide information to assist in capturing the RGB data and depth data is superimposed on at least one of the RGB data and depth data being captured.

Description

Technical Field

[0001] The present disclosure relates to a display method, a program, and a terminal device.

Background Art

[0002] Patent Document 1 discloses a system that assists a user in identifying the part number of a lighting fixture based on an image obtained by imaging the lighting fixture.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] During construction work for replacing lighting equipment or air conditioning equipment installed on a ceiling, a construction worker performs estimation work. Generally, in the estimation work, the construction worker performs work to create a ceiling layout drawing to be presented to the customer, and the work of creating this ceiling layout drawing incurs costs. The ceiling layout drawing is a drawing including information indicating the types of equipment installed on the ceiling.

[0005] A technique for estimating the type of equipment based on an image of the equipment, as disclosed in Patent Document 1, can assist in the creation of a ceiling layout drawing.

[0006] For example, it is conceivable to create a ceiling layout drawing based on an image obtained by imaging the ceiling. In this case, in order to create the ceiling layout drawing with higher accuracy, it is important for the user to appropriately capture the image of the ceiling.

[0007] An object of the present disclosure is to provide a display method and the like that can assist a user in capturing data necessary for creating ceiling layout drawing data.

Means for Solving the Problems

[0008] One embodiment of the display method relating to this disclosure is a display method performed by a terminal device comprising an imaging unit and a display unit, comprising: a first display step in which, during the operation in which the imaging unit images RGB data and Depth data targeting the ceiling surface of the interior space used to create a ceiling plan of the interior space, the imaging unit displays at least one of the RGB data and Depth data being imaged on the display unit; and a second display step in which guide information to support the imaging of the RGB data and Depth data is superimposed on at least one of the RGB data and Depth data being imaged and displayed on the display unit.

[0009] Furthermore, one embodiment of the terminal device according to the present disclosure comprises a display unit, an imaging unit that captures RGB data and depth data targeting the ceiling surface of the indoor space used to create a ceiling plan of the indoor space, and an information processing unit that causes the imaging unit to display at least one of the RGB data and depth data being captured on the display unit, wherein the information processing unit displays guide information to support the capture of the RGB data and depth data by superimposing it on at least one of the RGB data and depth data being captured.

[0010] Furthermore, this disclosure can be implemented not only as the above-described display method, but also as a program that causes a computer to execute the above-described display method. Moreover, it can also be implemented as a computer-readable recording medium storing that program. [Effects of the Invention]

[0011] The display methods and other related information disclosed herein can assist users in capturing the data necessary to create ceiling plan data. [Brief explanation of the drawing]

[0012] [Figure 1] Figure 1 is a block diagram showing the functional configuration of a ceiling plan creation system according to an embodiment. [Figure 2] Figure 2 is a sequence diagram illustrating the operation by which the ceiling plan creation system according to the embodiment creates ceiling plan data. [Figure 3] Figure 3 shows an example of ceiling plan data according to the embodiment. [Figure 4] Figure 4 is a flowchart illustrating the first display operation of guide information during imaging of the terminal device according to the embodiment. [Figure 5] Figure 5 shows an example of the display of warning guide information according to the embodiment. [Figure 6] Figure 6 shows another example of the display of warning guide information according to the embodiment. [Figure 7] Figure 7 is a flowchart illustrating the second display operation of guide information during imaging by the terminal device according to the embodiment. [Figure 8] Figure 8 shows an example of the display of area guide information according to the embodiment. [Modes for carrying out the invention]

[0013] (Embodiment) The following describes in detail, with reference to the drawings, embodiments of the display method, program, and terminal device relating to this disclosure. The embodiments described below are all preferred examples of this disclosure. The numerical values, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples and are not intended to limit this disclosure. Furthermore, components in the following embodiments that are not described in an independent claim are described as any component constituting a preferred configuration.

[0014] Please note that each figure is a schematic diagram and not necessarily a strictly accurate representation. Furthermore, in each figure, substantially identical components are denoted by the same reference numerals, and redundant explanations may be omitted or simplified.

[0015] Further, in the present disclosure, ordinal numbers such as "first" and "second" do not indicate the number or order of constituent elements or the like unless otherwise specified, and are used for the purpose of avoiding confusion and distinguishing constituent elements or the like of the same type.

[0016] [Background Leading to the Conception of the Present Disclosure] When performing work to replace lighting equipment, air-conditioning equipment, or the like in a tenant space, an office, or the like, estimation work is performed in advance. For example, estimation work when performing work to change fluorescent lamps to LEDs (Light Emitting Diodes) can be considered.

[0017] In such estimation work, it is usual for site representatives, site supervisors, and the like of electrical contractors who perform construction work handling wiring for lighting and electrical outlets to actually visit the site and conduct field surveys.

[0018] In the field survey, the site supervisor visually recognizes the type, position, and number of devices arranged on the ceiling, and takes handwritten notes. Thereafter, the estimation work is performed by creating a ceiling layout drawing based on these notes, and a construction proposal is made to the customer (for example, the owner of the tenant space or the office). The ceiling layout drawing is a drawing including information indicating the type, position, and number of devices arranged on the ceiling.

[0019] The work of creating this ceiling layout drawing is a high-load work for site supervisors. This is because the number of work sites is large, and the work becomes useless if the construction order is lost.

[0020] Accordingly, the inventor has conceived a system that supports the work of creating such a ceiling layout drawing.

[0021] [Configuration] First, the configuration of a ceiling layout drawing creation system that executes a device estimation method according to an embodiment will be described. Figure 1 is a block diagram showing the functional configuration of the ceiling layout drawing creation system 10 according to the embodiment.

[0022] The ceiling plan creation system 10 is a system for creating ceiling plan data for an indoor space. An indoor space is, for example, an office or a tenant space. In this specification, "space" means an indoor space. A user of the ceiling plan creation system 10 is, for example, a contractor who performs work related to equipment installed on the ceiling of an office or tenant space.

[0023] The ceiling plan creation system 10 comprises a ceiling plan creation device 20 and a terminal device 50. The ceiling plan creation device 20 and the terminal device 50 are connected to each other so as to be able to communicate via a public network 60.

[0024] The user goes to the indoor space where the ceiling plan data is created, carrying the terminal device 50. By operating the terminal device 50, the user can generate the information necessary for creating the ceiling plan data and verify the ceiling plan data generated by the ceiling plan creation device 20.

[0025] The ceiling plan creation device 20 is a device that creates ceiling plan data. The ceiling plan data is a drawing of the ceiling of an interior space. Specifically, the ceiling plan data includes information showing the outline of the ceiling of the interior space. The ceiling plan data also includes information showing the location and type of equipment installed on the ceiling of the interior space. The ceiling plan creation device 20 comprises a drawing creation device 30 and an equipment estimation device 40. The ceiling plan creation device 20 may be implemented by two server devices, one server device that implements the functions of the drawing creation device 30 and the other server device that implements the functions of the equipment estimation device 40, or it may be implemented by one server device that implements the functions of both the drawing creation device 30 and the equipment estimation device 40.

[0026] The drawing creation device 30 is a device that creates ceiling drawing data and ceiling plan data. The drawing creation device 30 is implemented, for example, by a cloud server. The drawing creation device 30 comprises a communication unit 31, an information processing unit 32, and a storage unit 33.

[0027] The communication unit 31 is a communication circuit (communication module) for the drawing creation device 30 to communicate with the equipment estimation device 40 and the terminal device 50 via the public network 60. The communication performed by the communication unit 31 is, for example, wireless communication. The communication unit 31 may communicate wirelessly with the terminal device 50 and with the equipment estimation device 40 via wired communication. The communication standard for the communication performed by the communication unit 31 is not particularly limited.

[0028] The information processing unit 32 performs information processing for the drawing creation device 30 to create ceiling drawing data and ceiling plan data. The information processing unit 32 is implemented by, for example, a microcomputer, but may also be implemented by a processor or dedicated circuit. The functions of the information processing unit 32 are realized by the execution of a computer program (software) stored in the storage unit 33 by the hardware, such as a microcomputer or processor, that constitutes the information processing unit 32. Specifically, the information processing unit 32 includes an acquisition unit 321, a creation unit 322, and an output unit 323.

[0029] The acquisition unit 321 acquires RGB data and Depth data of the indoor space from the terminal device 50. The RGB data is an RGB image captured with respect to the ceiling surface of the indoor space. The RGB data shows multiple pieces of equipment installed on the ceiling surface of the indoor space. The Depth data is data that includes distance values ​​to the ceiling surface, walls, and equipment of the indoor space. The Depth data may also be three-dimensional point cloud data showing the outline of the indoor space, generated by converting the distance values ​​to the ceiling surface, walls, and equipment of the indoor space into point cloud data.

[0030] The creation unit 322 creates drawing data of the ceiling of the interior space based on the RGB data and depth data of the interior space acquired by the acquisition unit 321. The ceiling drawing data is data that shows the outer shape of the ceiling of the interior space, and the location and number of equipment installed on the ceiling of the interior space.

[0031] Furthermore, the creation unit 322 creates ceiling plan data based on the ceiling drawing data and the equipment estimation results created by the equipment estimation device 40. The ceiling plan data is data that associates the location and type of equipment installed on the ceiling of the room space.

[0032] The output unit 323 outputs the ceiling plan data created by the creation unit 322.

[0033] The memory unit 33 is a storage device that stores information necessary for information processing, such as computer programs executed by the information processing unit 32. The memory unit 33 is implemented, for example, by semiconductor memory. The computer programs stored in the memory unit 33 include a program for creating ceiling drawing data based on RGB data and depth data of the interior space.

[0034] The equipment estimation device 40 is a device that estimates the type of equipment installed on the ceiling. The equipment estimation device 40 is implemented, for example, by a cloud server. The equipment estimation device 40 comprises a communication unit 41, an information processing unit 42, and a storage unit 43. The type of equipment is, for example, if the equipment is lighting equipment, a classification based on the shape of the equipment, such as downlights or base lights. The type of equipment is, for example, if the equipment is air conditioning equipment, exhaust vents and intake vents, etc.

[0035] The communication unit 41 is a communication circuit (communication module) for the equipment estimation device 40 to communicate with the drawing creation device 30 and the terminal device 50 via the public network 60. The communication performed by the communication unit 41 is, for example, wireless communication. The communication unit 41 may communicate wirelessly with the terminal device 50 and with the drawing creation device 30 via wired communication. The communication standard for the communication performed by the communication unit 41 is not particularly limited.

[0036] The information processing unit 42 performs information processing for the equipment estimation device 40 to estimate the type of equipment. The information processing unit 42 is implemented, for example, by a microcomputer, but may also be implemented by a processor or dedicated circuit. The functions of the information processing unit 42 are realized by the execution of a computer program (software) stored in the storage unit 43 by the hardware, such as a microcomputer or processor, that constitutes the information processing unit 42. Specifically, the information processing unit 42 includes an acquisition unit 421, an estimation unit 422, a correction unit 423, an output unit 424, and a learning unit 425.

[0037] The acquisition unit 421 acquires RGB data of the indoor space. For example, the acquisition unit 421 acquires RGB data of the indoor space captured by the terminal device 50.

[0038] The estimation unit 422 estimates the type of equipment included in the RGB data based on the RGB data of the indoor space acquired by the acquisition unit 421. The estimation unit 422 estimates the type of equipment using the trained equipment estimation model 100 stored in the storage unit 43.

[0039] The correction unit 423 corrects the type of device estimated by the estimation unit 422. The correction unit 423 corrects the type of device using the learned device estimation model 100 stored in the memory unit 43. In this embodiment, it is not essential that the device estimation device 40 includes the correction unit 423.

[0040] The output unit 424 outputs at least one of the results of the estimation of the equipment type by the estimation unit 422 and the results of the correction of the equipment type by the correction unit 423.

[0041] The learning unit 425 trains the device estimation model 100.

[0042] The memory unit 43 is a storage device that stores information necessary for information processing, such as computer programs executed by the information processing unit 42. The memory unit 43 is implemented, for example, by semiconductor memory. The computer program stored in the memory unit 43 includes a device estimation model 100 for estimating the type of device.

[0043] The device estimation model 100 is a pre-trained AI (Artificial Intelligence) model. The device estimation model 100 is pre-trained by the learning unit 425.

[0044] The method for creating the equipment estimation model 100 is, for example, supervised learning using a training dataset. Specifically, for example, the person setting up the system prepares training data, which is RGB data (image data) captured on the ceiling surface of a room, associating the area in which equipment installed on the ceiling is visible with the type of equipment. The learning unit 425 then uses this training data to machine-learn the equipment estimation model 100. The equipment estimation model 100 is trained to output the area in which equipment is visible and the type of equipment when RGB data captured on the ceiling surface of a room is input. The equipment estimation model 100 trained by the learning unit 425 is stored in the storage unit 43.

[0045] The terminal device 50 is an information terminal used by the user. The user can use the terminal device 50 to check the ceiling plan data of the room space. The terminal device 50 is implemented, for example, by a dedicated terminal of the ceiling plan creation system 10. The terminal device 50 may also be implemented by a general-purpose terminal such as a smartphone or tablet terminal. Specifically, the terminal device 50 comprises a communication unit 51, an information processing unit 52, a storage unit 53, an operation reception unit 54, a display unit 55, and an imaging unit 56.

[0046] The communication unit 51 is a communication circuit (communication module) for the terminal device 50 to communicate with the ceiling plan creation device 20 via a public network 60 such as the Internet. The communication performed by the communication unit 51 is, for example, wireless communication. The communication standard for the communication performed by the communication unit 51 is not particularly limited.

[0047] The information processing unit 52 performs information processing to allow the user to acquire ceiling plan data using the terminal device 50. The information processing unit 52 is implemented by, for example, a microcomputer, but may also be implemented by a processor or dedicated circuit. The functions of the information processing unit 52 are realized by the execution of a computer program (software) stored in the storage unit 53 by the hardware, such as the microcomputer or processor that constitutes the information processing unit 52. Specifically, the information processing unit 52 includes an acquisition unit 521, a control unit 522, a determination unit 523, and an output unit 524.

[0048] The acquisition unit 521 acquires the ceiling plan data created by the ceiling plan creation device 20.

[0049] The control unit 522 displays the ceiling plan data acquired by the acquisition unit 521 on the display unit 55. The control unit 522 also displays guide information on the display unit 55 to assist the user in capturing RGB data and depth data targeting the ceiling surface of the room.

[0050] The determination unit 523 determines whether the movement of the terminal device 50 meets predetermined conditions. The determination unit 523 also determines whether the imaging range meets predetermined conditions.

[0051] The output unit 524 outputs the RGB data and depth data captured by the imaging unit 56.

[0052] The memory unit 53 is a storage device that stores computer programs and the like executed by the information processing unit 52. The computer programs stored in the memory unit 53 include a predetermined application program for presenting ceiling plan data to the user. The memory unit 53 is implemented, for example, by semiconductor memory.

[0053] The operation reception unit 54 receives user input. The operation reception unit 54 is implemented, for example, by a touch panel.

[0054] The display unit 55 is a display for presenting images to the user, for example, by presenting the user with an image of the ceiling plan data created by the ceiling plan creation device 20. The image displayed on the display unit 55 includes an image of at least one of the RGB data and depth data being captured by the imaging unit 56, and an image of guide information to assist in the capture of the RGB data and depth data. The display unit 55 is implemented by a display panel such as a liquid crystal panel or an organic EL (Electronic Luminescence) panel.

[0055] The imaging unit 56 captures an image of the indoor space. The user uses the imaging unit 56 of the terminal device 50 to perform imaging on the ceiling surface of the indoor space. Specifically, the imaging unit 56 captures RGB data and depth data of the space. The imaging unit 56 is implemented, for example, by a camera capable of capturing visible light images and LiDAR (Light Detection And Ranging).

[0056] The detection unit 57 detects the movement of the terminal device 50. The movement of the terminal device 50 may be, for example, movement of the terminal device 50, or movement of the angle that the terminal device 50 makes with respect to the horizontal plane may be a change. The detection unit 57 is realized by a sensor built into the terminal device 50. The detection unit 57 may be realized by, for example, an acceleration sensor and a gyroscope sensor. The detection unit 57 may also be realized by an IMU (Inertial Measurement Unit) sensor.

[0057] [Action to create ceiling diagram data] The operation of the ceiling plan creation device 20 according to the embodiment in which ceiling plan data is created will be described. Figure 2 is a sequence diagram illustrating the operation of the ceiling plan creation system 10 according to the embodiment in which ceiling plan data is created.

[0058] First, the terminal device 50 captures RGB data of the space (S101). Specifically, the user uses the imaging unit 56 of the terminal device 50 to capture RGB data of the ceiling surface of the room.

[0059] Next, the terminal device 50 captures depth data of the space (S102). The user uses the imaging unit 56 of the terminal device 50 to capture depth data targeting the ceiling surface of the room space. The processing in steps S101 and S102 will be described in detail later.

[0060] When spatial RGB data and depth data are captured, the output unit 524 of the terminal device 50 transmits the spatial RGB data and depth data to the drawing creation device 30 via the public network 60 using the communication unit 51 (S103). The spatial RGB data and depth data transmitted by the terminal device 50 are received by the communication unit 31 of the drawing creation device 30.

[0061] The acquisition unit 321 of the drawing creation device 30 acquires spatial RGB data and depth data (S104).

[0062] Once RGB data and depth data are acquired, the creation unit 322 of the drawing creation device 30 creates drawing data for the ceiling (S105). Specifically, the drawing data for the ceiling is drawing data that shows the outer shape of the ceiling of the room space, and the location and number of equipment installed on the ceiling of the room space.

[0063] Furthermore, the output unit 524 of the terminal device 50 transmits spatial RGB data to the device estimation device 40 via the public network 60 using the communication unit 51 (S106). The spatial RGB data transmitted by the terminal device 50 is received by the communication unit 41 of the device estimation device 40.

[0064] The acquisition unit 421 of the equipment estimation device 40 acquires spatial RGB data (S107).

[0065] The information processing unit 42 of the equipment estimation device 40 estimates the type of equipment installed on the ceiling of the room space based on the acquired RGB data of the space (S108). For example, the estimation unit 422 identifies the area of ​​equipment in the RGB data and estimates the type of equipment reflected in the RGB data using the trained equipment estimation model 100. In other words, the estimation unit 422 estimates the type of each of the one or more equipment whose areas have been identified.

[0066] Next, the information processing unit 42 of the equipment estimation device 40 transmits the equipment estimation result to the drawing creation device 30 using the communication unit 41 (S109). The equipment estimation result includes information indicating the type of equipment estimated by the equipment estimation device 40 and information indicating the position of the equipment in RGB data. The equipment estimation result transmitted by the equipment estimation device 40 is received by the drawing creation device 30.

[0067] Upon receiving the equipment estimation results, the creation unit 322 of the drawing creation device 30 creates ceiling plan data (S110). Specifically, the drawing creation device 30 creates ceiling plan data by applying the received equipment estimation results to the ceiling drawing data created in step S105. In other words, the creation unit 322 associates the equipment estimation results with the equipment whose positions are indicated in the ceiling drawing data. The ceiling plan data is data that associates the positions and types of equipment installed on the ceiling of an indoor space. Figure 3 shows an example of ceiling plan data 200 according to an embodiment.

[0068] As shown in Figure 3, the ceiling plan data 200 indicates that the ceiling is a rectangle with dimensions of Xm × Ym. It also indicates that nine pieces of equipment are installed on the ceiling, eight of which are base lights and one of which is an air conditioning unit. In other words, the ceiling plan data 200 shown in Figure 3 indicates the location and type of equipment installed on the ceiling of the interior space.

[0069] Once the ceiling plan data is created, the output unit 323 of the drawing creation device 30 outputs the ceiling plan data (S111). The output unit 323 of the drawing creation device 30 transmits the ceiling plan data to the terminal device 50 using the communication unit 31 (S112). The ceiling plan data transmitted by the drawing creation device 30 is received by the terminal device 50. The acquisition unit 521 of the terminal device 50 acquires the ceiling plan data using the communication unit 51.

[0070] Upon receiving the ceiling plan data, the terminal device 50 presents the ceiling plan data to the user (S113). For example, the control unit 522 presents the ceiling plan data to the user by displaying it on the display unit 55.

[0071] Thus, the ceiling plan creation system 10 according to this embodiment can create and present ceiling plan data of an indoor space based on RGB data and depth data captured of the indoor space by the user.

[0072] This ceiling plan creation system 10 reduces the effort required for users to draft ceiling plans.

[0073] [First display operation of guide information] As described above, the ceiling plan creation device 20 creates ceiling plan data of an indoor space based on RGB data and depth data captured of the indoor space by the user. At this time, by using appropriately captured RGB data and depth data, the accuracy of the ceiling plan data created by the ceiling plan creation device 20 can be improved.

[0074] The terminal device 50 according to this embodiment displays guide information to assist the user when capturing RGB data and depth data.

[0075] Specifically, the terminal device 50 according to the embodiment displays warning guide information indicating an abnormality in the imaging of RGB data and depth data when the movement of the terminal device 50 satisfies predetermined conditions. Such a terminal device 50 can help the user to more appropriately image RGB data and depth data.

[0076] The following describes the first display operation in which such a terminal device 50 displays warning guide information. Figure 4 is a flowchart illustrating the first display operation of guide information during imaging by the terminal device according to the embodiment.

[0077] First, the imaging unit 56 of the terminal device 50 starts imaging the ceiling surface (S201). In other words, the user starts imaging RGB data and depth data of the ceiling surface of the room using the imaging unit 56 of the terminal device 50. For example, the user moves around the room with the terminal device 50 and images RGB data and depth data of the ceiling surface of the room. This imaging may be performed by capturing still images multiple times, or by capturing moving images.

[0078] At this time, the control unit 522 causes the display unit 55 to display at least one of the RGB data and Depth data being captured by the imaging unit 56. This allows the user to perform the imaging while checking at least one of the RGB data and Depth data being captured.

[0079] Next, the determination unit 523 determines the detection result of the detection unit 57 (S202). This allows the determination unit 523 to determine whether or not there is a problem with the movement of the terminal device 50 detected by the detection unit 57.

[0080] For example, if the detection unit 57 is an acceleration sensor, the speed of movement of the terminal device 50 is detected. The determination unit 523 determines that there is a problem with the movement of the terminal device 50 if the speed of movement of the terminal device 50 detected by the detection unit 57 is greater than or equal to a predetermined first threshold. The first threshold is, for example, 0.5 m / s.

[0081] Furthermore, if the detection unit 57 is a gyro sensor, for example, the tilt of the terminal device 50 is detected. In other words, the angle of the terminal device 50 with respect to the ceiling surface is detected. The determination unit 523 determines that there is a problem with the movement of the terminal device 50 if the angle of the terminal device 50 with respect to the ceiling surface detected by the detection unit 57 is greater than or equal to a predetermined second threshold. The second threshold is, for example, 30 degrees.

[0082] In step S202, the determination unit 523 determined the detection result of the detection unit 57. However, the determination unit 523 may also determine the movement of the terminal device 50 by determining at least one of the RGB data and Depth data being captured by the imaging unit 56. For example, the determination unit 523 may calculate the angle of the imaging unit 56 with respect to the ceiling surface by identifying the area in the RGB data being captured by the imaging unit 56 that shows the ceiling surface, and then perform the determination.

[0083] If a problem is detected (Yes in S203), the control unit 522 causes the display unit 55 to display warning guide information (S204). The display unit 55 displays the warning guide information superimposed on at least one of the RGB data and Depth data being captured. The warning guide information indicates that there is an abnormality in the imaging of the RGB data and Depth data.

[0084] For example, if the control unit 522 determines, based on the detection results, that there is a problem with the speed of movement of the terminal device 50, the control unit 522 causes the display unit 55 to display warning guide information. Figure 5 is a diagram showing an example of the display of warning guide information according to the embodiment. As shown in Figure 5, the display unit 55 of the terminal device 50 displays RGB data being captured targeting the ceiling surface 80 of the space. The control unit 522 causes the display unit 55 to display warning guide information 71 in order to inform the user that the movement of the terminal device 50 is too fast. For example, in the example shown in Figure 5, the text "Moving too fast" is displayed on the display unit 55 as warning guide information 71.

[0085] If the terminal device 50 moves too quickly, blurring may occur in the captured RGB data and depth data, causing errors in the data and potentially reducing the accuracy of the ceiling plan data created by the ceiling plan creation device 20. By displaying warning guide information 71, as shown in Figure 5, on the display unit 55, the user can be made to visually confirm the warning guide information 71, and the user can be helped to move the terminal device 50 at an appropriate speed to perform imaging.

[0086] Furthermore, for example, if the detection results determine that there is a problem with the angle of the imaging unit 56 relative to the ceiling surface, the control unit 522 causes the display unit 55 to display warning guide information. Figure 6 is a diagram showing another example of the display of warning guide information according to the embodiment. As shown in Figure 6, the display unit 55 of the terminal device 50 displays RGB data being captured for the ceiling surface of the space. The control unit 522 causes the display unit 55 to display warning guide information 72 in order to inform the user that the angle of the imaging unit 56 relative to the ceiling surface is too steep. For example, in the example shown in Figure 6, the warning guide information 72 displayed on the display unit 55 is the text "Shooting angle is too steep. Please lay it down" and an arrow to guide the movement of the user's terminal device 50.

[0087] If the angle of the imaging unit 56 with respect to the ceiling surface is steep, the area in which the equipment is captured in the captured RGB data and depth data may become distorted, which may reduce the accuracy of the ceiling plan data created based on that data. In addition, if the difference between the RGB data in the training data used to train the equipment estimation model 100 and the captured RGB data becomes large, there is a risk that misestimation of the type of equipment estimated using the trained equipment estimation model 100 may occur. By displaying warning guide information 72 as shown in Figure 6 on the display unit 55, the user can be made to visually confirm the warning guide information, and the user can be helped to perform imaging while keeping the imaging unit 56 at an appropriate angle.

[0088] If the detection result is determined to be without problems (No in S203), the process in step S204 is skipped and the process proceeds to step S205.

[0089] Finally, the control unit 522 generates RGB data and depth data for creating ceiling plan data (S205). Specifically, the control unit 522 generates RGB data for the ceiling plan creation device 20 to create ceiling plan data by identifying one or more appropriately captured RGB data from among multiple captured RGB data. The control unit 522 also generates depth data for the ceiling plan creation device 20 to create ceiling plan data by identifying one or more appropriately captured depth data from among multiple captured depth data. Note that the processing in step S205 may be omitted.

[0090] According to this first display operation of the terminal device 50, by displaying warning guide information indicating that there is an abnormality in the imaging of RGB data and depth data, it is possible to deter the user from imaging inappropriate RGB data and depth data.

[0091] [Second display operation of guide information] Furthermore, the terminal device 50 according to the embodiment displays area guide information as guide information for specifying the target area for imaging RGB data and depth data when the first area that is the target of imaging RGB data and depth data in the indoor space satisfies predetermined conditions. Such a terminal device 50 can help the user to image RGB data and depth data more efficiently.

[0092] The following describes a second display operation in which such a terminal device 50 displays area guide information. Figure 7 is a flowchart illustrating the second display operation of guide information during imaging by the terminal device 50 according to the embodiment.

[0093] First, the imaging unit 56 of the terminal device 50 starts imaging the ceiling surface (S301). In other words, the user uses the imaging unit 56 of the terminal device 50 to start imaging RGB data and depth data of the ceiling surface of the room. At this time, the control unit 522 displays at least one of the RGB data and depth data being captured by the imaging unit 56 on the display unit 55.

[0094] Next, the determination unit 523 determines the areas that have already been imaged (S302). An area that has already been imaged is an area in the indoor space that has been imaged in the past. In other words, an area that has already been imaged is an area in the indoor space from which RGB data and Depth data for creating ceiling plan data have already been imaged. An imaged area is an example of a second area.

[0095] For example, the determination unit 523 uses LiDAR technology to determine the areas that have been captured since the start of imaging of depth data on the ceiling surface.

[0096] When an area that has already been imaged is determined, the determination unit 523 determines whether or not the imaged area is included in the imaging range (S303). The imaging range is the area in the indoor space that is the target of imaging of the RGB data and Depth data. The imaging range is an example of a first area.

[0097] If it is determined that the imaging range includes an already imaged area (Yes in S303), the control unit 522 causes the display unit 55 to display an area guide (S304). The display unit 55 displays area guide information superimposed on at least one of the RGB data and Depth data being imaged. The area guide information is information for specifying the target of imaging for the RGB data and Depth data. Figure 8 is a diagram showing an example of the display of area guide information according to this embodiment.

[0098] As shown in Figure 8, the display unit 55 of the terminal device 50 displays RGB data being captured for the ceiling surface of the space. In other words, the area encompassed by the entire displayed RGB data is the imaging range. In the example shown in Figure 8, the area approximately 1 / 3 to the right of the displayed RGB data is determined to be the area that has already been captured.

[0099] At this time, the control unit 522 displays area guide information 73 on the display unit 55 to inform the user that the imaging range includes an already-imaged area. For example, in the example shown in Figure 8, the area guide information 73, which is a dot pattern, is superimposed on the portion of the already-imaged area in the RGB data being captured.

[0100] When a large proportion of the imaging range has already been imaged, the efficiency of the user in capturing RGB and Depth data for creating ceiling diagram data is reduced. The user can perform imaging more efficiently by checking the displayed area guide information 73 while imaging.

[0101] If it is determined that the imaging range does not include any areas that have already been imaged (No in S303), the process in step S304 is omitted, and the process proceeds to step S305. It should be noted that if the imaging range does not include any areas that have already been imaged, there is a possibility that gaps may be present in the area that was targeted for imaging, and this may result in the creation of ceiling plan data with gaps.

[0102] Finally, the control unit 522 generates RGB data and depth data for creating ceiling plan data (S305). Specifically, the control unit 522 excludes RGB data from the multiple captured RGB data where the same area is repeatedly captured, and generates RGB data for the ceiling plan creation device 20 to create ceiling plan data. The control unit 522 also excludes Depth data from the multiple captured Depth data where the same area is repeatedly captured, and generates Depth data for the ceiling plan creation device 20 to create ceiling plan data. Note that the processing in step S305 may be omitted.

[0103] According to this second display operation of the terminal device 50, by displaying area guide information for specifying the target for imaging RGB data and depth data, it is possible to support the user in imaging RGB data and depth data more efficiently.

[0104] [Effects, etc.] The technologies derived from the disclosures in this specification include, for example, the following. The technologies derived from the disclosures in this specification will be described below, along with the effects obtained by such technologies.

[0105] Technology 1 is a display method performed by a terminal device 50 equipped with an imaging unit 56 and a display unit 55, and includes a first display step in which the imaging unit 56 displays at least one of the RGB data and Depth data being captured on the display unit 55 while the imaging unit 56 is capturing RGB data and Depth data targeting the ceiling surface of an indoor space used to create a ceiling plan of the indoor space, and a second display step in which guide information to assist in capturing the RGB data and Depth data is superimposed on at least one of the RGB data and Depth data being captured and displayed on the display unit 55.

[0106] This display method, by displaying guide information, can assist users in capturing the data necessary for creating ceiling plan data. In other words, it can help users create more suitable data when creating the data required for creating ceiling plan data. This improves the quality of the data used to create ceiling plan data, contributing to improved drawing accuracy and a higher accuracy rate in estimating the type of equipment. Furthermore, it makes it possible to create data with less variation, regardless of the user's skill level.

[0107] Technology 2 is a display method of Technology 1, in which, in the second display step, if the movement of the terminal device 50 satisfies predetermined conditions, warning guide information indicating that there is an abnormality in the imaging of RGB data and Depth data is displayed as guide information.

[0108] This display method can deter users from capturing inappropriate RGB and depth data by displaying warning guide information.

[0109] Technology 3 is a display method of Technology 2, in which, in the second display step, warning guide information is displayed when the speed of movement of the terminal device 50 is equal to or greater than a predetermined first threshold.

[0110] This display method can warn the user if there is a risk that the data for creating the ceiling plan data may become inappropriate because the terminal device 50 is moving at a speed exceeding a predetermined speed.

[0111] Technology 4 is a display method of Technology 2, in which, in the second display step, warning guide information is displayed when the angle of the imaging unit 56 of the terminal device 50 with respect to the ceiling surface is greater than or equal to a predetermined second threshold.

[0112] This display method can warn the user if the angle of the imaging unit 56 with respect to the ceiling surface is too steep, which could result in the data for creating the ceiling plan data being unsuitable.

[0113] Technology 5 is a display method from Technology 1 to 4, in which, in the second display step, when the first region that is the target of imaging of RGB data and depth data in the indoor space satisfies predetermined conditions, region guide information for specifying the target of imaging of RGB data and depth data is displayed as guide information.

[0114] This display method, by displaying area guide information, can help users more efficiently acquire RGB and depth data.

[0115] Technology 6 is a display method of Technology 5, in which, in the second display step, if the first region in the indoor space includes a second region that has been imaged in the past, region guide information is superimposed and displayed on the portion of the second region in at least one of the RGB data and Depth data being imaged.

[0116] This display method helps to mitigate the inefficiency that can result from repeatedly capturing the same area, which can hinder the user's ability to capture RGB and Depth data for creating ceiling diagram data.

[0117] Technology 7 is a program that causes the terminal device 50 to execute the display method of Technology 1 or 2.

[0118] Such programs can assist users in capturing the data necessary to create ceiling plan data.

[0119] Technology 8 is a terminal device 50 comprising a display unit 55, an imaging unit 56 that captures RGB data and depth data targeting the ceiling surface of an indoor space used to create a ceiling plan of the indoor space, and an information processing unit that causes the imaging unit 56 to display at least one of the RGB data and depth data being captured on the display unit 55, wherein the information processing unit superimposes and displays guide information to support the capture of RGB data and depth data on at least one of the RGB data and depth data being captured.

[0120] Such a terminal device 50 can assist the user in capturing data necessary for creating ceiling plan data.

[0121] (Other embodiments) Although embodiments have been described above, the present invention is not limited to the embodiments described above.

[0122] For example, in the above embodiment, the system, such as the ceiling plan creation system, was implemented by multiple devices, but it may also be implemented as a single device. Thus, the system in this specification may consist of a single device or it may consist of multiple devices. When the system is implemented by multiple devices, the components of the system may be distributed among the multiple devices in any way.

[0123] Furthermore, in the above embodiment, a process executed by a specific processing unit may be executed by another processing unit. Also, the order of multiple processes may be changed, or multiple processes may be executed in parallel.

[0124] Furthermore, in the above embodiment, each component may be realized by executing a software program suitable for each component. Each component may also be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0125] Furthermore, each component may be implemented by hardware. Each component may also be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or they may be separate circuits. Also, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0126] Furthermore, general or specific embodiments of the present invention may be implemented as a system, apparatus, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM. Alternatively, they may be implemented as any combination of a system, apparatus, method, integrated circuit, computer program, and recording medium.

[0127] For example, the present invention may be implemented as a terminal device as described in the above embodiment. Alternatively, the present invention may be implemented as a display method executed by a terminal device. The present invention may also be implemented as a program for causing a computer to execute a display method, or as a computer-readable non-temporary recording medium on which such a program is stored.

[0128] Furthermore, the present invention also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art could conceive, or forms realized by arbitrarily combining the components and functions of each embodiment without departing from the spirit of the present invention. [Explanation of Symbols]

[0129] 20 Ceiling Plan Creation Device 50 Terminal devices 52 Information Processing Section 55 Display section 56 Imaging Unit 100 device estimation models 200 Ceiling Plan Data

Claims

1. A display method performed by a terminal device equipped with an imaging unit and a display unit, A first display step in which, during the operation of the imaging unit to capture RGB data and depth data targeting the ceiling surface of the interior space used to create a ceiling plan of the interior space, the imaging unit displays at least one of the RGB data and depth data being captured on the display unit; The second display step includes displaying guide information for assisting the imaging of the RGB data and depth data on the display unit, superimposed on at least one of the RGB data and depth data being imaged. Display method.

2. In the second display step, if the movement of the terminal device satisfies predetermined conditions, warning guide information indicating that there is an abnormality in the imaging of the RGB data and the Depth data is displayed as the guide information. The display method according to claim 1.

3. In the second display step, if the speed of movement of the terminal device is greater than or equal to a predetermined first threshold, the warning guide information is displayed. The display method according to claim 2.

4. In the second display step, if the angle of the imaging unit with respect to the ceiling surface is greater than or equal to a predetermined second threshold, the warning guide information is displayed. The display method according to claim 2.

5. In the second display step, if the first region in the indoor space that is the target of imaging the RGB data and depth data satisfies predetermined conditions, region guide information for specifying the target of imaging the RGB data and depth data is displayed as the guide information. The display method according to claim 1.

6. In the second display step, if the first region in the indoor space includes a second region that has been imaged in the past, the region guide information is superimposed and displayed on the portion of the second region in at least one of the RGB data and depth data being imaged. The display method according to claim 5.

7. A program for causing the terminal device to execute the display method described in any one of claims 1 to 6.

8. Display unit and An imaging unit that captures RGB data and depth data targeting the ceiling surface of the interior space, which is used to create a ceiling plan of the interior space, The imaging unit includes an information processing unit that causes at least one of the RGB data and depth data being captured to be displayed on the display unit. The information processing unit displays guide information to support the imaging of the RGB data and depth data by superimposing it on at least one of the RGB data and depth data during imaging. Terminal device.

Images