Support device for identifying problems, support method for identifying problems, and support program for identifying problems.

Abstract

To provide a pointing-out support device, a pointing-out support method, and a pointing-out support program for assisting efficient and accurate pointing-out work at a work site.SOLUTION: A support device 20 supports pointing-out at the time of situation confirmation by using a user device 10. A control unit 21 of this support device 20 includes a learning unit 211 that generates, for a part of a building, a prediction model for predicting pointing-out candidate information according to the part by using an image including pointing-out position information at the time of situation confirmation as training data. The control unit 21 further includes a prediction unit 213 that, when a new image is acquired from the user device 10, predicts pointing-out candidate information according to the part by inputting the new image to the prediction model, and outputs it to the user device 10.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a pointing support device, a pointing support method, and a pointing support program for supporting pointing work during on-site situation confirmation.

Background Art

[0002] On-site, workers may check the on-site situation and point out problem situations. In this case, technologies for obtaining defect information of a target building in consideration of the performance information of buildings constructed in the past have also been studied (for example, see Patent Document 1). In the technology disclosed in this document, the acquisition unit of the building information processing device acquires the design information of the target building. Next, the defect information acquisition unit inputs the design information acquired by the acquisition unit into a learned model to acquire the defect information of the target building. This learned model is generated in advance based on learning data in which the design information of the learning building and the defect information representing the defects of the learning building are associated with each other.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when checking the on-site situation, design information may not always be available.

Means for Solving the Problems

[0005] The identification support device for solving the above problems uses a user device to support identification during situation confirmation. This identification support device comprises a learning unit that generates a predictive model that predicts candidate identification information corresponding to a building part, using images containing location information of identifications during situation confirmation as training data for that part of the building, and a prediction unit that, when a new image is acquired from the user device, inputs the new image into the predictive model, predicts candidate identification information corresponding to the part, and outputs it to the user device. Other features and embodiments will become apparent from the following detailed description, drawings, and claims. [Effects of the Invention]

[0006] According to this disclosure, it is possible to support efficient and accurate identification work at the work site. [Brief explanation of the drawing]

[0007] [Figure 1] A diagram illustrating the system of the embodiment. [Figure 2] A diagram illustrating the hardware configuration of the embodiment. [Figure 3] Diagram illustrating the inspection information storage unit of the embodiment. [Figure 4] An explanatory diagram of an image containing the points of reference in the embodiment. [Figure 5] Diagram illustrating the learning result storage unit of the embodiment. [Figure 6] Diagram illustrating the information storage unit for the cooperating company in the embodiment. [Figure 7] Diagram illustrating the predictive evaluation information storage unit of the embodiment. [Figure 8] A diagram illustrating the learning process in the embodiment. [Figure 9] An explanatory diagram of the inspection process in the embodiment. [Figure 10] An explanatory diagram of the input screen for the embodiment. [Figure 11] An explanatory diagram of the input screen for the embodiment. [Figure 12] An explanatory diagram of the input screen for the embodiment. [Figure 13] Diagram illustrating the predictive model of another embodiment. [Figure 14] A diagram illustrating the inspection process in another embodiment. [Figure 15] Diagram illustrating the display screen of another embodiment. [Modes for carrying out the invention]

[0008] This description provides a comprehensive understanding of the described method, apparatus, and / or system. Modifications and equivalents of the method, apparatus, and / or system will be obvious to those skilled in the art. The sequence of operations is illustrative and may be modified as obvious to those skilled in the art, with the exception of cases where the operations necessarily occur in a specific order. Descriptions of functions and structures that are well known to those skilled in the art may be omitted. The exemplary embodiments may take different forms and are not limited to those described. However, the examples described are complete and finished and convey the entire scope of the disclosure to those skilled in the art. The following describes one embodiment of the identification support device, identification support method, and identification support program, using Figures 1 to 12. In this embodiment, the identification support device, identification support method, and identification support program are described as being used when making identifications while checking the site conditions at a building construction site. Throughout the drawings and detailed descriptions, the same reference numeral refers to the same element. Drawings may not be to scale, and relative dimensions, proportions, and depictions of elements in the drawings may be exaggerated for clarity, illustration, and convenience. In this embodiment, as shown in Figure 1, a user device 10 and a support device 20 are used that are interconnected via a network.

[0009] (Description of hardware configuration) Figure 2 illustrates the hardware configuration of the information processing device H10, which constitutes the user device 10 and the support device 20. The information processing device H10 includes a communication device H11, an input device H12, a display device H13, a storage device H14, and a processor H15. Note that this hardware configuration is just one example, and it can be implemented with other hardware.

[0010] The communication device H11 is an interface that establishes a communication path with other devices and performs data transmission and reception, such as a network interface, a wireless interface, or the like.

[0011] The input device H12 is a device that accepts input of various information, such as a mouse, a keyboard, or the like. The display device H13 is a display or the like that displays various information. Note that a touch panel display may be used as the input device H12 and the display device H13.

[0012] The storage device H14 is a storage device that stores data and various programs for executing the various functions of the user device 10 and the support device 20. Examples of the storage device H14 include a ROM, a RAM, a hard disk, and the like.

[0013] The processor H15 controls each process in the user device 10 and the support device 20 using the programs and data stored in the storage device H14. Examples of the processor H15 include, for example, a CPU, an MPU, or the like. This processor H15 expands the program stored in a ROM or the like into a RAM and executes various processes for each process.

[0014] The processor H15 is not limited to performing software processing for all processes it executes. For example, the processor H15 may include a dedicated hardware circuit (e.g., an application-specific integrated circuit: ASIC) that performs hardware processing for at least a part of the processes it executes. That is, the processor H15 may be configured as follows.

[0015] (1) One or more processors that operate according to a computer program (software) (2) One or more dedicated hardware circuits that execute at least a part of the various processes (3) A combination thereof, including circuitry A processor includes a CPU and memory such as RAM and ROM, where memory stores program code or instructions configured to cause the CPU to perform processing. Memory, or computer-readable media, includes any available media that can be accessed by a general-purpose or dedicated computer.

[0016] (System Configuration) Next, we will explain each function of the feedback support system using Figure 1. The user device 10 is a computer terminal for inputting information about issues identified during site inspections at a work site. For example, a portable smartphone or tablet device can be used as the user device 10. This user device 10 is used by the person responsible for inspecting the site conditions (the user). The user device 10 stores an inspection input application. The person responsible uses the user device 10 to photograph problem areas and input the identified information. The user device 10 includes a touch panel display 11 as an input device H12 and display device H13, a camera 12 as an input device H12, and a detection device 13.

[0017] The touch panel display 11 functions as both an input unit for inputting various types of information and an output unit for outputting various types of information. Note that the input and output units are not limited to the touch panel display 11; any hardware capable of inputting and outputting information is acceptable.

[0018] Camera 12 is a photographic device used to photograph a subject. The detection device 13 is a detection sensor for determining the shooting direction. Using the information output by this detection device 13, the elevation angle (vertical angle relative to the horizontal) of the camera 12 can be determined.

[0019] The support device 20 is a computer system that performs processing to support the confirmation of on-site conditions. This support device 20 comprises a control unit 21, an inspection information storage unit 22, a learning result storage unit 23, a cooperating company information storage unit 24, and a predictive evaluation information storage unit 25.

[0020] The control unit 21 performs the processing described later (including the learning phase, inspection phase, prediction phase, etc.). By executing the processing program for this purpose, the control unit 21 functions as the learning unit 211, inspection unit 212, prediction unit 213, etc.

[0021] The learning unit 211 generates a predictive model to support the input of comments by performing machine learning using training data generated based on inspection records. The inspection unit 212 acquires the information about the issues entered into the user device 10. This inspection unit 212 maintains a work type management table for identifying the type of work in association with the part and the issue. The prediction unit 213 uses a prediction model to support input regarding the user's device 10.

[0022] As shown in Figure 3, the inspection information storage unit 22 records inspection result information 220 related to findings during on-site inspections. This inspection result information 220 is recorded when finding information is obtained from the user device 10. The inspection result information 220 includes information on the project ID, finding ID, date and time, location, image, shooting conditions, part, finding, type of work, and cooperating company.

[0023] The project ID is an identifier used to identify a construction site (project). The issue ID is an identifier used to identify each issue in this project. The date and time information refers to the date and time when this report was registered. Location information is an identifier used to identify the space where the issue was raised (for example, a room including a bedroom or toilet).

[0024] The image is a photograph of the area where the issue was pointed out. This image contains information about the location where the issue was identified. For example, as shown in Figure 4, a pin object 501 is set in image 500. The pin object 501 indicates the location of the point of interest included in image 500.

[0025] The shooting conditions are those under which this image was taken. These shooting conditions include information about the camera's orientation (elevation / depression angle). The part information refers to the architectural element to which this criticism was made.

[0026] The inspection report contains information about the inspection items that were identified. Examples of issues that might be recorded include "scratches" on materials or problems such as areas that were not cleaned properly. Work type information is information used to identify the type of construction work. This type of work includes roofing work, block work, rebar work, glass work, waterproofing work, interior finishing work, joinery work, etc. The information on subcontractors is information about identifiers (subcontractor names) used to identify the subcontractors that performed the construction work.

[0027] As shown in Figure 5, the learning result storage unit 23 stores first and second prediction models 231 and 232 to support the identification work during situation confirmation. These first and second prediction models 231 and 232 are recorded when the learning process is performed. In this embodiment, the first and second prediction models 231 and 232 output candidate information for identification.

[0028] As shown in Figure 5, the first prediction model 231 predicts a part D11 as a candidate for identification information based on posture information D03 when a location D02 is set in an image D01 taken at a construction site. The second prediction model 232 predicts an item D12 as a candidate for identification information based on posture information D03 when a location D02 is set in an image D01 taken at a construction site.

[0029] As shown in Figure 6, the subcontractor information storage unit 24 records subcontractor information 240 concerning subcontractors who perform construction work on each part of the building site. This subcontractor information 240 is recorded when a subcontractor is registered. The subcontractor information 240 includes information on location, type of work, and name of the subcontractor.

[0030] Location information is an identifier used to identify the space (e.g., room) where the issue was raised. Work type information refers to information about the trades involved in the construction work. The name of the cooperating company is the name of the cooperating company that carried out the construction work, and it serves as an identifier to identify the cooperating company.

[0031] As shown in Figure 7, the prediction evaluation information storage unit 25 stores prediction evaluation information 250 for evaluating the prediction results of the prediction model. This prediction evaluation information 250 is recorded when inspection processing is performed. The prediction evaluation information 250 includes information on the project ID, the issue ID, the prediction result, the selection result, and the judgment flag.

[0032] The project ID is an identifier used to identify a construction site (project). The issue ID is an identifier used to identify each issue in this project. The prediction results information consists of candidate sites and potential issues predicted using the first and second prediction models 231 and 232. Here, the most likely candidate sites and potential issues are recorded.

[0033] The selection results information consists of the parts selected and the points of concern by the person in charge. The judgment flag determines whether the most likely candidate body part and issue match the body part and issue selected by the person in charge.

[0034] [Support processing for identifying issues] The feedback support process will be explained using Figures 8 to 12. The explanation will proceed in the following order: learning process (Figure 8) followed by inspection process (Figure 9).

[0035] (Processing during learning) First, we will explain the learning process using Figure 8. Here, the control unit 21 of the support device 20 executes the process of acquiring inspection records (step S101). Specifically, the learning unit 211 of the control unit 21 acquires the inspection result information 220 recorded in the inspection information storage unit 22.

[0036] Next, the control unit 21 of the support device 20 executes the process of creating training data (step S102). Specifically, the learning unit 211 of the control unit 21 creates multiple datasets consisting of [image, shooting conditions, body part] and [image, shooting conditions, comments] as training data.

[0037] Next, the control unit 21 of the support device 20 executes the process of generating a prediction model (step S103). Specifically, the learning unit 211 of the control unit 21 uses training data [image, shooting conditions, body part] to generate a first prediction model by machine learning, with the image and shooting conditions as explanatory variables and the body part where the pin object is set as the target variable. Furthermore, the learning unit 211 uses training data [image, shooting conditions, comments] to generate a second prediction model by machine learning, with the image and shooting conditions as explanatory variables and the comments on the body part where the pin object is set as the target variable. As a machine learning method, for example, a CNN (Convolutional Neural Network) can be used. Then, the learning unit 211 records the generated first prediction model 231 and second prediction model 232 in the learning result storage unit 23.

[0038] (Processing during inspection) Next, Figure 9 will be used to explain the inspection process. During inspection, the person in charge carries the user device 10 and tours the construction site. If the person in charge finds a problem, they launch the inspection input application on the user device 10. This causes the user device 10 to output a problem input screen with the project ID set.

[0039] As shown in Figure 10, the issue input screen 600 includes an input field 610. This input field 610 includes a date field, a location field, a photo button, a part field, an issue item field, a work type field, a cooperating company field, and a registration button. The date field displays information about the current date and time obtained from the system timer. Furthermore, the person in charge enters information about their current location in the location field.

[0040] When the person in charge points out a problem, they use the capture button to take a picture of the area being pointed out. In this case, the user device 10 displays the image captured by the camera 12 on the touch panel display 11.

[0041] Then, the control unit 21 of the support device 20 executes the process of acquiring captured images (step S201). Specifically, the inspection unit 212 of the control unit 21 acquires the captured images taken by the person in charge from the user device 10.

[0042] Next, the control unit 21 of the support device 20 performs the process of acquiring shooting posture information (step S202). Specifically, the inspection unit 212 of the control unit 21 acquires the elevation / depression angle at the time of shooting, which is detected by the detection device 13 from the user device 10.

[0043] Next, the control unit 21 of the support device 20 executes the process of acquiring the location of the issue (step S203). Specifically, the person in charge specifies the location of the point of issue (point of issue) in the captured image displayed on the touch panel display 11.

[0044] In this case, as shown in Figure 11, the user device 10 places a pin object 601 at the location of the point of reference on the captured image of the point of reference input screen 600. Then, the inspection unit 212 of the control unit 21 acquires information (coordinates) related to the point of reference.

[0045] Next, the control unit 21 of the support device 20 performs a process to identify candidate issues (step S204). Specifically, the prediction unit 213 of the control unit 21 predicts candidate body parts at the issue location by inputting the captured image and posture information of the issue location into a first prediction model. Furthermore, the prediction unit 213 predicts candidate issues at the issue location by inputting the captured image and posture information of the issue location into a second prediction model. Information regarding the likelihood (probability of correctness) is associated with the predicted candidate body parts and candidate issues.

[0046] Next, the control unit 21 of the support device 20 performs output processing for candidate issues (step S205). Specifically, the inspection unit 212 of the control unit 21 outputs candidate body parts and candidate issues in order of increasing likelihood in the input field of the issue input screen. For example, as shown in Figure 12, in the input field 610 of the complaint input screen 600, the candidate body part 611 and the candidate complaint item 612 are displayed as complaint candidate information in order of highest probability. The candidate body part 611 and the candidate complaint item 612 are then displayed in their respective fields using pull-down menus arranged from top to bottom in order of highest probability. These pull-down menus include blank fields that can be edited.

[0047] Next, the control unit 21 of the support device 20 performs the process of setting the content of the complaint (step S206). Specifically, the person in charge selects the desired part and complaint from the candidate parts and complaints on the complaint input screen 600. If the desired content is not included in the candidate parts and complaints, the person in charge enters the part and complaint into the input field 610 as needed. In this case, the inspection unit 212 of the control unit 21 acquires the part and complaint set (selected or entered) on the complaint input screen 600.

[0048] Next, the control unit 21 of the support device 20 performs the work type identification process (step S207). Specifically, the inspection unit 212 of the control unit 21 identifies the work type using the work type management table based on the part and the points of concern set in the input field 610. Then, the inspection unit 212 displays the identified work type in the input field 610 of the point of concern input screen 600.

[0049] Next, the control unit 21 of the support device 20 performs the process of identifying the cooperating company (step S208). Specifically, the inspection unit 212 of the control unit 21 obtains the name of the cooperating company corresponding to the location and part set in the input field 610 from the cooperating company information storage unit 24. Then, the inspection unit 212 displays the obtained cooperating company name in the input field 610 of the identification input screen 600.

[0050] Next, the control unit 21 of the support device 20 performs the process of recording the identified information (step S209). Specifically, if the registration button is selected on the identified information input screen 600, the inspection unit 212 of the control unit 21 generates inspection result information 220 with an identified information ID and records it in the inspection information storage unit 22. This inspection result information 220 includes the identified information ID, shooting conditions, an image including the identified location, the date and time, location, part, identified item, type of work, and information about the cooperating company displayed in the input field 610.

[0051] Next, the control unit 21 of the support device 20 performs the process of recording evaluation information (step S210). Specifically, the inspection unit 212 of the control unit 21 generates predictive evaluation information 250 that records the project ID and the issue ID, and records it in the predictive evaluation information storage unit 25. This predictive evaluation information 250 records the most likely candidate parts and issue items as prediction results. Furthermore, it records the parts and issue items set by the person in charge as selection results. Finally, it records a flag that identifies whether the most likely candidate parts and issue items match or do not match the parts and issue items selected by the person in charge.

[0052] Next, the control unit 21 of the support device 20 performs a process to determine whether relearning is necessary (step S211). Specifically, the inspection unit 212 of the control unit 21 counts the number of registered predictive evaluation information 250 in which a mismatch flag is recorded in the predictive evaluation information storage unit 25. If the number of predictive evaluation information 250 with a mismatch flag is equal to or greater than a certain number, the inspection unit 212 outputs a message prompting relearning to the administrator device (not shown). Alternatively, the inspection unit 212 may output a message prompting relearning to the administrator device if the percentage of mismatches relative to the total number exceeds a certain value.

[0053] According to this embodiment, the following effects can be obtained. (1) In this embodiment, the control unit 21 of the support device 20 executes the process of generating a prediction model (step S103). Specifically, the learning unit 211 of the control unit 21 generates a first prediction model 231 by machine learning, with the image and shooting conditions as explanatory variables and the part where the pin object is set as the target variable. This makes it possible to identify candidate parts of the pointed-out area using the captured image. Therefore, even if the shapes are similar, such as walls, ceilings, and floors, the part of the structure can be accurately predicted by using posture information.

[0054] (2) In this embodiment, the control unit 21 of the support device 20 executes the process of generating a prediction model (step S103). Specifically, the learning unit 211 of the control unit 21 generates a second prediction model 232 by machine learning, with the image and shooting conditions as explanatory variables and the points raised in the area where the pin object is set as the objective variable. This makes it possible to identify candidate points raised in the raised area using the captured image.

[0055] (3) In this embodiment, the control unit 21 of the support device 20 performs the process of acquiring captured images (step S201) and the process of acquiring captured posture information (step S202). This makes it possible to acquire input data for making predictions.

[0056] (4) In this embodiment, the control unit 21 of the support device 20 performs the process of identifying candidate issues (step S204) and the process of outputting candidate issues (step S205). This reduces the input burden on the person in charge regarding parts and issues, and enables efficient inspection work.

[0057] (5) In this embodiment, the control unit 21 of the support device 20 identifies the type of work using the work type management table based on the part and points to note set in the input field 610 during the work type identification process (step S207). This reduces the input burden on the person in charge of the work type. (6) In this embodiment, the control unit 21 of the support device 20 acquires the name of the cooperating company corresponding to the location and part set in the input field 610 during the cooperating company identification process (step S208). This reduces the input burden on the person in charge of cooperating company information.

[0058] (7) In this embodiment, the control unit 21 of the support device 20 performs the process of recording evaluation information (step S210) and the process of determining whether or not relearning is necessary (step S211). This allows the prediction model to be corrected by relearning if the prediction model is not accurate. In this case, [image, shooting conditions, selection results (part or points of concern)] are used as training data.

[0059] This embodiment can be implemented with the following modifications. This embodiment and the following modifications can be combined with each other to the extent that they do not contradict each other technically. The above embodiment is used when conducting inspections at a building construction site. However, the scope of application of this disclosure is not limited to construction sites.

[0060] The above embodiment describes a case where a user device 10 and a support device 20 are used as the hardware configuration. However, the hardware configuration is not limited to these. For example, the inspection unit 212, prediction unit 213, and learning result storage unit 23 may be stored in the user device 10. In this case, the user device 10 uses a prediction model to identify candidate issues.

[0061] In the above embodiment, a predictive model is generated using training data that includes captured images containing the identified areas and posture information. The training data is not limited to these. For example, it may also include BIM (Building Information Modeling) information.

[0062] Furthermore, BIM information and self-localization information may be used in combination. In BIM information, each 3D model (object) placed in the virtual 3D space holds various information related to the building as attributes. For self-localization information, for example, GNSS can be used. Also, for this self-localization estimation, for example, visual odometry (VIO: visual-inertial odometry), which estimates the distance traveled according to the relative changes in images sequentially captured by camera 12, can be used. In this technique, the detection device 13 detects feature points in the captured image and estimates the movement of camera 12 according to the correlated positional relationships of the feature points. Note that the method of determining the position is not limited to using visual odometry. For example, a 3-axis accelerometer and a gyroscope can be used individually or in combination. Then, a prediction model is generated using the 3D model of the subject at the self-localization and the captured image as input.

[0063] Furthermore, the location within the virtual space may be determined using BIM information through self-localization in the sequentially captured images. This allows the control unit 21 of the support device 20 to set location information in the input field 610 of the identification input screen 600.

[0064] Alternatively, mixed reality (MR) technology may be used to display an image on the touch panel display 11 that superimposes a real image and a virtual image. In this case, BIM information in the mixed reality image may be used to identify the body part.

[0065] Additionally, voice memos and comments may be recorded in the inspection information storage unit 22. Voice memos and comments are information entered by the person in charge on the user device 10. Voice memos are converted to text using speech recognition.

[0066] Alternatively, ambient sounds acquired by the microphone of the user device 10 may be used. Furthermore, the images are not limited to still images; videos may also be used. For example, a few seconds of video taken before the examination record is created may be used. Then, this information is included in the training data, and machine learning is used to generate each predictive model.

[0067] In the above embodiment, the prediction evaluation information storage unit 25 stores prediction evaluation information 250 for evaluating the prediction results of the prediction model. The prediction evaluation information 250 includes information on the prediction result, selection result, and judgment flag. This prediction result information records the most likely part candidate and the point of concern candidate. The prediction result information is not limited to the most likely candidate. For example, multiple top candidates with high probability may be recorded. In this case, if the selection result is included in the prediction result, it is determined to be a match. Then, during retraining, weakly supervised learning is performed using training data consisting of multiple candidates including the selection result. This makes it possible to determine whether an effective prediction that helps the model is being made by using the inclusion relationship between the top candidates and the selection result.

[0068] In the above embodiment, the control unit 21 of the support device 20 performs a prediction model generation process (step S103). Here, first and second prediction models 231 and 232 are generated to predict candidate body parts and candidate issues as candidate information for issues. The prediction models are not limited to these. For example, a prediction model may be generated that uses "body parts and issues" as explanatory variables. Alternatively, a second prediction model may be generated that predicts issues from images, posture information, and body parts by adding the body parts predicted by the first prediction model 231 as explanatory variables.

[0069] In the above embodiment, image and posture information, including the location of the issue, are used to predict candidate body parts and candidate issues as candidate information for the issue. If the items to be entered in the issue can be predicted, it is not necessary to predict both, and one or the other items may be predicted. Alternatively, the system may use images taken before any issues are pointed out to predict potential locations where issues might be identified. In this case, as shown in Figure 13, a third prediction model 233 is generated using image D01 and posture information D03 to predict candidate locations for the points of concern, and this model is recorded in the learning result storage unit 23. In this case, a heatmap D13 is generated that shows the probability distribution of the likelihood of each candidate location for the points of concern. This heatmap D13 shows the likelihood of a pin object being placed in the image. Then, using this heatmap D13, the person in charge specifies the point of concern D02. In this case, the point of concern D02 is displayed within image D01.

[0070] As shown in Figure 14, first, the control unit 21 of the support device 20 performs the process of acquiring captured images (step S201) and the process of acquiring captured orientation information (step S202). These captured images do not contain pin objects. Then, the control unit 21 of the support device 20 executes the heat map generation process (step S301). Specifically, the prediction unit 213 of the control unit 21 inputs the image D01 and posture information D03 to the third prediction model 233 of the learning result storage unit 23 and calculates the probability (correct answer probability) for each pointed-out location. Then, it generates a heat map in which the color scheme is darker for areas with a higher correct answer probability.

[0071] As shown in Figure 15, the display screen 700 shows two heatmaps 710 and 720 as potential issues to be addressed. Heatmap 710 shows the distribution of issues "forgotten cleaning" in the area "shelves". Heatmap 720 shows the heatmap of issues "scratches" in the area "walls".

[0072] The person in charge then checks the heat map and enters any necessary comments. Subsequently, the control unit 21 of the support device 20 executes the processes from acquiring the location of the comments (step S203) onward.

[0073] Next, the technical concepts that can be understood from the above embodiments and alternative examples are described below. (a) The new image includes location information for the body part, The identification support device according to claim 1, characterized in that the prediction unit predicts the content of the identification corresponding to the location information of the identification and outputs it to the user device.

[0074] (b) The identification support device according to (a), characterized in that the identified information includes information about the part that has a problem. (c) The notification support device according to (a) or (b), characterized in that the notification content includes information regarding the matters pointed out during the situation confirmation.

[0075] (d) The prediction unit predicts candidate locations for pointing out during situation confirmation in the new image and outputs a heat map showing the probability distribution of the candidate locations to the user device, characterized in that the pointing out support device according to any one of 1, (a) to (c). Various modifications in form and detail may be made to the above examples without departing from the spirit and scope of the claims and their equivalents. The above examples are for illustrative purposes only and not for limitation. The descriptions of features in each example should be considered applicable to similar features or embodiments in other examples. Appropriate results may be achieved when sequences are performed in different orders, and / or when components in the described systems, architectures, apparatuses, or circuits are combined in different ways, and / or when they are replaced or complemented by other components or their equivalents. The scope of the disclosure is not determined by the detailed description but by the claims and their equivalents. All variations within the scope of the claims and their equivalents are included in the disclosure. [Explanation of symbols]

[0076] 10...User device, 11...Touch panel display, 12...Camera, 13...Detection device, 20...Support device, 21...Control unit, 211...Learning unit, 212...Inspection unit, 213...Prediction unit, 22...Inspection information storage unit, 23...Learning result storage unit, 24...Cooperating company information storage unit, 25...Prediction and evaluation information storage unit.

Claims

1. A user device equipped with a detection device that detects posture information is used to provide a feedback support device that assists in providing feedback during situation confirmation. A learning unit generates a predictive model that predicts candidate information for each part of a building, using images containing posture information and location information of points raised during situation checks as training data. A pointing support device characterized by comprising: a prediction unit that, when a new image is acquired from the user device, inputs the new image with the pointing position set, along with the posture information detected by the detection device, into the prediction model to predict candidate pointing information corresponding to the body part, and outputs it to the user device.

2. The learning unit generates a first prediction model that predicts candidate locations for the location being pointed out, and a second prediction model that predicts candidate information for the location corresponding to the location, The identification support device according to claim 1, characterized in that when a new image is acquired from the user device, the prediction unit inputs the new image into the first prediction model to identify a part, and inputs the new image into the second prediction model to predict candidate information for identification corresponding to the part.

3. Further comprising an inspection unit that holds a work type management table for identifying work types in correspondence with parts and issues, The inspection unit is characterized by identifying the type of work based on the part and the identified item, and outputting it to the user device, as described in claim 1 or 2.

4. The identification support device according to any one of claims 1 to 3, characterized in that the prediction unit generates a heat map showing the probability distribution of the likelihood of an identification location candidate for each identification item and outputs it to the user device.

5. A method for supporting pointing out issues during situation confirmation, using a pointing out support device connected to a user device equipped with a detection device for detecting posture information, The aforementioned identification support device, Using images containing posture information and location information from situation checks as training data for building components, a predictive model is generated to predict candidate information for each component. A method for supporting identification, characterized in that when a new image is acquired from the user device, the new image with the identification location set is input to the prediction model along with the posture information detected by the detection device, the model predicts candidate identification information corresponding to the body part, and outputs it to the user device.

6. A program for assisting in pointing out issues during situation confirmation, using a pointing out support device connected to a user device equipped with a detection device that detects posture information, The aforementioned identification support device, A learning unit generates a predictive model that predicts candidate information for each building part, using images containing posture information and location information of points raised during situation checks as training data. A pointing support program characterized in that, when a new image is acquired from the user device, the new image with the pointing location set, along with the posture information detected by the detection device, is input to the prediction model, which then predicts candidate pointing information corresponding to the body part and outputs it to the user device, thereby functioning as a prediction unit.

Images