Image processing apparatus, and image processing method

Abstract

To provide a technique to generate an opinion sentence obtained by adding appropriate deformation to a report and individual judgement rankings.SOLUTION: An image processing apparatus acquires deformation information related to deformation detected from a picked-up image and instruction information for instructing details to be included in an opinion sentence about the deformation, and generates the levels of the opinion sentence and deformation as a result of output from a learned model based on the deformation information and instruction information.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to image processing technology.

Background Art

[0002] Concrete surfaces of bridges and buildings may be deformed (cracked, leaking, etc.) due to various factors. Since defects in structures due to deformation need to be detected and repaired at an early stage, inspection workers regularly visually checked the concrete surface and directly marked the deformed parts with chalk for inspection. After inspection, an inspection report was created by writing on the drawing based on the chalk marks and submitted to the country or government offices. Recently, for labor saving, a system has emerged that inputs inspection images of the concrete surface taken by a camera into the system and automatically detects deformation. However, when using this method, it is necessary to finally confirm whether the detection accuracy of a plurality of deformations automatically detected during the creation of the inspection report is comparable to that in the case of visual inspection, and to correct it if necessary, and then submit photos or drawings clearly indicating the deformed parts. Therefore, the procedure for creating the inspection report is still a labor-intensive task for inspection workers. In Non-Patent Document 1, for the automation of the inspection report creation work, a technique of inputting inspection images into AI and outputting damage situation sentences in the inspection images has been studied.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional inspection system, all detected deformations were superimposed and output on a single inspection image. However, among the detection results, there may be a mixture of accurate detection results that match the actual deformations and inaccurate detection results that do not match the actual deformations. On the other hand, only the accurate detection results need to be included in the inspection report. Therefore, in order to create the inspection report, the user has to find the parts that require reporting from the detection results downloaded from the system and write out the observations and individual judgment texts while considering the judgment criteria for each local government, material, and surrounding environment for each damage. This work is still a heavy burden for the inspection workers. The present invention provides a technique for generating observation texts and individual judgment ranks that take into account appropriate deformations in the report.

Means for Solving the Problems

[0005] One aspect of the present invention includes an acquisition means for acquiring deformation information related to a deformation detected from a captured image and instruction information for instructing the content to be included in the observation text of the deformation, and a generation means for generating the observation text and the level of the deformation as an output result of a learned model based on the deformation information and the instruction information.

Effects of the Invention

[0006] According to the configuration of the present invention, it is possible to provide a technique for generating observation texts and individual judgment ranks that take into account appropriate deformations in the report.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0008] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are given the same reference numerals, and duplicate explanations are omitted.

[0009] [First Embodiment] In this embodiment, as an output result of a learned model based on deformation information related to a deformation detected from a captured image of a structure including a deformation and instruction information for instructing the content to be included in the findings text of the deformation, a findings text and an individual determination rank that is the level of the deformation are generated. An example of an image processing apparatus will be described. In this embodiment, as an example, a case where an AI (Artificial Intelligence) model is used as the learned model will be described.

[0010] Note that "abnormal conditions" refer to cracks, water leakage, etc. occurring on the concrete surface due to damage, deterioration, or other factors in structures such as motorways, bridges, tunnels, dams, etc. "Cracks" refer to linear damages with starting points, ending points, lengths, and widths occurring on the walls of structures due to aging deterioration, earthquake impacts, etc. "Water leakage" indicates a state where water infiltrates through cracks or gaps formed in the concrete due to the influence of rain or the like, and water is leaking.

[0011] First, a hardware configuration example of the image processing apparatus 101 according to the present embodiment will be described with reference to the block diagram of FIG. 1. A computer device such as a PC, smartphone, tablet terminal device, etc. can be applied to the image processing apparatus 101 according to the present embodiment. Also, the image processing apparatus 101 may be implemented by a single computer device, or may be implemented by a plurality of computer devices. In the latter case, each computer device cooperates to execute the processing described as the processing performed by the image processing apparatus 101.

[0012] The control unit 111 is an arithmetic processing processor such as a CPU or MPU, and executes various processes using computer programs and data stored in the volatile memory 112. Thereby, the control unit 111 controls the overall operation of the image processing apparatus 101, and also executes or controls various processes described as the processes performed by the image processing apparatus 101.

[0013] The volatile memory 112 is a memory for primary storage, for example, a RAM. The volatile memory 112 has an area for storing computer programs and data loaded from the non-volatile memory 113, and an area for storing computer programs and data loaded from the storage device 114. Further, the volatile memory 112 has an area for storing computer programs and data received from the outside by the communication device 117. Further, the volatile memory 112 has a work area used when the control unit 111 executes various processes. Thus, the volatile memory 112 can appropriately provide various areas.

[0014] The non-volatile memory 113 is, for example, a ROM. The non-volatile memory 113 stores setting data of the image processing apparatus 101, computer programs and data related to the startup of the image processing apparatus 101, computer programs and data related to the basic operations of the image processing apparatus 101, and the like.

[0015] The storage device 114 is a large-capacity information storage device such as a hard disk drive. The storage device 114 stores an OS, computer programs and data for causing the control unit 111 to execute or control various processes described as processes performed by the image processing apparatus 101, and the like.

[0016] Note that the storage device 114 may be a memory card as an internal device of the image processing apparatus 101 or a memory card externally attached to the image processing apparatus 101. Further, the storage device 114 may be a disk drive that reads and writes computer programs and data to and from optical disks such as DVDs and Blue-ray Discs.

[0017] The input device 115 is a user interface such as a keyboard, a mouse, or a touch panel screen, and various instructions and information can be input to the image processing apparatus 101 by the user operating it. In the following description, the "user operation" shall mean an operation performed by the user using the input device 115.

[0018] The output device 116 is a display device having an LCD, an organic EL, or the like, and can display the processing result by the control unit 111 as an image, characters, or the like. Note that the output device 116 may be a projection device such as a projector that projects images and characters.

[0019] The communication device 117 performs data communication with the outside via a network such as the Internet or a LAN. The system bus 118 includes an address bus, a data bus, and a control bus. The control unit 111, the volatile memory 112, the non-volatile memory 113, the storage device 114, the input device 115, the output device 116, and the communication device 117 are all connected to the system bus 118.

[0020] The processing performed in the image processing apparatus 101 according to the present embodiment is realized, for example, by an application computer program or data. Note that the application has software for using the basic functions of the OS installed in the image processing apparatus 101. Note that the OS of the image processing apparatus 101 may have software for realizing various processes described as processes performed by the image processing apparatus 101.

[0021] An example of the functional configuration of the image processing apparatus 101 is shown in the block diagram of FIG. 2. In FIG. 2, the storage units 214, 218, and 219 can be implemented using, for example, memory devices such as the volatile memory 112, the non-volatile memory 113, and the storage device 114. In the present embodiment, a case where the image management unit 211, the image analysis unit 213, the generation unit 217, and the management unit 220 are all implemented by software (computer program) will be described. Hereinafter, the image management unit 211, the image analysis unit 213, the generation unit 217, and the management unit 220 may be described as the main bodies of the processing. However, actually, the control unit 111 executes a computer program corresponding to these functional units to realize the functions of the functional units. Note that one or more of the functional units of the image management unit 211, the image analysis unit 213, the generation unit 217, and the management unit 220 may be implemented by hardware.

[0022] Next, the processing performed by the image processing apparatus 101 to generate text describing the findings of the abnormality and the individual determination rank of the abnormality from the captured image of the structure including the abnormality and display it on the output device 116 will be described.

[0023] The image analysis unit 213 acquires a captured image that is the object to be detected for deformation (analysis target). The method of acquiring the captured image is not limited to a specific acquisition method. For example, the image analysis unit 213 may acquire the captured image transmitted from an external device (such as a server device or an imaging device) via the communication device 117, or may acquire it from the non-volatile memory 113 or the storage device 114.

[0024] Then, the image analysis unit 213 inputs the captured image into an AI model (AI that has been trained to detect deformation in the input image, trained in machine learning and deep learning) and performs the arithmetic processing of the AI model to obtain deformation information, which is the deformation information in the captured image. Then, the image analysis unit 213 stores a set of the captured image and the deformation information of each deformation obtained from the captured image in the storage unit 214 as an analysis result set.

[0025] An example of the composition of the deformation information will be described with reference to FIG. 3. FIG. 3 shows the deformation information of each deformation detected from the captured image. The deformation information is vector data including a deformation ID 311, a deformation type 312, a length / area / width 313, and vertex coordinates 314, and is stored in the storage unit 214 in a format such as CSV or DXF.

[0026] The deformation ID 311 is unique identification information for each deformation information, and in FIG. 3, it is represented by numerical values such as 1, 2, 3,.... The deformation type 312 is the type of deformation, and in FIG. 3, it is either "crack" or "leakage". For example, the deformation type 312 included in the deformation information with the deformation ID 311 being "2" is "crack".

[0027] The length / area / width 313 is the size of the deformation. When the type of deformation is "crack", it is the "length and width of the deformation (damage width)", and when the type of deformation is "leakage", it is the "area of the deformation". For example, the length / area / width 313 included in the deformation information with the deformation ID 311 being "2" is "length: 7 cm, width: 0.1 mm". For example, the length / area / width 313 included in the deformation information with the deformation ID 311 being "4" is "area: 5 m 2It is "」. Consider expressing the size here in millimeters as relative coordinates based on the position of the detection target of the image in the rectangular coordinate system. The vertex coordinate 314 is the coordinate in the captured image of the endpoint (vertex) of each line segment when the deformation is represented by a plurality of line segments.

[0028] The image management unit 211 acquires the analysis result set stored in the storage unit 214 and causes the GUI (Graphical User Interface) 401 illustrated in FIG. 4 to be displayed on the output device 116. The image management unit 211 causes the captured images included in the acquired analysis result set to be displayed in the area 413.

[0029] Here, when the user checks the checkbox 412 using the input device 115, the image management unit 211 selects, as the first display candidate, the deformation information in which the deformation type 312 is "crack" among the deformation information included in the analysis result set acquired from the storage unit 214.

[0030] Then, when the user checks the checkbox 412a using the input device 115, the image management unit 211 selects, as the display target, the first display candidates among the first display candidates whose width in the length / area / width 313 is "less than 0.01 mm".

[0031] Also, when the user checks the checkbox 412b using the input device 115, the image management unit 211 selects, as the display target, the first display candidates among the first display candidates whose width in the length / area / width 313 is "0.01 mm or more and less than 0.5 mm".

[0032] Also, when the user checks the checkbox 412c using the input device 115, the image management unit 211 selects, as the display target, the first display candidates among the first display candidates whose width in the length / area / width 313 is "0.5 mm or more".

[0033] Also, when the user checks the check box 419 using the input device 115, the image management unit 211 selects, as display targets, the deformation information in which the deformation type 312 is "water leakage" among the deformation information included in the analysis result set acquired from the storage unit 214.

[0034] Then, the image management unit 211 generates an object representing the deformation corresponding to the selected deformation information as a display target by connecting the coordinates between the coordinates in order from the first coordinate at the vertex coordinates 314 of the deformation information with line segments. Then, the image management unit 211 superimposes the generated object on the captured image and displays it in the area 413.

[0035] The object 414 is an object generated by connecting the coordinates between the coordinates in order from the first coordinate at the vertex coordinates 314 of the deformation information in which the deformation type 312 is "crack" with line segments.

[0036] The object 415 is an object generated by connecting the coordinates between the coordinates in order from the first coordinate at the vertex coordinates 314 of the deformation information in which the deformation type 312 is "water leakage" with line segments.

[0037] Note that the image management unit 211 determines in advance the overlapping order of the objects for each deformation type so that the user can clearly visually recognize each deformation. For example, water leakage represents a wide range as a deformation, while a crack is represented by the length and width of a line segment. Therefore, in the GUI 401 of FIG. 4, the object 415 over a wide range such as water leakage is superimposed on the captured image 413, and the object 414 such as a crack is superimposed and displayed thereon.

[0038] Further, when the user instructs the button 416 using the input device 115, the image management unit 211 causes the GUI 501 illustrated in FIG. 5 to be displayed on the output device 116. The GUI 501 is a GUI for generating, for a region specified according to a user operation in a captured image, a finding text of a change in the region and an individual determination rank that is the level of the change. FIG. 5 shows a state in which all of the check boxes 412, 412a, 412b, 412c, and 419 are checked.

[0039] Here, the finding text is text data that expresses including the change type 312 and the length / area / width 313 included in the change information of the change. The individual determination rank is distinguished based on the size of the change of the structure and the overlap with different changes, and is generated for each change information. The criteria for determining the individual determination rank are determined with reference to the road inspection guidelines of each country and local government.

[0040] When the user instructs the button 511 using the input device 115, the image management unit 211 accepts a range designation operation for the captured image displayed in the area 413. For example, as shown in FIG. 7, the user moves the mouse cursor 711 using the mouse as the input device 115, and designates a range (deformation range) including an object of a change to be the subject of creating a finding statement and an individual determination rank in the captured image displayed in the area 413. The method of designating the deformation range (that is, a partial range in the captured image) is not limited to a specific designation method. For example, the user may click the mouse multiple times within the area 413 to designate the range within the figure having the clicked position as the vertex as the deformation range. Also, for example, a figure (such as a circle or a rectangle) of a shape selected according to the user operation is displayed in the area 413, and the range within the figure may be designated as the deformation range by enlarging / reducing / moving the figure according to the user operation. When the deformation range is designated according to the user operation, the image management unit 211 displays the frame 712 of the designated deformation range. In FIG. 7, a deformation range including the object 414 and the object 415 is designated according to the user operation, and the frame 712 of the deformation range is displayed. Also, the deformation range may be a range detected from the captured image using a learned model for detecting a change in the image. Also, in the subsequent processing, instead of the deformation range, an image within the deformation range or a processed image obtained by subjecting the image to image processing (such as noise removal, edge enhancement, resizing) may be used.

[0041] In addition, the user can input finding statement content instruction text data into the area 513 using the input device 115. The finding statement content instruction text data is text data in which content for instructing what content to include in the finding statement when outputting the finding statement to the AI model is described. Such content is, for example, content such as "Describe cracks in detail" or "Also describe the overlap of cracks and water leakage". Not limited to this, instructions including other viewpoints may be input.

[0042] When the user instructs button 512 using input device 115, image management unit 211 identifies, as target deformation information, the deformation information among the deformation information stored in storage unit 214 in which the vertex coordinates 314 are included in the deformation range. In FIG. 7, the deformation information corresponding to object 414 and the deformation information corresponding to object 415 are identified as the target deformation information.

[0043] Then, image management unit 211 stores in storage unit 218 a set including unique identification information for each deformation range, the coordinates of each vertex of the deformation range, the finding text content instruction text data input to area 513, and the file name of the target deformation information. A configuration example of the set stored in storage unit 218 is shown in FIG. 6.

[0044] Selection range ID 611 is unique identification information for each deformation range. In FIG. 6, a case where selection range IDs 611 are issued in order as 1, 2, 3,... is assumed, but it is not limited to this.

[0045] Coordinate information 612 is the coordinates of each vertex of the deformation range. As another method for setting coordinate information 612, a method using a heat map calculated from a portion where deformation information is concentrated or an automatic selection method using an AI model specialized for range selection may be used. Finding text content instruction text data 613 is the finding text content instruction text data input to area 513. Deformation information file 614 is the file name of the target deformation information.

[0046] When the user instructs the button 514, the generation unit 217 infers the finding statement and the individual determination rank using the set stored in the storage unit 218 and the AI model. The AI model used by the generation unit 217 is an AI model that has been pre-trained by machine learning or the like to output (estimate) the corresponding finding statement and individual determination rank when input with deformation information, deformation range, finding statement content instruction text data, etc. Therefore, the generation unit 217 inputs the deformation information specified by the deformation information file 614 in the set stored in the storage unit 218, the coordinate information 612 in the set stored in the storage unit 218, and the finding statement content instruction text data in the set stored in the storage unit 218 into the AI model and performs the arithmetic processing of the AI model to estimate the corresponding finding statement and individual determination rank. Then, the generation unit 217 stores the data set including the result of the estimation in the storage unit 219. A configuration example of the data set stored in the storage unit 219 is shown in FIG. 8.

[0047] The deformation information file 614 is the file name of the deformation information specified by the deformation information file 614 in the set stored in the storage unit 218. The selection range ID 611 is the selection range ID 611 in the set stored in the storage unit 218. The text 811 is the text including the result (finding statement and individual determination rank) inferred by the AI model for the set stored in the storage unit 218.

[0048] Then, the management unit 218 displays the text 811 in the data set stored in the storage unit 219 in the area 515 in the GUI 501 of FIG. 5. When the user instructs the button 516 using the input device 115, the management unit 218 copies the text 811 displayed in the area 515 for use in, for example, an inspection report.

[0049] The processing performed by the image processing apparatus 101 after displaying the GUI 501 of FIG. 5 will be described according to the flowchart of FIG. 9. Since the details of the processing in each step shown in FIG. 9 are as described above, they will be briefly described below.

[0050] In step S901, when the user instructs button 511 using input device 115, image management unit 211 accepts a range designation operation for the captured image displayed in area 413.

[0051] In step S902, image management unit 211 accepts input of finding text data for area 513. Then, when the user instructs button 512 using input device 115, image management unit 211 identifies the target change information and stores in storage unit 218 a set including the identification information of the change range received in step S901, the coordinates of each vertex of the change range, the finding text data received in step S902, and the file name of the target change information.

[0052] In step S904, when the user instructs button 514, generation unit 217 acquires the set stored in storage unit 218 and also acquires from storage unit 214 the change information specified by change information file 614 in the set.

[0053] In step S905, generation unit 217 determines whether the finding text data received in step S902 is invalid data such as NULL. If the user does not input finding text data for area 513 using input device 115, the finding text data is invalid data.

[0054] As a result of this determination, if the finding text data received in step S902 is invalid data such as NULL, the process proceeds to step S907. On the other hand, if the finding text data received in step S902 is not invalid data such as NULL, the process proceeds to step S906.

[0055] In step S906, the generation unit 217 inputs the deformation information specified by the deformation information file 614 in the set stored in the storage unit 218, the coordinate information 612 in the set stored in the storage unit 218, and the finding text content instruction text data in the set stored in the storage unit 218 into the AI model and performs the arithmetic processing of the AI model, thereby estimating the corresponding finding text and the individual determination rank. Then, the generation unit 217 stores a data set including the result of the estimation in the storage unit 219 in association with the selection range ID in the set stored in the storage unit 218.

[0056] In step S907, the generation unit 217 inputs the deformation information specified by the deformation information file 614 in the set stored in the storage unit 218 and the coordinate information 612 in the set stored in the storage unit 218 into the AI model and performs the arithmetic processing of the AI model, thereby estimating the corresponding finding text and the individual determination rank. In this case, the AI model performs the same processing as in step S906 using invalid data as the finding text content instruction text data. Then, the generation unit 217 stores a data set including the result of the estimation in the storage unit 219 in association with the selection range ID in the set stored in the storage unit 218.

[0057] In step S908, the management unit 218 causes the text 811 in the data set stored in the storage unit 219 to be displayed in the area 515 in the GUI 501 of FIG. 5. Note that the display method of the text 811 is not limited to a specific display method.

[0058] [Second Embodiment] In the following embodiments including this embodiment, differences from the first embodiment will be described, and unless otherwise specifically mentioned below, it is assumed that they are the same as the first embodiment. In this embodiment, the finding text and the individual determination rank are estimated in consideration of the captured image. In this embodiment, instead of the GUI 501 in FIG. 5, the GUI 1001 in FIG. 10 is used.

[0059] In this embodiment, the AI model used by the generation unit 217 is an AI model that has been pre-trained by machine learning or the like so as to output (estimate) a corresponding finding sentence and an individual determination rank when input with deformation information, a deformation range, finding sentence content instruction text data, and imaging images of past deformations in addition. This finding sentence includes the analysis result of the image with respect to the imaging image of the deformation and a sentence explaining the analysis result.

[0060] When the user checks the checkbox 1011 using the input device 115 in the GUI 1001, the generation unit 217 inputs the deformation information specified by the deformation information file 614 in the set stored in the storage unit 218, the coordinate information 612 in the set stored in the storage unit 218, the finding sentence content instruction text data in the set stored in the storage unit 218, and the imaging image displayed in the region 413 into the AI model and performs the arithmetic processing of the AI model, thereby estimating the corresponding finding sentence and individual determination rank. When the checkbox 1011 is not checked, the generation unit 217 operates in the same manner as in the first embodiment.

[0061] As described above, according to this embodiment, it becomes easier to generate a finding sentence and an individual determination rank in consideration of the material, color, etc. of the structure included in the imaging image, and it becomes possible to describe the finding sentence and make an individual determination more suitable for the inspection report.

[0062] [Third Embodiment] In this embodiment, when the accuracy of the finding sentence and individual determination rank estimated using the AI model is low or there are deficiencies in the content, the finding sentence and individual determination rank are regenerated. The processing performed by the image processing apparatus 101 after displaying the GUI 501 in FIG. 5 will be described according to the flowchart in FIG. 11. In FIG. 11, the same step numbers as those of the processing steps shown in FIG. 9 are assigned to the processing steps similar to the processing steps shown in FIG. 9, and the description related to the processing steps is omitted.

[0063] As a result of checking the text 811 displayed on the output device 116, if the user wants to correct the content or expression, the user re-enters the finding text content instruction text data into the area 513. Therefore, in step S1109, the image management unit 211 accepts this re-entry.

[0064] When the user instructs the button 512 using the input device 115, the image management unit 211 identifies the target change information as in the first embodiment, and stores in the storage unit 218 a set of the identification information of the change range received in step S901, the coordinates of each vertex of the change range, the finding text content instruction text data received in step S1109, and the file name of the target change information.

[0065] In step S1110, when the user instructs the button 514, the generation unit 217 acquires the set stored in the storage unit 218 and acquires from the storage unit 214 the change information specified by the change information file 614 in the set. Then, the generation unit 217 determines whether the finding text content instruction text data received in step S1109 is invalid data such as NULL. As a result of this determination, if the finding text content instruction text data received in step S1109 is invalid data such as NULL, the process according to the flowchart of FIG. 11 ends. On the other hand, if the finding text content instruction text data received in step S1109 is not invalid data such as NULL, the process proceeds to step S1111.

[0066] In step S1111, the generation unit 217 inputs the change information specified by the change information file 614 in the set stored in the storage unit 218, the coordinate information 612 in the set stored in the storage unit 218, the finding text content instruction text data in the set stored in the storage unit 218, and the previous text 811 into the learned AI model and performs arithmetic processing of the AI model to estimate the corresponding finding text and the individual determination rank. Then, the generation unit 217 stores (overwrites) in the storage unit 219 the data set including the result of the estimation in association with the selection range ID in the set stored in the storage unit 218.

[0067] After that, in the same way as step S908 described above, the management unit 218 causes the text 811 in the dataset stored in the storage unit 219 to be displayed in the area 515 in the GUI 501 of FIG. 5.

[0068] [Fourth Embodiment] In this embodiment, when the image processing apparatus 101 generates a finding sentence or an individual determination rank by an AI model, a method for limiting the type of abnormality to be the target for generating the finding sentence or the individual determination rank and for selecting the level of detail of the content for each type of abnormality will be described.

[0069] By limiting the type of abnormality, the user can generate a finding sentence or an individual determination rank only for the abnormality that the user wants to describe in the inspection report. Also, by selecting the level of detail of the content, it is possible to instruct the AI model as to which finding sentence of which type of abnormality to specifically write, and it becomes possible to generate a finding sentence and an individual determination rank of the content more desired by the user. For example, when increasing the level of detail for cracks, findings of cracks with a short length are also included in the finding sentence. Also, when decreasing the level of detail for water leakage, findings of minor water leakage such as a narrow range are not included in the finding sentence.

[0070] In this embodiment, instead of the GUI 501 of FIG. 5, the GUI 1201 of FIG. 12(a) is used. When the user instructs the button 1211 using the input device 115, the image management unit 211 causes the GUI 1202 illustrated in FIG. 12(b) to be displayed on the output device 116.

[0071] When the user checks the checkbox 1291 using the input device 115, the image management unit 211 includes "crack" in the types of abnormalities for which a finding sentence and an individual determination rank are generated.

[0072] When the user checks the checkbox 1291a using the input device 115, the image management unit 211 sets to generate a finding sentence and an individual determination rank for an abnormality in which the type of abnormality is "crack" and the width is "less than 0.01 mm".

[0073] When the user checks the checkbox 1291b using the input device 115, the image management unit 211 is set to generate a finding statement and an individual determination rank for a deformation whose type of deformation is "crack" and whose width is "0.01 mm or more and less than 0.5 mm".

[0074] When the user checks the checkbox 1291c using the input device 115, the image management unit 211 is set to generate a finding statement and an individual determination rank for a deformation whose type of deformation is "crack" and whose width is "0.5 mm or more".

[0075] Also, when the user checks the checkbox 1292 using the input device 115, the image management unit 211 includes "water leakage" in the types of deformations for which a finding statement and an individual determination rank are generated.

[0076] In addition, the button group 1222 has buttons for specifying the content detail level indicating how detailed the content of the finding statement is expressed and reflected in the finding statement - individual determination rank for each of the conditions "less than 0.01 mm", "0.01 mm or more and less than 0.5 mm", and "0.5 mm or more". In FIG. 12, the degree of content detail level is set to three levels: "roughly", "normally", and "in detail", but it may be divided into even finer levels or specified numerically. The user can use the input device 115 to instruct any button in the button 1222.

[0077] When the user instructs the button 1223 using the input device 115, the image management unit 211 sets the type of deformation and the content detail level for generating the finding statement and the individual determination rank to the type of deformation and the content detail level set in the GUI 1202 of FIG. 12(b), respectively. On the other hand, when the user instructs the button 1224 using the input device 115, the image management unit 211 discards the content set in the GUI 1202 of FIG. 12(b).

[0078] When the user instructs button 514, the generation unit 217 infers the findings text and the individual determination rank using the set stored in the storage unit 218 and the AI model. At this time, the generation unit 217 uses the variation information having the variation type 312 that matches the "variation type for generating the findings text and the individual determination rank", rather than all the variation information specified by the set.

[0079] That is, the generation unit 217 inputs the variation information having the variation type 312 that matches the "variation type for generating the findings text and the individual determination rank", the variation range and the findings text content instruction text data in the set stored in the storage unit 218, into the AI model and performs the arithmetic processing of the AI model, thereby estimating the corresponding findings text and the individual determination rank. At this time, the generation unit 217 ensures that the findings text corresponding to the set content detail level is output. For example, the generation unit 217 may use a learned AI model so that the findings text corresponding to the set content detail level is output.

[0080] [Fifth Embodiment] In the above embodiment, the user inputs the findings text content instruction text data as a character string using the input device 115. However, in this embodiment, the content for instructing what content to include in the findings text is selected from the menu screen. In this embodiment, instead of the GUI 501 in FIG. 5, the GUI 1301 in FIG. 13(a) is used.

[0081] When the user instructs button 1311 using the input device 115, the image management unit 211 causes the output device 116 to display the GUI 1302 illustrated in FIG. 13(b). In the GUI 1302, a list of options preset in advance as options for the content to be included in the findings of the variation is displayed together with check boxes. All the options in this list are options compliant with the policies of countries, local governments, etc.

[0082] The user can check one from the group of check boxes displayed on the GUI 1302 by using the input device 115. Then, when the user instructs the button 1322 using the input device 115, the image management unit 211 sets the option corresponding to the checked check box in the finding text content instruction text data, and displays the finding text content instruction text data in the area 513. On the other hand, when the user instructs the button 1323 using the input device 115, the image management unit 211 discards the content set in the GUI 1302.

[0083] Thus, according to the present embodiment, the user can set the finding text content instruction text data in a simpler method, and can more efficiently generate a finding text that conforms to the policies of countries, local governments, etc.

[0084] [Sixth Embodiment] In the above embodiment, the information regarding the structure to be inspected is not described as an input to the AI model, but information such as the type, shape, material, structural age, and location of the structure may also be used as an input to the AI model. This makes it possible to generate higher-quality finding texts and individual judgment ranks that take into account information such as the type and location of the structure. Thus, various types of information can be considered as the information input to the AI model to obtain the finding text and individual judgment rank, and therefore, the input to the above AI model is only an example. Note that, including this embodiment, in the above embodiments as well, the AI model is a model that has been pre-trained so as to obtain an output for an input, as is well known.

[0085] Also, in the above embodiment, a method of copying the generated finding text and individual judgment rank is described. However, the functions that the image processing apparatus 101 can provide are not limited to this. For example, the image processing apparatus 101 may have a function of downloading, to the terminal device that has accessed the image processing apparatus 101, "a partial image within the area set according to the user operation for the captured image displayed in the area 413" and "the text data of the generated finding text and individual judgment rank".

[0086] Moreover, the structures dealt with in the above embodiments are merely examples of objects for which deformation is to be detected. Therefore, the definition of "deformation" when the object for which deformation is to be detected is other than a structure is not limited to the above definition.

[0087] Also, the configuration and operation method of the GUI used in the above embodiments are merely examples and are not limited to a specific configuration or a specific operation method. For example, each GUI may be displayed as a separate window, or may be switched and displayed using tabs or the like within one window. Further, when the GUI is displayed on a touch panel screen, operations on the GUI can be realized by operations on the touch panel screen.

[0088] The numerical values, processing timings, processing order, processing entities, configuration / acquisition method / transmission destination / source / storage location of data (information), etc. used in the above embodiments are given as examples for the purpose of specific explanation and are not intended to be limited to such examples.

[0089] Also, some or all of the above-described embodiments may be used in appropriate combination. Also, some or all of the above-described embodiments may be selectively used.

[0090] (Other Embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0091] The invention of this specification includes the following image processing apparatus, image processing method, and computer program. (Item 1) An acquisition means for acquiring deformation information related to a deformation detected from a captured image and instruction information for instructing content to be included in a finding sentence of the deformation. A generation means for generating the finding sentence and the level of the change based on the change information and the output result of the learned model based on the instruction information An image processing apparatus characterized by comprising the same. (Item 2) The acquisition means according to item 1, wherein the acquisition means acquires change information related to a change included in a partial range in the captured image. (Item 3) The acquisition means according to item 1 or 2, wherein the acquisition means acquires, as the instruction information, text input in response to a user operation as text for instructing content to be included in the finding sentence of the change included in a partial range in the captured image. (Item 4) The image processing apparatus according to item 2 or 3, wherein the partial range is a range designated in response to a user operation on the captured image. (Item 5) The image processing apparatus according to item 2 or 3, wherein the partial range is a range detected from the captured image using a learned model so as to detect a change in the image. (Item 6) The acquisition means according to item 1 or 2, wherein the acquisition means acquires, as the instruction information, an option selected in response to a user operation from a list of options preset as options for content to be included in the finding sentence of the change. (Item 7) The generation means according to any one of items 1 to 6, wherein the generation means generates the finding sentence and the level as an output result of a learned model based on the change information, the instruction information, and the captured image. (Item 8) The generation means according to item 2 or 3, wherein the generation means generates the finding sentence and the level as an output result of a learned model based on the change information, the instruction information, the captured image, and information on the partial range. (Item 9) The image processing apparatus according to any one of Items 1 to 8, wherein the acquisition means acquires the re-input instruction information. (Item 10) The image processing apparatus according to any one of Items 1 to 9, wherein the generation means generates the finding sentence and the level as output results of a learned model based on the variation information related to the variation of the type specified according to a user operation, instruction information for instructing the content to be included in the finding sentence of the variation, and the degree of detail of the content specified according to a user operation. (Item 11) Furthermore, The image processing apparatus according to any one of Items 1 to 10, further comprising means for displaying text including the finding sentence and the level. (Item 12) Furthermore, The image processing apparatus according to any one of Items 1 to 11, further comprising means for downloading a partial image in the captured image and text including the finding sentence and the level to a terminal device. (Item 13) An image processing method performed by an image processing apparatus, an acquisition step in which an acquisition means of the image processing apparatus acquires variation information related to a variation detected from a captured image and instruction information for instructing the content to be included in the finding sentence of the variation; a generation step in which a generation means of the image processing apparatus generates the finding sentence and the level of the variation as output results of a learned model based on the variation information and the instruction information; An image processing method characterized by comprising: (Item 14) A computer program for causing a computer to function as each means of the image processing apparatus according to any one of Items 1 to 12.

[0092] The invention is not limited to the above embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the claims are attached to disclose the scope of the invention.

Explanation of Symbols

[0093] 211: Image Management Unit 213: Image Analysis Unit 214: Storage Unit 217: Generation Unit 218: Storage Unit 219: Storage Unit 220: Management Unit

Claims

1. An acquisition means for acquiring deformation information related to a deformation detected from a captured image and instruction information for instructing content to be included in a finding sentence of the deformation; A generation means for generating the finding sentence and the level of the deformation as an output result of a learned model based on the deformation information and the instruction information; An image processing apparatus comprising the same.

2. The image processing apparatus according to claim 1, wherein the acquisition means acquires deformation information related to a deformation included in a partial range in the captured image.

3. The image processing apparatus according to claim 1, wherein the acquisition means acquires, as the instruction information, text input in response to a user operation as text for instructing content to be included in a finding sentence of a deformation included in a partial range in the captured image.

4. The image processing apparatus according to claim 2, wherein the partial range is a range designated in response to a user operation on the captured image.

5. The image processing apparatus according to claim 2, wherein the partial range is a range detected from the captured image using a learned model trained to detect deformations in the image.

6. The image processing apparatus according to claim 1, wherein the acquisition means acquires, as the instruction information, an option selected in response to a user operation from a list of options preset as options for content to be included in a finding sentence of a deformation.

7. The image processing apparatus according to claim 1, wherein the generation means generates the finding sentence and the level as an output result of a learned model based on the deformation information, the instruction information, and the captured image.

8. The image processing apparatus according to claim 2, wherein the generation means generates the finding sentence and the level as an output result of a learned model based on the deformation information, the instruction information, the captured image, and information on the partial range.

9. The image processing apparatus according to claim 1, wherein the acquisition means acquires the re-input instruction information.

10. The generation means generates the finding text and the level based on the deformation information related to the type of deformation specified according to the user operation, the instruction information for instructing the content to be included in the finding text of the deformation, and the detail level of the content specified according to the user operation, as the output result of the learned model. The image processing apparatus according to claim 1, characterized in that.

11. Furthermore, The image processing apparatus according to claim 1, further comprising means for displaying text including the finding text and the level.

12. Furthermore, The image processing apparatus according to claim 1, further comprising means for downloading a partial image in the captured image, text including the finding text and the level, to a terminal device.

13. An image processing method performed by an image processing apparatus, An acquisition step in which the acquisition means of the image processing apparatus acquires deformation information related to a deformation detected from a captured image and instruction information for instructing the content to be included in the finding text of the deformation; A generation step in which the generation means of the image processing apparatus generates the finding text and the level of the deformation as an output result of a learned model based on the deformation information and the instruction information. An image processing method characterized by comprising:

14. A computer program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 12.

Images