Display control apparatus, control method, medium, and program product

Abstract

A display control apparatus, a control method, a medium, and a program product. The display control apparatus includes: a management unit for managing defect information related to a plurality of defects occurring in a target object; and a display control unit for displaying information on one or more defects corresponding to the partial region of the target object based on the defect information. The defect information includes shape information, identification information, and additional information of each of the plurality of defects. The display control unit displays, based on the shape information, one or more defect objects respectively corresponding to the one or more defects, one or more first tags for indicating identification information respectively corresponding to the one or more defects, and a second tag that does not overlap with other tags, and not displaying at least one second tag that overlaps the other tags.

Description

Technical Field

[0001] This disclosure generally relates to display control devices, control methods, media, and program products, and particularly to techniques for displaying information related to defects in images of inspected targets. Background Technology

[0002] Defects (cracks and leaks, etc.) can occur on the concrete surfaces of bridges or buildings due to various factors. In recent years, systems that automatically detect defects by inputting inspection images of concrete surfaces taken with a camera into a computer have become increasingly common, and the digitization of inspection has developed. Japanese Patent Application Publication No. 2024-000324 discloses a method for managing defect detection results as defect information by assigning identification information specific to individual defects detected from inspection images.

[0003] Inspectors need to display inspection images and defect information on a computer screen to confirm the validity of defect detection results. However, when multiple defects exist at a high density within the area of ​​the inspection image to be displayed, there is a problem of degraded readability of the identification information / additional information of individual defects. Summary of the Invention

[0004] This disclosure provides display control techniques for improving the readability of information related to multiple defects in an inspection image.

[0005] According to one aspect of the present invention, a display control device includes: a management unit for managing defect information relating to a plurality of defects occurring in a target object; and a display control unit for displaying information relating to one or more defects corresponding to a portion of the target object based on the defect information, wherein the defect information includes shape information, identification information, and additional information of each of the plurality of defects, and the display control unit displays one or more defect objects corresponding to one or more defects respectively on a display unit based on the shape information, displays one or more first labels corresponding to one or more defects for indicating identification information on the display unit, and displays, on the display unit, one or more second labels corresponding to one or more defects for indicating additional information, a second label that does not overlap with the first or second labels of other defects on the display unit, and at least one second label that overlaps with the first or second labels of other defects on the display unit.

[0006] According to another aspect of the present invention, a display control method for displaying defects occurring in a target object, the control method comprising: obtaining defect information relating to a plurality of defects occurring in the target object; and displaying, on a display unit, information relating to one or more defects corresponding to a partial area of ​​the target object based on the defect information, wherein the defect information includes shape information, identification information, and additional information of each of the plurality of defects, and in the display, displaying, on the display unit, one or more defect objects corresponding to one or more defects respectively, on the display unit, displaying, on the display unit, one or more first labels corresponding to one or more defects respectively for indicating identification information, and among one or more second labels corresponding to one or more defects respectively for indicating additional information, displaying, on the display unit, second labels that do not overlap with the first labels or second labels of other defects, and not displaying, on the display unit, at least one second label that overlaps with the first labels or second labels of other defects.

[0007] The features of this disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings. The following description of the embodiments is given by way of example. Attached Figure Description

[0008] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the specification, serve to explain the principles of the embodiments.

[0009] Figure 1 This is a diagram illustrating the hardware configuration of the display control device.

[0010] Figure 2 This is a diagram illustrating the functional configuration of the display control device.

[0011] Figure 3 It is a diagram that describes defect information.

[0012] Figure 4 This is an example diagram illustrating a defect display screen.

[0013] Figure 5 This is a diagram describing the method used to display defect IDs / attached defect information.

[0014] Figure 6 It is a diagram describing the control when the defect ID and additional defect information are displayed.

[0015] Figure 7 This is a flowchart used for display.

[0016] Figure 8 It is a diagram that describes a variation of the display method. Detailed Implementation

[0017] In the following, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claims. Several features are described in the embodiments, but not all such features are required, and several such features can be appropriately combined. Furthermore, in the drawings, the same reference numerals are given the same or similar configuration, and redundant descriptions are omitted.

[0018] First Embodiment

[0019] As a first embodiment of the display control device according to the present invention, an information processing device for controlling the display of information (defect object / identification information / additional information) related to defects in an image (inspection image) of an inspected target object will be described below as an example.

[0020] Equipment Configuration

[0021] Figure 1 This is a diagram illustrating the hardware configuration of an information processing device. In the embodiments described below, an example using a general-purpose computer (PC) as the information processing device will be described.

[0022] Computer 101 includes a control unit 111, volatile memory 112, non-volatile memory 113, storage device 114, input device 115, output device 116, communication device 117, and system bus 118. A graphics processing unit (GPU) 151 may be added as needed.

[0023] The control unit 111 includes an arithmetic processing processor, such as a central processing unit (CPU), that integrates and controls the entire computer. Volatile memory 112 is random access memory (RAM) that temporarily stores programs and data supplied from external devices, etc. Non-volatile memory 113 is read-only memory (ROM) that stores programs and parameters to be executed by the processor of the control unit.

[0024] Storage device 114 is a storage unit such as flash memory or hard disk that is built into computer 101 or externally connected to computer 101. Storage device 114 can be configured by a combination of storage medium such as DVD and a drive that reads data from / writes data to storage medium.

[0025] Input device 115 is an operating unit such as a mouse, keyboard, or touch panel that receives user operations and transmits operation commands to control unit 111. Output device 116 is a display device such as a monitor or display screen that displays the processing results of the software executed by control unit 111. Communication device 117 is connected to the Internet or local area network (LAN) and communicates with external devices. System bus 118 includes an address bus, a data bus, and a control bus that connect the various components of computer 101, enabling data exchange.

[0026] Non-volatile memory 113 stores the Basic Input / Output System (BIOS) used to activate computer 101 and control various hardware components. Upon activation, the computer reads and executes the BIOS, thereby activating the operating system (OS) installed on storage device 114.

[0027] In this embodiment, the information processing of the computer 101 is implemented by the control unit 111 executing an OS or additional software installed in the OS. Multiple computers 101 can be used as needed. In this case, a server-client configuration can be adopted. For example, the information processing involving a large amount of computation can be performed by the server computer, and the results of the information processing can be viewed on the client computer via a browser or the like. Alternatively, a web application provided by the server computer can be executed by the client computer.

[0028] Figure 2 This is a diagram illustrating the functional configuration of an information processing device. As described above, each function (information processing) of the information processing device is implemented by the control unit 111 executing the OS or additional software installed in the OS.

[0029] Image storage unit 211 stores user-uploaded images (inspected images). Image management unit 212 uniquely identifies and manages the image files stored in image storage unit 211. For example, image management unit 212 assigns a Universally Unique Identifier (UUID) to the image files stored in image storage unit 211 and manages filenames, registration dates, and times, etc.

[0030] Image analysis unit 213 performs image analysis on image files stored in image storage unit 211 and detects structural defects. An example of the image analysis technique is a method for detecting and outputting defects from an input image using an artificial intelligence (AI) model pre-learned through machine learning; however, any other technique can be employed. For high-speed inference processing by AI, GPU 151 can be configured to be used.

[0031] Image analysis result storage unit 214 stores the analysis results of image analysis unit 213. For example, defects detected in the inspection image are classified according to various defect types (cracks and leaks, etc.), and the defects are stored as shape data such as polylines or polygons.

[0032] The image analysis result management unit 215 manages the association between the images stored in the image storage unit 211 and the analysis results stored in the image analysis result storage unit 214. Note that even when an image is analyzed multiple times, the image analysis result management unit 215 can collectively associate an image with multiple analysis results.

[0033] The user interface (UI) control unit 216 displays data (images and image analysis results) managed and stored by the various functional units described above on a display unit, which serves as an output device 116. The UI, used for receiving user input, is displayed on the display unit. The UI control unit 216 can be implemented by a local application of the computer 101 or by a web application provided by a server executed by the browser of the computer 101.

[0034] Defect information and its display

[0035] Examples of defect types in concrete surface inspection include cracks, leaks, rust spots, exposed rebar, and weathering. The following describes how individual defects are assigned identification information and managed as targets for “cracks” and “leaks” (which are representative defect types). Additional information (width, length, and area, etc.) is managed for individual defects.

[0036] Figure 3 This is a diagram describing the defect information stored in the image analysis result storage unit 214. Table 301 is a table storing defect information for multiple defects. Each record (each defect) in Table 301 includes the following information: identification information (ID) 311, defect type 312, defect ID 313, size information 314, and vertex coordinates 315.

[0037] ID 311 is identification information used to uniquely identify a defect. Defect type 312 stores the defect type, such as cracks or leaks. Defect ID 313 is identification information where defect type 312 and ID 311 are combined to improve readability. As described later, defect ID 313 is used as the string to be displayed on the defect label in the GUI display. That is, it is identification information used to help the user identify individual defects. Here, the prefix indicating defect type 312 (crack (CR) and leak (WL), etc.) is combined with ID 311, but other expressions can be used.

[0038] Size information 314 relates to the actual size of the defect. For defects represented by polylines (linear defects such as cracks), the length and width of the polyline (e.g., the maximum opening width of the crack) are stored. For defects represented by polygons (defects with two-dimensional expansion such as leaks), the maximum width of the polygon in the horizontal direction and the maximum height in the vertical direction are stored. Size information is not limited to this and can store more detailed information or other size information. Vertex coordinates 315 are the shape information of the defect and are the vertex coordinates of the polyline / polygon.

[0039] In image analysis of captured images (inspected images), a pixel coordinate system is used to process the captured images in pixels. However, it is also possible to convert to the actual size based on the image resolution (mm / pixel) of the captured object.

[0040] Figure 4 This is an example diagram illustrating a defect display screen (GUI) that the UI control unit 216 will display on the display unit. GUI 400 is a GUI used to present inspection images and defect detection results to the user.

[0041] The original image 401 is a portion of the inspection image obtained from the image storage unit 211. The polyline 402 is a diagram of crack defects drawn using polylines. The polygon 403 is a diagram of leakage defects drawn using polygons. By overlaying one or more defect objects (polylines 402 and polygons 403) onto the original image 401 as a background image, the user can visually confirm the location and size of the detected defects in the image.

[0042] The zoom level display 411 is a UI component that indicates the zoom level of the displayed image. Buttons 412 and 413 are UI components that control zooming in and out. That is, by pressing (clicking on a mouse or touching) buttons 412 and 413, the user can change the display level of the original image 401. The display in the zoom level display 411 is changed accordingly.

[0043] Checkbox 421 is a UI component that overlays polylines 402 and polygons 403 to turn defects on / off for various defect types. Since the severity of a crack defect is roughly known through its width, therefore... Figure 4 The checkboxes shown can be turned on / off for each crack width.

[0044] Checkbox 422 is a UI component that toggles the defect label on / off. (Example) Figure 4 As shown, a checkbox 423 can be further provided for turning on / off the overlay of additional defect information.

[0045] Figure 5These are diagrams describing methods for displaying defect IDs / additional defect information. Screen 500 illustrates a method for determining the display position of the information label when only polylines exist, and screen 510 illustrates a method for determining the display position of the information label when only polygons exist.

[0046] Screen 500 is a diagram illustrating the arrangement when an information label is displayed for a polyline 501 indicating a crack. Point 502 is the midpoint of the polyline 501. For example, when the polyline comprises five segments (line segments and arcs), the midpoint of the third segment at the center can be set as the midpoint of the polyline. Alternatively, the midpoint can be determined by any method such as calculating the total length of the polyline and defining the position of half the length of the polyline as the midpoint of the polyline.

[0047] Region 503 is the label display area for the defect ID. Since the width and height of region 503 are determined by the number of characters in the defect ID and the font used for display, in this example, region 503 of the defect ID label is centered at point 502. In this case, region 503 is arranged such that its center in the width / height direction coincides with point 502 (the midpoint). The arrangement of region 503 can be determined by any method, such as arranging the left and top edges of region 503 to coincide with point 502 (the midpoint).

[0048] Area 504 is the label display area for additional defect information. Similar to area 503 described above, the width and height of area 504 are determined by the number of characters in the additional defect information and the font used for display. The example of screen 500 assumes that the "left edge of area 503" and the "left edge of area 504" are aligned in the same position and positioned directly below area 503 (so as not to overlap). Note that the positions of the "center of area 504" and the "center of area 503" can be aligned with each other, or area 504 can be positioned at the right edge of area 503. The arrangement of areas 503 and 504 can be determined by any method.

[0049] Screen 510 is a diagram illustrating the arrangement of information labels superimposed on a polygon 511 indicating a leak. Point 512 is the center point of polygon 511. For example, the circumscribed rectangle of the polygon can be obtained, and the center point of the circumscribed rectangle can be set as 511. However, since defects of various shapes can appear in the structure, the center point can be determined by any method such as using the centroid of the polygon as the center point. Region 513 is the label display area for the defect ID, and region 514 is the label display area for additional defect information. How to determine the size and arrangement of regions 513 and 514 is similar to that of regions 503 and 504 described above, therefore their description will be omitted.

[0050] Screen 520 illustrates an example of information label display when polylines and polygons overlap. Label display 521 is a label display for the defect ID of polyline 501, and label display 522 is a label display for the defect ID of polygon 511. Label display 521 is displayed in a location similar to area 503 on screen 500, and label display 522 is displayed in a location similar to area 513 on screen 510.

[0051] Because label display 521 and label display 522 are positioned close to each other, they partially overlap. Here, it is assumed that the priority for the defect type was previously specified as "crack > leak". Therefore, label display 521 appears in front of label display 522.

[0052] Therefore, the order of drawing processing on screen 520 (from back to front) is: original image 401, polygon 511 (leakage), polyline 501 (crack), label display 522 (leakage defect ID), and label display 521 (crack defect ID). This can be controlled so that information on defect types with higher priority is displayed in front.

[0053] Figure 6 This is a diagram describing the control details when displaying the defect ID and additional defect information. For example, display screen 600 illustrates a display example in the GUI (FIG.4) when the zoom level is increased to 200%.

[0054] Label display 601 is a label display for additional defect information of a polyline, and label display 602 is a label display for additional defect information of a polygon. Here, the positions of the various label displays are determined similarly to those described in reference frames 500 and 510. An example of displaying size information 314 as additional defect information is shown.

[0055] As shown in the figure, in screen 600, since label display 601 is displayed at the forefront, there is no problem with readability. On the other hand, part of label display 602 is hidden by the "label display of the polyline defect ID," and there are readability problems (the characters are unreadable or difficult to see). Displaying an excessive amount of unnecessary information to the user, as in screen 600, degrades usability. Unlike the defect ID, the additional defect information is not necessary.

[0056] Therefore, in cases where label display overlap occurs, as shown in screen 600, the display is controlled as shown in screen 610. Specifically, if the label display 602 hidden by the overlapping portion is low-importance information (additional defect information), label display 602 is not displayed. This provides a display screen with high readability.

[0057] Equipment operation

[0058] Figure 7 This is a flowchart of the display control processing in the information processing device. Note that the display control processing is performed by the UI control unit 216, which is implemented by the control unit 111 executing the OS or additional software installed in the OS. Note that before processing, the UI control unit 216 obtains the original image and the corresponding analysis results (Table 301) in advance from the image storage unit 211 and the image analysis result storage unit 214. Assume that the original image is displayed on the GUI 400.

[0059] In S701, the UI control unit 216 determines whether a change in the display area (display position and / or display magnification) of the original image 401 has been detected. A change in the display area of ​​the original image can be received by user interaction with UI components on the GUI 400 (such as dragging the original image 401 and pressing buttons 412 and 413). If yes, processing proceeds to S702 to update the display; otherwise, processing ends.

[0060] In S702, the UI control unit 216 selects only the defect information whose coordinates fall within the display area from the multiple defect information included in Table 301 and inputs this defect information into the workpiece array. In the following steps, the defect information included in the workpiece array is processed.

[0061] In S703, the UI control unit 216 sorts the workpiece array according to its priority in ascending order (lower priority towards the head of the array) using defect information in the workpiece array. Here, it is assumed that priorities are assigned according to defect type 312 as described above, but other criteria can be used to determine priorities. For example, defect information can be analyzed, and the order of defect progression can be used (in descending order of crack width, in descending order of total crack extension, and in the case of surface shape, in descending order of area, etc.).

[0062] In S704, the workpiece array sorted in S703 is processed cyclically from the head to the tail (loop 1). In S704, the UI control unit 216 displays the "defect objects (polylines, polygons)" of defects in the workpiece array, and displays the "defect ID label display" sequentially in the order of the workpiece array (i.e., in ascending order of priority). Thus, the labels of the defect IDs of defects with higher priority are displayed relatively first. Note that the "defect objects (polylines, polygons)" and the "defect ID label display" can be displayed in parallel sequentially.

[0063] In S705, the UI control unit 216 sorts the workpiece array in descending order of priority (higher priority towards the head of the array) using defect information in the workpiece array.

[0064] In S706 to S708, the workpiece array sorted in S705 is processed in a loop from the head to the tail (loop 2).

[0065] In S706, the UI control unit 216 confirms the display area (label display position / size) of the "label display for additional defect information" in the order of the workpiece array (i.e., in descending order of priority). For example, the additional defect information is the size information in Table 301, but it can be other additional information. See reference... Figure 5 As described in areas 504 and 514, the display area (the location / size of the label display) for "label display of additional defect information" is determined.

[0066] In S707, the UI control unit 216 determines whether the display area for additional defect information overlaps with other already displayed labels ("label display of defect ID" or "label display of additional defect information"). For example, Figure 6 Label display 601 is determined to be "non-overlapping", and label display 602, determined after label display 601, is determined to be "overlapping". If not, the process proceeds to S708; if yes, S708 is skipped. In S708, UI control unit 216 displays "label display of additional defect information".

[0067] As a result of the aforementioned display control, the label display for defect IDs allows for overlapping of labels to show all defects in the workpiece array. However, since the labels for defect IDs of high-priority defects are displayed relatively first, deterioration in readability is less likely to occur with labels for defect IDs of high-priority defects.

[0068] Regarding the display of labels for additional defect information, only defects that do not overlap in label display and those with relatively high priority among those that do overlap in label display are displayed. In other words, complex displays that degrade readability can be suppressed.

[0069] As described above, according to the first embodiment, when displaying defect labels, display control is performed based on the defect priority and the type of label display. This allows for the display of defect information that can suppress readability degradation.

[0070] Note that in S707 above, only it is determined whether the display area of ​​the additional defect information overlaps with other already displayed labels, but the degree of overlap is calculated, and the presence or absence of the display can be controlled based on whether the degree of overlap is equal to or less than a threshold. The overlap of label displays can be toggled based on user specifications.

[0071] When the display magnification of the screen in the GUI is increased beyond a predetermined magnification, additional defect information that was not displayed before the magnification can be displayed in a position that does not overlap with other defect IDs and additional defect information. When the display magnification of the screen in the GUI is decreased to a level exceeding a predetermined magnification, additional defect information that was already displayed before the decrease can be hidden.

[0072] Variations

[0073] In the first embodiment described above, the "label display of defect ID" and "label display of additional defect information" are examples of non-transmissive label displays. However, transmissive label displays can also be used.

[0074] Figure 8 It is a diagram describing the variation of the label display.

[0075] Screen 800 illustrates an example where the label display (rectangular area) is a transmission image. Thus, while displaying labels 801 and 802 showing the defect IDs, the user can confirm the original image used as the background and the defect object (polyline or polygon).

[0076] Screen 810 illustrates an example where the character outlines of the displayed string 811 on the label are bordered by a different color. Specifically, the character outlines are bordered by a color different from the character color to improve the readability of the label's characters. This improves the problem of the displayed label string being assimilated into the background and difficult to see when the original image used as the background is a black object, etc.

[0077] Screen 820 illustrates an example, besides screen 800, that uses not only strings but also graphic elements such as icons as labels. Defect objects (polylines / polygons) are displayed in predetermined colors. For example, the color of the defect shape and icon is set according to the width of the crack.

[0078] Solid line 821 is a polyline indicating a crack of 1.0 mm or larger, and is shown in red. Solid line 822 is a polyline indicating a crack of 0.5 mm or larger but smaller than 1.0 mm, and is shown in orange. Icon 831 is a red icon attached to the defect ID label corresponding to solid line 821. Icon 832 is an orange icon attached to the defect ID label corresponding to solid line 822. That is, by coloring the label display of defect objects and defect IDs according to the colors indicated in the legend, the relationship between defect objects and defect IDs is shown in an easily understandable way.

[0079] The format of the label display (defect ID / additional defect information) is not necessarily limited to strings, and various display styles and graphics can be used as needed.

[0080] The character color, background color, or color block of the defect label can be variable. Furthermore, priority can be determined based on multiple categories specified in the legend (crack width, defect type, etc.). When the mouse cursor is overlaid (hovered) on the defect label, the corresponding defect object can be highlighted.

[0081] Embodiments of the present invention can also be implemented by providing software (including computer program products of computer programs) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU) or microprocessor unit (MPU) of the system or device) reads and executes the computer program.

[0082] While this disclosure has been described with reference to embodiments, it should be understood that this disclosure is not limited to the disclosed embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. A display control device, comprising: The management unit is used to manage defect information related to multiple defects that occur in the target object; as well as A display control unit is configured to display information related to one or more defects corresponding to a portion of the target object, based on the defect information. The defect information includes the shape information, identification information, and additional information of each of the plurality of defects, as well as... The display control unit: Based on the shape information, display one or more defect objects corresponding to the one or more defects respectively. Display one or more first tags for indicating identification information corresponding to each of the one or more defects, and In one or more second tags used to indicate additional information corresponding to the one or more defects respectively, a second tag that does not overlap with the first and second tags of other defects is displayed, and at least one second tag that overlaps with the first or second tags of other defects is not displayed.

2. The display control device according to claim 1, further comprising: The acquisition unit is used to acquire multiple images of the target object that have been captured. The display control unit displays one or more images corresponding to the partial area, and overlays the one or more defective objects, the one or more first labels, and the one or more second labels onto the one or more images.

3. The display control device according to claim 1, further comprising: The changing unit is used to receive changes in the said partial region. When the changing unit receives a change in the partial area, the display control unit updates the display.

4. The display control device according to claim 1, further comprising: A priority specification unit is used to receive priority specifications for the plurality of defects. The display control unit: The first label is displayed sequentially in ascending order of defects, and The display area of ​​the corresponding second label is determined in descending order of defects, and the overlap with the first or second label of other defects is determined.

5. The display control device according to claim 4, wherein, The priority is based on at least one of the defect type and the degree of defect progress.

6. The display control device according to claim 1, wherein, The display control unit: The display area of ​​the first label is determined based on the shape information of the corresponding defect, and the display area of ​​the second label is determined based on the display area of ​​the first label.

7. The display control device according to claim 1, wherein, The display control unit displays the defect object and the first label corresponding to the same defect in the same color.

8. The display control device according to claim 1, wherein, The display control unit displays the defective object and the first label in different colors according to the degree of defect progression.

9. A control method for displaying defects, said defects occurring in a target object, the control method comprising: Obtain defect information related to multiple defects occurring in the target object; as well as Based on the defect information, information related to one or more defects corresponding to a portion of the target object is displayed on the display unit, wherein... The defect information includes the shape information, identification information, and additional information of each of the plurality of defects, as well as... In the display, Based on the shape information, one or more defect objects corresponding to the one or more defects are displayed on the display unit. The display unit displays one or more first tags indicating identification information corresponding to the one or more defects, respectively. In one or more second labels used to indicate additional information corresponding to the one or more defects respectively, a second label that does not overlap with the first and second labels of other defects is displayed on the display unit, and at least one second label that overlaps with the first or second labels of other defects is not displayed on the display unit.

10. A computer-readable storage medium storing a program for causing a computer to perform the control method according to claim 9.

11. A computer program product comprising a program for causing a computer to perform the control method according to claim 9.

Images