Display method and program

Abstract

To provide a display method that helps to photograph a still image satisfying a prescribed condition.SOLUTION: A display device includes: a first display step of displaying images being taken by a photographing unit 27 during an operation of a first mode for taking a low end part in a vertical direction of an elongated structure deteriorating due to rust by the photographing unit 27, and a guide for including the low end part in a photographing range, on a display part 22; and an output step of, when a prescribed trigger condition is satisfied, outputting still images taken by the photographing unit 27.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a display method and a program. [Background technology]

[0002] A technology relating to the maintenance of infrastructure facilities has been proposed. Patent Document 1 discloses a report management device that can effectively utilize reports relating to infrastructure facilities such as street lights. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-117878 Summary of the Invention [Problem to be solved by the invention]

[0004] When managing a structure together with a still image showing the structure, the still image may need to meet certain requirements, such as showing a specific part of the structure.

[0005] The present invention provides a display method and the like that can assist in taking a still image that meets predetermined requirements. [Means for solving the problem]

[0006] A display method according to one aspect of the present invention is a display method executed by an information terminal equipped with a photographing unit and a display unit, and includes a first display step of displaying on the display unit an image being photographed by the photographing unit and a guide for including the lower end in the vertical direction of a long structure that is deteriorating due to rust, during operation of a first mode for the photographing unit to photograph the lower end, and an output step of outputting a still image photographed by the photographing unit when a predetermined trigger condition is satisfied.

[0007] The program according to one aspect of the present invention is a program for causing the information terminal to execute the above-described display method.

Advantages of the Invention

[0008] The display method and the like of the present invention can assist in photographing a still image that satisfies predetermined requirements.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments will be specifically described with reference to the drawings. Note that all of the embodiments described below show comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present invention. In addition, among the components in the following embodiments, the components not described in the independent claims are described as optional components.

[0011] Note that each drawing is a schematic diagram and is not necessarily drawn precisely. Also, in each drawing, the same reference numerals are given to substantially the same configurations, and duplicate explanations may be omitted or simplified.

[0012] (Embodiment) [Configuration] First, the configuration of the information processing system according to the embodiment will be described. FIG. 1 is a block diagram showing the functional configuration of the information processing system according to the embodiment.

[0013] The information processing system 10 is a system that collects still images in which long structures such as utility poles and street lights (road lights) are reflected, and manages the state of the structures based on the collected still images. The information processing system 10 includes a plurality of information terminals 20, an image management system 30, a diagnosis system 40, a structure management system 50, and a local government terminal 60. Hereinafter, these devices or systems will be specifically described.

[0014] Each of the plurality of information terminals 20 is an information terminal owned by a general user (such as a citizen). A general user (hereinafter also simply referred to as a user) uses the information terminal 20 to capture a still image of a structure and transmits the image information of the captured still image to the image management system 30. FIG. 2 is a diagram showing an example of a state where a user uses the information terminal 20 to capture a street lamp (an example of a structure). The information terminal 20 is, for example, a portable information terminal such as a smartphone or a tablet terminal. The information terminal 20 includes an operation reception unit 21, a display unit 22, an information processing unit 23, a storage unit 24, a communication unit 25, a position measurement unit 26, and a photographing unit 27.

[0015] The operation reception unit 21 receives a user's operation. The operation reception unit 21 is realized, for example, by a touch panel.

[0016] The display unit 22 displays an image (a moving image or a still image). The display unit 22 is realized, for example, by a display panel such as a liquid crystal panel or an organic EL (Electro Luminescence) panel.

[0017] The information processing unit 23 performs processing such as displaying an image on the display unit 22 according to an operation received by the operation reception unit 21. The information processing unit 23 is realized, for example, by a microcomputer, but may also be realized by a processor. The functions of the information processing unit 23 are realized, for example, when a microcomputer or a processor constituting the information processing unit 23 executes a computer program stored in the storage unit 24.

[0018] The storage unit 24 is a storage device that stores a computer program executed by the information processing unit 23 and the like. The computer program stored in the storage unit 24 includes a dedicated application program (hereinafter also referred to as a dedicated app) for the information terminal 20 to access the image management system 30. The storage unit 24 is realized, for example, by a semiconductor memory.

[0019] The communication unit 25 is a communication circuit (communication module) for the information terminal 20 to communicate with the image management system 30 via a mobile communication network and a wide-area communication network such as the Internet. The communication unit 25 is, for example, a wireless communication circuit that performs wireless communication. The communication standard of the communication performed by the communication unit 25 is not particularly limited. The communication unit 25 may be a communication circuit that performs wireless communication according to a communication standard such as Wi-Fi (registered trademark).

[0020] The position measurement unit 26 measures the current position (coordinates) of the information terminal 20 and outputs position information indicating the measured current position. The position measurement unit 26 is realized by, for example, a GNSS (Global Navigation Satellite System) module such as a GPS (Global Positioning System) module.

[0021] The imaging unit 27 is a camera that captures an image (moving image or still image) in which the structure is reflected. The imaging unit 27 is realized by an image sensor, an optical system for imaging, and the like.

[0022] The image management system 30 receives, from the information terminal 20, image information including a still image in which the structure is reflected captured by the information terminal 20 and the position information of the information terminal 20 when the still image was captured, and stores (manages) the received image information. Further, the image management system 30 communicates with the diagnostic system 40 to transmit (provide) the image information to the diagnostic system 40. The image management system 30 is realized by one or more server devices (cloud servers), but may also be realized in a serverless manner.

[0023] Diagnostic system 40 receives image information from the image management system 30 and diagnoses the degree of deterioration of the structure shown in the still image included in the received image information. The diagnostic system 40 includes, for example, a machine learning model that can take an image as input and output a score indicating the degree of deterioration of the structure shown in the still image, and diagnoses the degree of deterioration of the structure using the machine learning model. Further, the diagnostic system 40 communicates with the structure management system 50 to transmit (provide) to the structure management system 50 the individual information of the structure including the image information and the diagnostic result of the degree of deterioration of the structure shown in the still image included in the image information. The diagnostic system 40 is realized by one or more server devices (cloud servers), but may also be realized in a serverless manner.

[0024] The structure management system 50 receives the individual information of the structure from the diagnostic system 40 and stores the structure management information (management ledger of multiple structures) obtained by aggregating a large number of the received individual information of the structure. Further, the structure management system 50 communicates with the local government terminal 60 to transmit (provide) to the local government terminal 60 the individual information of the structure designated by the local government terminal. The structure management system 50 is realized by one or more server devices (cloud servers), but may also be realized in a serverless manner.

[0025] The local government terminal 60 is an information terminal used by the staff of the local government. The staff of the local government can access the structure management system 50 using the local government terminal 60 to receive the provision of the individual information of the structures subject to the management of the local government from the structure management system 50. The local government terminal 60 is, for example, a stationary information terminal such as a personal computer, but may also be a portable information terminal such as a smartphone or a tablet terminal.

[0026] Here, the street lamp 70 shown in FIG. 2 will be described. The street lamp 70 is an example of a long-structured object (lighting device) deteriorated by rust, and is an example of an object to be managed by the structure management system 50. In the following embodiments, the object is described as the street lamp 70. The street lamp 70 includes a column 71, a lamp 72, and a cover 73 that is opened during maintenance (inspection). The cover 73 is, for example, a lid that is opened for a maintenance worker to access a power supply device (power box) that supplies power to the lamp 72 (light source).

[0027] [GUI Display Operation] As described above, the diagnosis system 40 diagnoses the degree of deterioration of the street lamp 70 reflected in the still image based on the still image taken by the information terminal 20 while the dedicated application is being executed. When the diagnosis system 40 diagnoses the degree of deterioration using the machine learning model, the accuracy of the diagnosis of the degree of deterioration depends on how the street lamp 70 is reflected in the still image. That is, the object needs to be properly reflected in the still image. However, since the still image is taken with the cooperation of a general user and is performed by the general user, there is a possibility that how the street lamp 70 is reflected in the still image will vary. That is, in order to improve the accuracy of the diagnosis of the degree of deterioration, it is an issue to suppress the variation in the quality of the still image taken by the information terminal 20 and improve the quality of the still image.

[0028] Therefore, the dedicated application (information processing unit 23) attempts to suppress the variation in the quality of the still image by superimposing a guide on the moving image being taken by the imaging unit 27 as a GUI (Graphical User Interface) for taking the still image. Hereinafter, such a GUI display operation will be described. FIG. 3 is a flowchart showing an example of the GUI display operation.

[0029] The information processing unit 23 of the information terminal 20 performs the operation in the first mode for the imaging unit 27 to image the lower end portion of the street lamp 70 in the vertical direction (S11). During the operation in the first mode, the information processing unit 23 displays, on the display unit 22, the moving image being imaged by the imaging unit 27 and a guide for including the lower end portion of the street lamp 70 in the imaging range. FIG. 4 is a diagram showing an example of the GUI displayed on the display unit 22 during the operation in the first mode.

[0030] As shown in FIG. 4, during the operation in the first mode, as a guide for including the lower end portion of the street lamp 70 in the imaging range, text and a dashed-line frame 22a are displayed. In the example of FIG. 4, in addition to the dashed-line frame 22a, a dashed-line frame 22b for including the boundary portion with the ground surface (that is, the lowermost end of the column 71) of the lower end portion in the imaging range is also displayed. The dashed-line frame 22a and the dashed-line frame 22b are an example of a guide for designating the position of the lower end portion of the street lamp 70 in the moving image.

[0031] Note that in step S11, at least one of the text and the dashed-line frame 22a may be displayed as a guide, and it is not essential that both the text and the dashed-line frame 22a are displayed. Also, as other guides, a semi-transparent color filter may be displayed within the imaging range or the like.

[0032] When an operation on the imaging button (at the lower part of the screen in FIG. 4) is received by the operation reception unit 21 during the operation in the first mode, the information processing unit 23 uses the imaging unit 27 to image a still image at the timing when the operation is received. Then, the information processing unit 23 generates image information in which the still image is associated with the position information of the information terminal 20 when the still image is taken, and stores the generated image information in the storage unit 24. That is, the information processing unit 23 outputs the image information to the storage unit 24. Note that the position information is measured by the position measurement unit 26.

[0033] Next, the information processing unit 23 performs an operation in the second mode for photographing the entire streetlight 70 by the photographing unit 27 (S12). During the operation in the second mode, the information processing unit 23 displays on the display unit 22 the moving image being photographed by the photographing unit 27 and a guide for including the entire streetlight 70 in the photographing range. FIG. 5 is a diagram showing an example of the GUI displayed on the display unit 22 during the operation in the second mode.

[0034] As shown in FIG. 5, during the operation in the second mode, as a guide for including the entire streetlight 70 in the photographing range, text and a dashed-line frame are displayed. In step S12, at least one of the text and the dashed-line frame may be displayed as the guide, and it is not essential that both the text and the dashed-line frame are displayed.

[0035] When an operation on the photographing button (at the lower part of the screen in FIG. 5) is received by the operation reception unit 21 during the operation in the second mode, the information processing unit 23 photographs a still image at the timing when the operation is received using the photographing unit 27. The information processing unit 23 generates image information associating the photographed still image with the position information of the information terminal 20 when the still image was photographed, and stores the generated image information in the storage unit 24. That is, the information processing unit 23 outputs the image information to the storage unit 24.

[0036] Next, the information processing unit 23 performs an operation in the third mode for photographing the lamp unit 72 by the photographing unit 27 (S13). During the operation in the third mode, the information processing unit 23 displays on the display unit 22 the moving image being photographed by the photographing unit 27 and a guide for including the lamp unit 72 in the photographing range. FIG. 6 is a diagram showing an example of the GUI displayed on the display unit 22 during the operation in the third mode.

[0037] As shown in FIG. 6, during the operation in the third mode, as a guide for including the lamp unit 72 in the photographing range, text and a dashed-line frame are displayed. In step S13, at least one of the text and the dashed-line frame may be displayed as the guide, and it is not essential that both the text and the dashed-line frame are displayed.

[0038] When an operation on the shooting button (at the bottom of the screen in FIG. 6) is received by the operation reception unit 21 during the operation in the third mode, the information processing unit 23 shoots a still image at the timing when the operation is received using the shooting unit 27. The information processing unit 23 generates image information in which the position information of the information terminal 20 when the still image is shot is associated with the shot still image, and stores the generated image information in the storage unit 24. That is, the information processing unit 23 outputs the image information to the storage unit 24.

[0039] Next, the information processing unit 23 performs an operation in the fourth mode for shooting the identification information of the lighting fixture 72 (hereinafter also referred to as the lighting fixture number) by the shooting unit 27 (S14). During the operation in the fourth mode, the information processing unit 23 displays, on the display unit 22, a moving image being shot by the shooting unit 27 and a guide for including the identification information of the lighting fixture 72 in the shooting range. FIG. 7 is a diagram showing an example of the GUI displayed on the display unit 22 during the operation in the fourth mode. The identification information of the lighting fixture 72 is shown, for example, on the support column 71.

[0040] As shown in FIG. 7, during the operation in the fourth mode, as a guide for including the identification information of the lighting fixture 72 in the shooting range, text and a dashed line frame are displayed. In step S14, it is sufficient that at least one of the text and the dashed line frame is displayed as a guide, and it is not essential that both the text and the dashed line frame are displayed.

[0041] When an operation on the shooting button (at the bottom of the screen in FIG. 7) is received by the operation reception unit 21 during the operation in the fourth mode, the information processing unit 23 shoots a still image at the timing when the operation is received using the shooting unit 27. The information processing unit 23 generates image information in which the position information of the information terminal 20 when the still image is shot is associated with the shot still image, and stores the generated image information in the storage unit 24. That is, the information processing unit 23 outputs the image information to the storage unit 24.

[0042] Next, the information processing unit 23 performs an operation in the fifth mode for photographing the cover 73 (also referred to as a lid) that is opened during maintenance by the photographing unit 27 (S15). During the operation in the fifth mode, the information processing unit 23 displays on the display unit 22 a moving image being photographed by the photographing unit 27 and a guide for including the cover 73 in the photographing range. FIG. 8 is a diagram showing an example of the GUI displayed on the display unit 22 during the operation in the fifth mode.

[0043] As shown in FIG. 8, during the operation in the fifth mode, as a guide for including the cover 73 in the photographing range, text and a dashed-line frame are displayed. Note that in step S15, at least one of the text and the dashed-line frame may be displayed as the guide, and it is not essential that both the text and the dashed-line frame are displayed.

[0044] When an operation on a photographing button (at the lower part of the screen in FIG. 8) is received by the operation reception unit 21 during the operation in the fifth mode, the information processing unit 23 photographs a still image at the timing when the operation is received using the photographing unit 27. The information processing unit 23 generates image information in which the position information of the information terminal 20 when the still image was photographed is associated with the photographed still image, and stores the generated image information in the storage unit 24. That is, the information processing unit 23 outputs the image information to the storage unit 24.

[0045] Next, the information processing unit 23 transmits (outputs) the image information (still images) generated and stored in steps S11 to S15 to the image management system 30 by communicating with the image management system 30 using the communication unit 25 (S16). FIG. 9 is a diagram showing an example of the GUI for transmitting the image information. In the example of FIG. 9, triggered by the operation on the photographing button (at the lower part of the screen in FIG. 9) being received by the operation reception unit 21, the information processing unit 23 transmits the image information to the image management system 30 using the communication unit 25. As a result, the still images photographed in steps S11 to S15 are uploaded to the image management system 30.

[0046] As described above, the dedicated application (information processing unit 23) can suppress variations in the quality of still images by superimposing a guide on a moving image being captured by the imaging unit 27 as a GUI for capturing still images.

[0047] Note that the order in which the first mode to the fifth mode shown in FIG. 3 are performed is an example and is not particularly limited. The order in which the first mode to the fifth mode are performed may be selected by the user, for example.

[0048] Also, it is not essential that five types of image information corresponding to the five modes of the first mode to the fifth mode be transmitted, and at least one type of image information may be transmitted. Since a still image showing the lower end of the street lamp 70 captured in the first mode is mainly used for diagnosing the degree of deterioration, at least the image information corresponding to the first mode may be transmitted.

[0049] Note that the lower end of the street lamp 70 is a part where deterioration such as rust is likely to occur due to urine of dogs and cats adhering or being covered with vegetation. In order to suppress the collapse of the street lamp 70, it is important to diagnose the degree of deterioration using a still image of the lower end of the street lamp 70.

[0050] [Modification Example 1: Processing related to re - imaging] In the above - described display operation of the GUI, although the guide is displayed on the display unit 22, whether a still image in which the street lamp 70 is reflected as per the guide is captured is left to the user. However, after a still image is captured, the information processing unit 23 determines whether the still image meets a predetermined requirement. If it is determined that the still image does not meet the predetermined requirement, the information processing unit 23 may request the user to re - capture the still image or may stop the output (storage in the storage unit 24) of the still image. Here, the predetermined requirement is a requirement regarding whether a still image in which the street lamp 70 is reflected as per the guide is captured.

[0051] For example, when the dashed-line frame 22a is displayed as a guide in the first mode, the information processing unit 23 performs edge detection (contour extraction) processing on the street lamp 70 shown in the captured still image. When the detected edge is located within the dashed-line frame 22a, the information processing unit 23 can determine that a still image in which the street lamp 70 is shown as per the guide has been captured. That is, the information processing unit 23 can determine that the captured still image satisfies a predetermined requirement.

[0052] In this case, the predetermined requirement is that the street lamp 70 fits within the dashed-line frame 22a, but it may be allowed for the street lamp 70 to protrude from the dashed-line frame 22a by a certain extent. That is, the predetermined requirement may be, for example, a requirement that the street lamp 70 does not protrude beyond a predetermined range from the guide within the still image.

[0053] When the information processing unit 23 determines that the captured still image satisfies the predetermined requirement, it ends the operation in the first mode. When the information processing unit 23 determines that the captured still image does not satisfy the predetermined requirement, it performs at least one of continuing the operation in the first mode and aborting the output of the still image.

[0054] Thereby, the image management system 30 can collect only the still images in which the street lamp 70 is shown as per the guide from the information terminal 20.

[0055] [Modification Example 2: Inputting the reason why the lower end cannot be photographed] When the lower end of the street lamp 70 is covered by vegetation, in the first mode, there may be a case where the information terminal 20 cannot photograph the boundary between the lower end of the street lamp 70 and the ground surface on which the street lamp 70 is installed. In such a case, the information processing unit 23 may display on the display unit 22 a screen (GUI) for inputting the reason why the boundary cannot be photographed. FIG. 10 is a diagram showing an example of a GUI for inputting the reason why the boundary cannot be photographed.

[0056] The text information of the text input through the GUI shown in FIG. 10 is transmitted to the image management system 30 in addition to or instead of the image information. Thereby, the image management system 30 can manage the reason why the boundary part cannot be photographed.

[0057] Similarly, when the identification information of the lighting fixture 72 is not attached to the support column 71 (such as when it has peeled off and been lost), a screen (GUI) for inputting the reason why the identification information of the lighting fixture 72 cannot be photographed is also displayed on the display unit 22, and text corresponding to the reason may be input.

[0058] [Modification Example 3: Improvement of the display quality of street lights in a still image] The street light 70 is a long and narrow structure that is long in the vertical direction. Therefore, as shown in FIGS. 4 to 8 above, during the operation in the first mode to the fifth mode, the information processing unit 23 sets the long side direction of the display unit 22 of the information terminal 20 as the vertical direction and the short side direction as the horizontal direction, and displays the moving image being photographed by the photographing unit 27. Thereby, the information terminal 20 can display the street light 70 prominently in a still image.

[0059] The information processing unit 23 may control the photographing unit 27 so as to permit the optical zoom of the photographing unit 27 and not permit the digital zoom of the photographing unit 27 during the execution of the dedicated application. That is, during the operation in the first mode to the fifth mode, the optical zoom of the photographing unit 27 may be permitted and the digital zoom of the photographing unit 27 may not be permitted. Thereby, the information terminal 20 can photograph a still image in which the street light 70 is clearly shown.

[0060] [Modification Example 4: Output of image information] In the display operation of the above GUI, the shooting of a still image and the output of the image information including the shot still image to the storage unit 24 (in other words, the storage of the image information in the storage unit 24) were performed triggered by a predetermined operation of the user (operation on the shooting button). However, the shooting of the still image and the output of the image information may be performed triggered by the elapse of a predetermined time from a certain timing by timer processing. The shooting of the still image and the output of the image information may be automatically performed triggered by the fact that the street lamp 70 shown in the moving image satisfies the above-described predetermined requirements (requirements regarding whether the street lamp 70 is shown as per the guide). In this case, the information processing unit 23 determines whether or not the predetermined requirements are satisfied. Thus, the shooting of the still image and the output of the image information may be performed when the predetermined trigger requirements are satisfied.

[0061] [Modification Example 5: Display of Rust State] When the lower end portion of the street lamp 70 is displayed on the display unit 22 during the operation in the first mode, the information processing unit 23 may display the rust occurrence state at the lower end portion on the display unit 22. FIG. 11 is a diagram showing an example of a GUI indicating the rust occurrence state.

[0062] In the example of FIG. 11, the rust occurrence state is scored by an indicator. The score indicating the rust occurrence state is an example of information regarding the rust state. In this case, a first machine learning model is stored in the storage unit 24 of the information terminal 20 as a part of the dedicated application, and the information processing unit 23 scores the rust occurrence state at the lower end portion of the street lamp 70 currently displayed on the display unit 22 using the first machine learning model.

[0063] The first machine learning model is a learned model that has learned a lot of learning data associating a still image (or a moving image) of the lower end portion of the street lamp 70 with the score of the rust occurrence state of the lower end portion shown in the still image. The first machine learning model can output a score indicating the rust occurrence state with the still image showing the lower end portion of the street lamp 70 currently displayed on the display unit 22 as an input.

[0064] Further, as the rust generation state at the lower end portion, the information processing unit 23 may display the rust generation range on the display unit 22. For example, the information processing unit 23 may display the rust generation range by changing the color of a specific color in a still image (or moving image) of the lower end portion of the street lamp 70 displayed on the display unit 22. The rust generation range is another example of information regarding the rust state.

[0065] In this case, a second machine learning model is stored in the storage unit 24 of the information terminal 20 as a part of the dedicated application, and the information processing unit 23 identifies the rust generation range in the still image (or moving image) of the lower end portion of the street lamp 70 using the second machine learning model.

[0066] The second machine learning model is a learned model that has learned a large amount of learning data associating a still image (or moving image) of the lower end portion of the street lamp 70 with the rust generation range of the lower end portion reflected in the still image. The second machine learning model can identify the rust generation range in the still image by taking as input the still image reflecting the lower end portion of the street lamp 70 currently displayed on the display unit 22.

[0067] In this way, during the operation in the first mode, the information processing unit 23 can assist the user in grasping the rust state of the street lamp 70 by displaying the information regarding the rust state of the street lamp 70 on the display unit 22. Note that the information processing unit 23 may display on the display unit 22 information regarding the rust state of a part other than the lower end portion of the street lamp 70, such as the cover 73.

[0068] Further, the information regarding the rust state of the street lamp 70 is an example of the information regarding the deterioration state of the street lamp 70, and the information processing unit 23 may display on the display unit 22 information regarding the deterioration state other than the rust state. The deterioration state other than the rust state is, for example, the degree of deformation (such as straightness) of the column 71 of the street lamp 70 and the inclination angle of the column 71 with respect to the ground surface. The information processing unit 23 can assist the user in grasping the deterioration state of the street lamp 70 by displaying the information regarding the deterioration state of the street lamp 70 on the display unit 22.

[0069] [Modification Example 6: Association between Image and Location Information] In the above-described embodiment, in order to associate a still image with location information, it has been described that when a still image is taken, the current location of the information terminal 20 is measured by the location measurement unit 26. However, the method of associating a still image with location information is not limited to the method using the location measurement unit 26.

[0070] For example, a map may be displayed on the display unit 22 of the information terminal 20, and the user may plot a point on the displayed map to specify the installation location of the street lamp 70.

[0071] Also, for example, the image management system 30 may provide list information of the installation locations (coordinates) of the street lamps 70 registered in advance in the information terminal 20 while a dedicated application is being executed, and a selection GUI for the installation location based on the list information may be displayed on the display unit 22 of the information terminal 20. The selection GUI is, for example, a GUI in which an icon is displayed at the installation location of the street lamp 70 on the map. The user can input the installation location of the street lamp 70 reflected in the taken still image by selecting the icon. In this case, since the installation location of the street lamp 70 is managed in the image management system 30, the information terminal 20 does not need to transmit location information to the image management system 30, but only needs to transmit the identification information of the selected icon.

[0072] [Modification Example 7: Display of Still Images Taken in the Past] In the above-described embodiment, among a plurality of still images having different shooting times and having the same street lamp 70 as the subject and being managed by the image management system 30, it is desirable that the street lamp 70 is shown at the same angle of view. Thereby, the street lamp 70 can be observed at a fixed point, and it becomes easier to judge the deterioration of the street lamp 70.

[0073] Therefore, in the GUI for shooting the street lamp 70, still images taken in the past may be displayed for reference of the angle of view of the street lamp 70 to be shot hereafter. FIG. 12 is a diagram showing a modification example of the GUI displayed on the display unit 22 of the information terminal 20 during the operation of the first mode.

[0074] In the example of FIG. 12, a still image taken in the past and a moving image showing the current street lamp 70 are displayed side by side at the same size and horizontally. By displaying such a GUI on the display unit 22 of the information terminal 20, the information processing unit 23 of the information terminal 20 can assist the user in photographing the street lamp 70 at the same angle of view as the past still image.

[0075] Note that it is not essential that the still image taken in the past and the moving image showing the current street lamp 70 be displayed side by side at the same size. For example, the still image taken in the past may be smaller than the moving image and may be displayed anywhere around the moving image. Also, the still image taken in the past may be semi-transparent and superimposed on the moving image. Although FIG. 12 shows the GUI in the first mode, the still image taken in the past may be displayed in the GUIs in the second to fifth modes.

[0076] Supplement on the display method of the still image taken in the past. For example, when the user is about to take a still image, the user selects the position of the street lamp 70 to be photographed from among the points on the map displayed on the display unit 22 of the information terminal 20. The information processing unit 23 transmits a request for image information including the position information corresponding to the selected point to the image management system 30 using the communication unit 25. Thereby, the information processing unit 23 can acquire the image information of the past still image of the street lamp 70 to be photographed from the image management system 30 and display it on the display unit 22.

[0077] [Modification Example 8: Mosaic Processing] Peripheral objects such as people, houses, and cars located around the street lamp 70 may be reflected in the still image. Also, when the surface of the street lamp 70 has a gloss, the photographer himself or herself or a peripheral object (hereinafter also referred to as a peripheral object, etc.) may be reflected in the street lamp 70 itself in the still image.

[0078] As described above, local government employees can access the structure management system 50 using the local government terminal 60 to view the still images stored (managed) in the structure management system 50. However, it may not be preferable from the perspective of personal information protection that peripheral objects and the like are reflected in the still images.

[0079] Therefore, in the still images provided from the structure management system 50 to the local government terminal 60, peripheral objects and the like reflected in the still images may be mosaicked. FIG. 13 is a diagram showing an example of the mosaic process.

[0080] For example, in the structure management system 50, an original still image without mosaic processing (FIG. 13(a)) and a still image with mosaic processing (FIG. 13(b)) are stored in a pair. When a local government employee accesses the structure management system 50, for example, they are required to input a user ID. Employees who are determined to have administrative authority based on the user ID are permitted to view the original still image, while employees who are determined not to have administrative authority are prohibited from viewing the original still image and are only permitted to view the still image with mosaic processing. By storing the original still image, it is possible to prevent information necessary for the management of the street lamp 70 from being accidentally lost due to the mosaic process.

[0081] The mosaic process may be performed manually by the administrator of the structure management system 50 or automatically by the structure management system 50. For example, the structure management system 50 can perform a mosaic process on the detected object by performing an object detection process (such as human detection or face detection) on the still image.

[0082] Note that it is not essential that the original still image and the still image with mosaic processing are stored in a pair. For example, in the structure management system 50, only the original still image may be stored, and the mosaic process may be performed each time the still image is provided.

[0083] Also, in the still images provided from the image management system 30 to the information terminal 20, peripheral objects or the like reflected in the still images may be subject to mosaic processing. For example, in the past still images displayed on the GUI of FIG. 12, peripheral objects or the like reflected in the still images may be subject to mosaic processing.

[0084] Also in this case, in the image management system 30, the original still image and the still image subjected to mosaic processing may be saved in a pair, or only the original still image may be saved in the image management system 30, and the mosaic processing may be performed each time a still image is provided.

[0085] [Effects, etc.] Hereinafter, the invention obtained from the disclosure of this specification will be exemplified, and the effects and the like obtained from the exemplified invention will be described.

[0086] Invention 1 is a display method executed by an information terminal 20 including a photographing unit 27 and a display unit 22, including: a first display step of displaying, on the display unit 22, an image being photographed by the photographing unit 27 and a guide for including a lower end portion in a photographing range during an operation in a first mode for photographing a lower end portion of a long structure deteriorated by rust with the photographing unit 27; and an output step of outputting a still image photographed by the photographing unit 27 when a predetermined trigger condition is satisfied. The output of the still image in the output step may be interpreted as an output to a storage unit 24 included in the information terminal 20, or may be interpreted as an output (transmission) to an image management system 30.

[0087] Such a display method can assist in photographing a still image in which the lower end portion of the structure is reflected by displaying a guide on the display unit 22.

[0088] Invention 2 is the display method of Invention 1, further including: a second display step of displaying, on the display unit 22, an image being photographed by the photographing unit 27 and a guide for including the whole in a photographing range during an operation in a second mode for photographing the whole of the structure with the photographing unit 27.

[0089] Such a display method can assist in taking a still image that shows the entire structure by displaying a guide on the display unit 22.

[0090] In Invention 3, the structure is a lighting device (street lamp 70) including a support column 71 and a lamp unit 72, and the display method further includes a third display step of displaying, on the display unit 22, an image being captured by the imaging unit 27 and a guide for including the lamp unit 72 in the imaging range during the operation in the third mode for capturing the lamp unit 72 by the imaging unit 27, and is the display method of Invention 1 or 2.

[0091] Such a display method can assist in taking a still image that shows the lamp unit 72 by displaying a guide on the display unit 22.

[0092] In Invention 4, the structure is a lighting device (street lamp 70) including a support column 71 and a lamp unit 72, and the display method further includes a fourth display step of displaying, on the display unit 22, an image being captured by the imaging unit 27 and a guide for including the identification information regarding the lamp unit 72 in the imaging range during the operation in the fourth mode for capturing the identification information regarding the lamp unit 72 by the imaging unit 27, and is the display method of any one of Inventions 1 to 3.

[0093] Such a display method can assist in taking a still image that shows the identification information regarding the lamp unit 72 by displaying a guide on the display unit 22.

[0094] In Invention 5, the structure is a lighting device (street lamp 70) including a support column 71, a lamp unit 72, and a cover 73 provided on the support column 71, and the display method further includes a fifth display step of displaying, on the display unit 22, an image being captured by the imaging unit 27 and a guide for including the cover 73 in the imaging range in the fifth mode for capturing the cover 73 by the imaging unit 27, and is the display method of any one of Inventions 1 to 4.

[0095] Such a display method can assist in taking a still image that shows the identification information regarding the cover 73 by displaying a guide on the display unit 22.

[0096] Invention 6 is a display method according to any one of Inventions 1 to 5, which displays information regarding the deterioration state of a structure on the display unit 22 in the first display step.

[0097] Such a display method can assist in grasping the deterioration state of the structure.

[0098] Invention 7 is a display method according to Invention 6, wherein the deterioration state of the structure includes the rust state in the structure.

[0099] Such a display method can assist in grasping the rust state in the structure.

[0100] Invention 8 is a display method according to Invention 7, wherein the rust state includes the range where rust has occurred in the structure.

[0101] Such a display method can assist in grasping the range where rust has occurred in the structure.

[0102] Invention 9 is a display method according to any one of Inventions 1 to 8, wherein the guide displayed in the first display step includes a guide for designating the position of the lower end portion of the structure in the image.

[0103] Such a display method can designate the position of the lower end portion of the structure in the image.

[0104] Invention 10 further includes a step of performing at least one of continuing the operation in the first mode and aborting the output step when the lower end portion of the structure shown in the still image does not meet a predetermined requirement during the operation in the first mode, which is a display method according to any one of Inventions 1 to 9.

[0105] Such a display method can suppress the capture of a still image that does not meet the predetermined requirement.

[0106] The invention 11 further includes a step of displaying, on the display unit 22, a screen for inputting a reason why the boundary between the lower end of the structure and the ground surface on which the structure is installed cannot be photographed, and is a display method according to any one of inventions 1 to 10.

[0107] Such a display method can assist in managing the reason why the boundary cannot be photographed.

[0108] In the first display step, the invention 12 is a display method according to any one of inventions 1 to 11, in which the long side direction of the display unit 22 of the information terminal 20 is set as the vertical direction and the short side direction is set as the horizontal direction, and the image being photographed by the photographing unit 27 is displayed.

[0109] Such a display method can display the structure prominently in a still image.

[0110] During the operation in the first mode, the invention 13 is a display method according to any one of inventions 1 to 12, in which the optical zoom of the photographing unit 27 is permitted and the digital zoom of the photographing unit 27 is not permitted.

[0111] Such a display method can photograph a still image in which the structure is clearly shown.

[0112] The invention 14 is a display method according to any one of inventions 1 to 13, in which the predetermined trigger condition includes the condition that a predetermined operation is performed by the user.

[0113] Such a display method can output a still image photographed by the photographing unit 27 triggered by the fact that a predetermined operation has been performed by the user.

[0114] The invention 15 is a program for causing the information terminal 20 to execute the display method according to any one of inventions 1 to 14.

[0115] According to such a program, the information terminal 20 can assist in photographing a still image in which the lower end of the structure is shown by displaying a guide on the display unit 22.

[0116] (Other Embodiments) Although the embodiments have been described above, the present invention is not limited to the above embodiments.

[0117] For example, in the above embodiments, a system such as an information processing system is realized by a plurality of devices, but it may also be realized as a single device. Thus, the system in this specification may be configured by a single device or by a plurality of devices. When the system is realized by a plurality of devices, the components included in the system may be distributed among the plurality of devices in any manner.

[0118] Also, in the above embodiments, the processing executed by a specific processing unit may be executed by another processing unit. Also, the order of a plurality of processes may be changed, or a plurality of processes may be executed in parallel.

[0119] Also, in the above embodiments, each component may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0120] Also, each component may be realized by hardware. Each component may be a circuit (or an integrated circuit). These circuits may constitute one circuit as a whole, or may be separate circuits respectively. Also, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0121] Also, the general or specific aspects of the present invention may be realized by a system, a device, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM. Also, it may be realized by any combination of a system, a device, a method, an integrated circuit, a computer program, and a recording medium.

[0122] For example, the present invention may be implemented as the information terminal of the above embodiment. Further, the present invention may be implemented as a display method executed by an information terminal. The present invention may be implemented as a program for causing an information terminal to execute the display method, or may be implemented as a computer-readable non-transitory recording medium on which such a program is recorded.

[0123] In addition, forms obtained by applying various modifications conceivable by those skilled in the art to each embodiment, or forms realized by arbitrarily combining the components and functions in each embodiment without departing from the spirit of the present invention are also included in the present invention.

Explanation of Reference Numerals

[0124] 10 Information processing system 20 Information terminal 21 Operation reception unit 22 Display unit 22a, 22b Dashed line frame (guide) 23 Information processing unit 24 Storage unit 25 Communication unit 26 Position measurement unit 27 Imaging unit 30 Image management system 40 Diagnosis system 50 Structure management system 60 Local government terminal 70 Street lamp 71 Column 72 Lamp 73 Cover

Claims

1. A display method executed by an information terminal including a photographing unit and a display unit, during operation in a first mode for photographing a lower end portion in the vertical direction of a long structure deteriorated by rust with the photographing unit, a first display step of displaying, on the display unit, an image being photographed by the photographing unit and a guide for including the lower end portion in a photographing range; and an output step of outputting a still image photographed by the photographing unit when a predetermined trigger condition is satisfied. The display method.

2. Furthermore, during operation in a second mode for photographing the entire structure with the photographing unit, the method includes a second display step of displaying, on the display unit, an image being photographed by the photographing unit and a guide for including the whole in a photographing range. The display method according to Claim 1.

3. The structure is a lighting device including a support column and a lamp, and the display method further includes a third display step of displaying, on the display unit, an image being photographed by the photographing unit and a guide for including the lamp in a photographing range during operation in a third mode for photographing the lamp with the photographing unit. The display method according to Claim 1.

4. The structure is a lighting device including a support column and a lamp, and the display method further includes a fourth display step of displaying, on the display unit, an image being photographed by the photographing unit and a guide for including identification information regarding the lamp in a photographing range during operation in a fourth mode for photographing the identification information regarding the lamp with the photographing unit. The display method according to Claim 1.

5. The structure is a lighting device including a support column, a lamp, and a cover provided on the support column, and the display method further includes a fifth display step of displaying, on the display unit, an image being photographed by the photographing unit and a guide for accommodating the cover in a photographing range in a fifth mode for photographing the cover with the photographing unit. The display method according to Claim 1.

6. In the first display step, information regarding a deterioration state of the structure is displayed on the display unit. The display method according to Claim 1.

7. The deterioration state of the structure includes a rust state in the structure. The display method according to Claim 6.

8. The rust state includes a range where the rust has occurred in the structure. The display method according to Claim 7.

9. The guide displayed in the first display step includes a guide specifying a position of the lower end portion of the structure in the image. The display method according to claim 1.

10. Furthermore, during the operation of the first mode, when the lower end portion of the structure reflected in the still image does not satisfy a predetermined requirement, it includes a step of performing at least one of continuing the operation of the first mode and stopping the output step The display method according to claim 1.

11. Furthermore, when the boundary between the lower end portion of the structure and the ground surface on which the structure is installed cannot be photographed, it includes a step of displaying on the display unit a screen for inputting the reason why the boundary cannot be photographed The display method according to claim 1.

12. In the first display step, with the long side direction of the display unit of the information terminal as the vertical direction and the short side direction as the horizontal direction, the image being photographed by the photographing unit is displayed The display method according to claim 1.

13. During the operation of the first mode, the optical zoom of the photographing unit is permitted, and the digital zoom of the photographing unit is not permitted The display method according to claim 1.

14. The predetermined trigger condition includes the condition that a predetermined operation is performed by the user The display method according to claim 1.

15. A program for causing the information terminal to execute the display method according to any one of claims 1 to 14.

Images