Information processing method, information processing device, and information processing program

Abstract

An information processing method according to the present invention comprises: displaying a bird's-eye view on a display of a terminal device; displaying, in the bird's-eye view, a plurality of image capture point icons that indicate image capture points, the plurality of image capture point icons being associated with text information; receiving input of character information from a user; identifying, from among the plurality of image capture point icons, a first icon that is associated with text information which includes the character information; and displaying the first icon in an emphasized manner.

Description

Information processing method, information processing device, and information processing program

[0001] The present disclosure relates to a technology for remotely checking the status of a site.

[0002] The technology of Patent Document 1 discloses a photographed image management system. This system includes an image input unit, a photographing location designation unit, and a photographed image display unit. The image input unit accepts input of photographed images of a site. The photographing location designation unit displays a floor map on a screen. The photographing location designation unit accepts a position clicked on the floor map as the photographing location of the photographed image. The photographing location designation unit displays a marker indicating the photographing location on the floor map. The photographed image display unit displays a photographed image corresponding to the photographing location indicated by the marker selected by the user on a screen. The photographed image display unit displays an icon superimposed on the photographed image. When an icon is selected by the user, the photographed image display unit displays related documents such as an explanation of the object photographed in the photographed image or a detailed drawing.

[0003] In the technology described in Patent Document 1, the number of processing steps on the computer increases in order to check the shooting location icon linked to the desired text information, which increases the burden on the computer.

[0004] Patent No. 6908953

[0005] The present disclosure has been made to solve such problems, and aims to provide a technology that can suppress an increase in the burden on a computer.

[0006] An information processing method in one aspect of the present disclosure is an information processing method in a computer, which includes displaying a bird's-eye view on a display of a terminal device, displaying a plurality of shooting location icons indicating shooting locations on the bird's-eye view, the plurality of shooting location icons being associated with text information, accepting input of text information from a user, identifying a first icon from the plurality of shooting location icons to which text information including the text information is associated, and displaying the first icon in an emphasized manner.

[0007] According to the present disclosure, it is possible to allow a user to easily grasp the location of a shooting location icon associated with desired text information. As a result, an increase in the number of processing steps of a computer is suppressed, and therefore an increase in the burden on the computer is suppressed.

[0008] 1 is a block diagram showing an example of an overall configuration of an information processing system according to a first embodiment of the present disclosure; FIG. 2 is a diagram showing an example of a display screen displayed on a display of a terminal device; FIG. 3 is a flowchart showing a flow of processing performed in the information processing system according to the first embodiment; FIG. 4 is a flowchart showing details of a second processing when annotation information is associated with a shooting location icon; FIG. 5 is a flowchart showing details of a second processing when additional text information is associated with a shooting location icon; FIG. 6 is a flowchart showing details of a third processing shown in FIG. 3; FIG. 7 is a diagram showing an example of a display screen displayed on a display in step S301 of FIG. 6; FIG. 8 is a diagram showing examples of a display screen displayed on a display in a first modified example and a second modified example; FIG. 9 is a diagram showing an example of a display screen displayed on a display in a third modified example; FIG. 10 is a diagram showing an example of a display screen displayed on a display in a fourth modified example; FIG. 11 is a diagram showing another example of a display screen displayed on a display in the fourth modified example.

[0009] (Background to one aspect of the present disclosure) Issues at construction sites and other worksites include the inability to communicate specific instructions to workers, the time it takes to explain those instructions, the need for manpower to cover the entire site, and the time it takes to travel to the site.

[0010] To solve these problems, it is conceivable to install multiple cameras on the construction site and have a remote user give instructions to workers while referring to images obtained from the multiple cameras, but this is not practical. Specifically, with the above method, as the work process at the construction site progresses, tasks such as removing the installed cameras and installing the removed cameras in different locations are required. This is because such tasks are time-consuming. Therefore, installing multiple cameras at a construction site is not practical. Therefore, the present inventors have investigated technology that allows remote confirmation of the status of a construction site without installing multiple cameras at the construction site.

[0011] As a result of investigation, the inventors have found that if there is a user interface in which a plurality of photography location icons are displayed superimposed on a bird's-eye view corresponding to a site, the user can remotely check the situation of the site. In detail, the inventors have found that if text information indicating comments and notes of workers on site is associated with the plurality of photography location icons, and when a user selects one of the photography location icons, the text information associated with the selected photography location icon is output, the user can remotely check the situation of the site.

[0012] However, with the above user interface, the user may be forced to find a photography location icon associated with desired text information. Specifically, when many photography location icons are displayed on a bird's-eye view and text information is associated with these photography location icons, it becomes difficult for the user to understand which photography location icon is associated with what text information. In this case, the user must select photography location icons one by one to find the photography location icon associated with the desired text information. This increases the number of processing steps in the computer, thereby increasing the burden on the computer.

[0013] Therefore, after further consideration, the inventor discovered that if it were possible to clearly show the user the shooting location icon associated with the text information the user desires, the task of finding the above-mentioned shooting location icon would be omitted, thereby preventing the number of processing steps on the computer from increasing.

[0014] The present disclosure has been made based on these findings.

[0015] (1) An information processing method in one aspect of the present disclosure is an information processing method in a computer, which includes displaying a bird's-eye view on a display of a terminal device, displaying a plurality of shooting location icons indicating shooting locations on the bird's-eye view, the plurality of shooting location icons being associated with text information, accepting input of text information from a user, identifying a first icon from the plurality of shooting location icons to which text information including the text information is associated, and displaying the first icon in an emphasized manner.

[0016] According to this configuration, the first icon, which is a photography location icon associated with text information including character information entered by the user, is highlighted. This allows the user to easily determine the location of the first icon, which is a photography location icon associated with the desired text information. As a result, an increase in the number of processing steps on the computer is suppressed, and therefore an increase in the burden on the computer is suppressed.

[0017] (2) In the information processing method described in (1) above, image information may be associated with the multiple shooting location icons, and highlighting the first icon may include displaying image information associated with the first icon on the display together with the highlighted first icon.

[0018] According to this configuration, the image information associated with the first icon is displayed on the display together with the highlighted first icon, thereby improving convenience.

[0019] (3) In the information processing method described in (1) or (2) above, highlighting and displaying the first icon may include not displaying on the bird's-eye view a second icon among the plurality of shooting location icons that is associated with text information that does not include the character information.

[0020] According to this configuration, the second icon is not displayed on the display, so that the user can more reliably know the location of the first icon.

[0021] (4) In the information processing method described in (2) or (3) above, the image information is associated with at least one attention display that is superimposed on the image information and displayed on the display, the text information is associated with at least one of the attention displays, and further, when multiple attention displays are associated with the image information, only the attention display that is associated with the text information may be displayed on the display.

[0022] According to this configuration, only the attention display associated with the text information is displayed on the display, so when multiple attention displays are associated with image information, the user can more reliably understand which attention display the text information is associated with.

[0023] (5) In the information processing method described in (2) or (3) above, the image information is associated with at least one attention display that is displayed on the display superimposed on the image information, the text information is associated with at least one of the attention displays, and further, when multiple attention displays are associated with the image information, the attention display associated with the text information may be displayed on the display in a manner different from the attention display not associated with the text information.

[0024] According to this configuration, it is possible to differentiate the display mode between an attention display associated with text information and an attention display not associated with text information, thereby enabling the user to more reliably understand which attention display is associated with the text information when multiple attention displays are associated with image information.

[0025] (6) In the information processing method described in (4) or (5) above, displaying the plurality of shooting location icons on the bird's-eye view may include changing the display mode of the shooting location icons between when the text information is associated with the attention display and when the text information is not associated with the attention display.

[0026] According to this configuration, it is possible to distinguish between a photography location icon whose attention display has text information associated therewith and a photography location icon whose attention display does not have text information associated therewith and present them to the user.

[0027] (7) In the information processing method described in any of (1) to (6) above, the information processing method may further include accepting input of filtering conditions, and highlighting the first icons may include highlighting only the first icons that match the filtering conditions.

[0028] According to this configuration, only the first icons that match the narrowing conditions are highlighted, so that the location of the first icons that match the conditions desired by the user can be made clear.

[0029] Furthermore, the present disclosure can be realized not only as an information processing method that executes the characteristic processes described above, but also as an information processing device having a characteristic configuration corresponding to the characteristic processes executed by the information processing method. Furthermore, the present disclosure can also be realized as a computer program that causes a computer to execute the characteristic processes included in such an information processing method. Therefore, the same effects as those of the above information processing method can also be achieved in the following other aspects.

[0030] (8) An information processing device according to another aspect of the present disclosure is an information processing device including a processor, wherein the processor executes the following operations: displaying a bird's-eye view on a display of a terminal device; displaying a plurality of shooting location icons indicating shooting locations on the bird's-eye view; accepting input of text information from a user, wherein text information including the text information is associated with the plurality of shooting location icons; identifying a first icon from the plurality of shooting location icons to which text information including the text information is associated; and displaying the first icon in an emphasized manner.

[0031] (9) An information processing program according to yet another aspect of the present disclosure causes a computer to display a bird's-eye view on a display of a terminal device, display a plurality of shooting location icons indicating shooting locations on the bird's-eye view, the plurality of shooting location icons being associated with text information, accepting input of text information from a user, identifying a first icon from the plurality of shooting location icons to which text information including the text information is associated, and displaying the first icon in an emphasized manner.

[0032] The present disclosure can also be realized as an information processing system operated by such an information processing program. Needless to say, such a program can be distributed on a non-transitory recording medium such as a computer-readable CD-ROM, or via a communication network such as the Internet.

[0033] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in an independent claim that represents a top-level concept will be described as optional components. Furthermore, in all embodiments, the respective contents can be combined.

[0034] (First Embodiment) FIG. 1 is a block diagram showing an example of the configuration of an information processing system 1 according to a first embodiment of the present disclosure. The information processing system 1 includes an information processing device 10, an image capturing device 20, and a terminal device 30. The information processing device 10, the image capturing device 20, and the terminal device 30 are communicably connected via a network. An example of a network is the Internet. The information processing device 10 is, for example, a cloud server configured with one or more computers. However, this is just one example, and the information processing device 10 may be configured as an edge server or may be implemented in the terminal device 30.

[0035] The image capture device 20 is, for example, an omnidirectional camera and captures images at a predetermined frame rate. The image capture device 20 is, for example, a portable image capture device carried by a user. The user may be, for example, a site worker or site supervisor. The site may be, for example, a construction site. The user moves around the construction site while capturing images of the construction site with the image capture device 20. Specifically, at the start point of capture, the user points the image capture direction of the image capture device 20 toward a reference direction, presses the capture button on the image capture device 20, and starts the capture operation. When the user reaches the end point of capture, the user presses the capture button again to end the capture operation. The reference direction is, for example, north, but is not limited to this. The image capture device 20 transmits image information indicating the captured images to the information processing device 10 via a network. This allows the information processing device 10 to obtain image information from multiple image capture locations at the construction site. Here, the image capture device 20 captures images at a predetermined frame rate, so the image capture locations are defined in frame cycles. However, this is just an example, and the image capture locations may also be defined every predetermined time (e.g., one second, one minute, etc.). The image information is associated with the location of the image capture location and the date and time of capture. The location of the image capture location is acquired by a position sensor such as a magnetic sensor or a GPS sensor provided in the image capture device 20, and is expressed as latitude and longitude. The date and time of capture is acquired by a clock provided in the image capture device 20, for example. The image capture device 20 includes an image sensor, an operation device, a communication circuit, a signal processing circuit, etc. The image capture device 20 may be configured as a portable computer such as a smartphone or a tablet computer.

[0036] The terminal device 30 is carried by a user. The terminal device 30 may be configured as a portable computer such as a smartphone or a tablet computer, or may be configured as a desktop computer. The terminal device 30 displays image information on a display 31 under the control of the information processing device 10. Although one terminal device 30 is illustrated in the example of FIG. 1, a plurality of terminal devices may be connected to the information processing device 10 via a network. The terminal device 30 includes a central processing unit (CPU), a memory, a display 31, operation devices such as a touch panel, a mouse, and a keyboard, and a communication circuit.

[0037] The information processing device 10 includes a processor 11, a memory 12, and a communication unit 13. The processor 11 is configured, for example, by a central processing unit (CPU). The processor 11 includes an acquisition unit 111, a setting unit 113, and an output unit 114. The acquisition unit 111 to the output unit 114 may be realized by the processor 11 executing an information processing program, or may be configured by a dedicated hardware circuit such as an ASIC.

[0038] The acquisition unit 111 acquires a display instruction from the terminal device 30 via the communication unit 13. The display instruction is an instruction input by a user to the terminal device 30, requesting that a specific blueprint be displayed on the display 31. The display instruction includes a blueprint ID, which is an identifier for the blueprint. The acquisition unit 111 reads out the blueprint indicated by the blueprint ID from the memory 12. The blueprint is image data showing a blueprint of the construction site. The blueprint is an example of a bird's-eye view. The acquisition unit 111 may acquire a floor plan, blueprint, map, or perspective view of the construction site instead of the blueprint. The floor plan, blueprint, map, and perspective view are other examples of a bird's-eye view. In the blueprint, latitude and longitude are previously associated with key point locations. The key point locations are, for example, the four corners of the blueprint.

[0039] Furthermore, the acquisition unit 111 acquires character information input by the user to the terminal device 30 via the communication unit 13. The acquisition unit 111 identifies at least one first icon from among the multiple shooting location icons 201 based on the character information. The first icon is the shooting location icon 201 associated with sentence information including the character information input by the user. The acquisition unit 111 inputs the identification result of the first icon to the output unit 114. At this time, the acquisition unit 111 also reads image information associated with the first icon from the memory 12 and inputs the image information to the output unit 114.

[0040] Furthermore, the acquisition unit 111 acquires a selection instruction from the terminal device 30 via the communication unit 13. The selection instruction is an instruction transmitted from the terminal device 30 to the acquisition unit 111 when the user selects one of the multiple photography location icons 201 displayed on the design drawing. When the selection instruction is acquired, the acquisition unit 111 acquires photography information associated with the photography location icon 201 selected by the user. The acquisition unit 111 inputs the photography information to the output unit 114.

[0041] The setting unit 113 determines a first display range when displaying image information associated with the first icon or image information associated with a shooting location icon 201 selected by the user in the image information display field R2 ( FIG. 2 ), which will be described later. As described above, in this embodiment, photography is performed using a 360-degree camera. Since it is difficult to display the entire image captured by the 360-degree camera in the image information display field R2 at once, the setting unit 113 determines a partial range of the entire image information as the first display range. This first display range is displayed in the image information display field R2. The setting unit 113 determines the first display range, for example, so that the reference direction (north in this example) is located at the center of the image information display field R2. In other words, the setting unit 113 determines the first display range so that the range around the reference direction is displayed in the image information display field R2. The setting unit 113 also determines the magnification of the first display range when displaying the first display range in the image information display field R2.

[0042] The output unit 114 generates control information including instructions for displaying various display screens on the display 31. The output unit 114 generates the control information based on various pieces of information received from the acquisition unit 111 and the setting unit 113.

[0043] 2 is a diagram showing an example of a display screen G1 that is displayed on the display 31 by the terminal device 30 that has received control information from the output unit 114. The display screen G1 is a basic screen of an application (user interface) provided by the information processing system 1. As shown in FIG. 2, the display screen G1 includes a blueprint display field R1, an image information display field R2, a text information display field R3, and a news display field R4.

[0044] The blueprint display field R1 displays a blueprint. As shown in Fig. 2, a plurality of shooting location icons 201 and a trajectory 202 are displayed superimposed on the blueprint shown in the blueprint display field R1.

[0045] The plurality of photographing location icons 201 indicate photographing locations of a plurality of pieces of image information photographed by the photographing device 20. In this example, each of the plurality of photographing location icons 201 is configured by a circular image. Each of the plurality of photographing location icons 201 is associated with text information and image information.

[0046] The multiple photography location icons 201 are a group of photography location icons indicating a series of multiple photography locations from a photography start location to a photography end location. They include a first icon and a second icon. The first icon is a photography location icon associated with text information including a keyword input by the user. The second icon is a photography location icon associated with text information that does not include a keyword input by the user.

[0047] When the first icon is identified by the acquisition unit 111, the output unit 114 transmits control information for highlighting the first icon to the terminal device 30. Details will be described later.

[0048] Furthermore, when the acquisition unit 111 acquires a selection instruction, that is, when the user selects one of the multiple photography location icons 201, the output unit 114 transmits, to the terminal device 30, control information for making the display mode of the photography location icon 201 selected by the user (hereinafter referred to as the selected icon) different from the display modes of the other photography location icons 201. For example, the output unit 114 transmits, to the terminal device 30, control information including an instruction for making the color of the selected icon different from the colors of the other photography location icons 201.

[0049] The trajectory 202 indicates the movement path trajectory of the user who captured the image information. In this example, the trajectory 202 is made up of lines connecting adjacent photography location icons 201. The photography location icon 201 located at the tip of the trajectory 202 and the photography location icon 201 located at the rear end of the trajectory 202 are displayed in a larger size than the other photography location icons 201. The photography location icon 201 located at the tip of the trajectory 202 (for example, the right end in FIG. 2) indicates the photography start position, and the photography location icon 201 located at the rear end of the trajectory 202 (for example, the left end in FIG. 2) indicates the photography end position.

[0050] The image information display field R2 displays image information associated with the shooting location icon 201. For example, when the acquisition unit 111 identifies a first icon, the image information display field R2 displays image information associated with the first icon. Furthermore, when the user selects a specific shooting location icon 201, the image information display field R2 may display image information associated with the selected icon. Note that when the process of identifying the first icon has not been performed and the user has not selected a shooting location icon 201, the image information display field R2 may be blank.

[0051] Image information may be associated with an attention indicator that is displayed on the display 31 superimposed on the image information. In this case, the image information display field R2 draws the attention indicator on the image information. For example, a rectangular frame D is drawn on the image information IM1 displayed in the image information display field R2 in FIG. 2. The rectangular frame D is an example of an attention indicator. For example, the rectangular frame D is added to the image information IM1 by a worker on-site or the like in order to highlight a specific object or a specific area that appears in the image information IM1. In other words, a frame such as a bounding box may be superimposed on the image information.

[0052] The text information display field R3 displays text information associated with the first icon. The text information is composed of text (character strings).

[0053] The text information according to the present embodiment is divided into annotation information and additional text information. Annotation information is text information associated with a rectangular frame D (attention mark) in the image information. In other words, when an attention mark such as a bounding box is displayed in the image information display field R2 and character strings such as memos, comments, or captions are linked (associated) with the bounding box, the information processing device 10 treats these character strings as annotation information.

[0054] On the other hand, text information associated with the first icon without being associated with an attention display is additional text information. Character strings such as notes, comments, and captions may be associated with the first icon regardless of the attention display. For example, a character string describing the overall characteristics of the image information associated with the first icon may be associated with the first icon. Furthermore, a character string describing the shooting location where the first icon is displayed may be associated with the first icon. In addition, messages exchanged between users may be associated with the first icon. The information processing device 10 treats these character strings as additional text information.

[0055] 2, the text information display field R3 displays a list of text information C1 input by multiple users for the image information displayed in the image information display field R2. In the text information display field R3, text information C1 input by users other than the user himself / herself is displayed on the left side, and text information C1 input by the user himself / herself is displayed on the right side.

[0056] As an example, if the text information is annotation information, the output unit 114 displays an attention display ID at the beginning of the text information. The attention display ID is an identifier for the attention display. For example, as shown in FIG. 2, a user inputs text information such as "Delivery will be received from 1:00 PM today, this location will be used." At this time, the user inputs an operation to link the text information to a rectangular frame D displayed in the image information display field R2. In this case, the output unit 114 obtains the attention display ID for the rectangular frame D and adds this attention display ID to the beginning of the text information input by the user. That is, as shown in FIG. 2, the text information "ID001: Delivery will be received from 1:00 PM today, this location will be used" is displayed.

[0057] In addition, the output unit 114 may display the attention display ID as link text in the text information display field R3, and when the user clicks on the attention display ID, the attention display corresponding to the attention display ID may be highlighted. For example, the output unit 114 may blink the rectangular frame D in the image information display field R2.

[0058] If the process of identifying the first icon has not been performed and the user has not selected the shooting location icon 201, the text information display field R3 may be blank.

[0059] The news display section R4 displays various news related to the scene. For example, a message indicating that text information is associated with a predetermined shooting location icon 201 is displayed in the news display section R4.

[0060] 1, the memory 12 is configured as a non-volatile rewritable storage device such as a hard disk drive, a solid state drive, etc. The memory 12 includes a design drawing information storage unit 121, a photography information storage unit 122, an image information storage unit 123, and a text information storage unit 124.

[0061] The blueprint information storage unit 121 stores at least one blueprint, a blueprint ID assigned to each blueprint, and the latitude and longitude of key points (e.g., four corners) of each blueprint.

[0062] The photographing information storage unit 122 stores photographing information, which is information relating to one photographing operation using the photographing device 20. The photographing information is generated each time one photographing operation is performed. One photographing operation refers to a series of operations from when a worker holding the photographing device 20 starts photographing at the work site until when the photographing ends. A plurality of pieces of image information are generated by one photographing operation.

[0063] The photography information includes a blueprint ID, a photography ID, a photography date and time, a representative value of the photography date and time, location information of the photography location, and location information of the photography location icon 201. The blueprint ID is an identifier of the blueprint of the site where the photography operation was performed. The photography ID is an identifier for identifying each photography included in one photography operation. The photography date and time is information indicating the year, month, day, and time when the photography indicated by the photography ID was performed. The representative value of the photography date and time is information indicating, for example, the year, month, day, and time when photography started. The photography location indicates the location (latitude and longitude) where the photography indicated by the photography ID was performed. The location information of the photography location icon 201 indicates the display position (coordinates) on the blueprint of the photography location icon 201 corresponding to the photography indicated by the photography ID. The display position of the photography location icon 201 on the blueprint is calculated by mapping the photography location on the blueprint for each photography indicated by the photography ID based on the latitude and longitude of the key points (four corners) of the blueprint stored in the blueprint information storage unit 121 and the location information (latitude and longitude) of the photography location.

[0064] The image information storage unit 123 stores image information. Image information refers to one image generated by each capture performed in one capture operation. As described above, in this embodiment, the capture device 20 captures images at a predetermined frame rate (e.g., 30 fps), and therefore, multiple pieces of image information are stored in the image information storage unit 123. A capture ID is associated with each piece of image information. Using this capture ID as a key, the information processing device 10 can identify the capture information associated with the image information. Furthermore, when an attention indicator is assigned to image information, an attention indicator ID for identifying the attention indicator is associated with the image information. When multiple attention indicators are associated with one piece of image information, multiple attention indicator IDs are associated with one piece of image information. Furthermore, when an attention indicator is assigned to image information, the image information storage unit 123 stores attention indicator position information that indicates the display position of the attention indicator on the image information. The attention indicator position information includes, for example, the coordinate values ​​of each vertex of a rectangular frame D on the image information.

[0065] The text information storage unit 124 stores text information associated with each photographing location icon 201. A photographing ID is associated with each piece of text information. Using this photographing ID as a key, the information processing device 10 identifies the display position on the blueprint of the photographing location icon 201 associated with the text information.

[0066] Furthermore, classification information associated with each piece of text information is stored in the text information storage unit 124. The classification information is information indicating whether the text information is annotation information or additional text information.

[0067] The text information storage unit 124 also stores an attention display ID for identifying which attention display the annotation information is associated with. When multiple attention displays are assigned to image information, the information processing device 10 uses the attention display ID to identify which attention display the annotation information is associated with.

[0068] In this way, since the shooting information and the image information are associated with a shooting ID, the location information of the shooting location icon 201 is identified using the shooting ID as a key. That is, the image information associated with the shooting location icon 201 is identified. Also, since the shooting ID is associated with the text information, the location information of the shooting location icon 201 associated with the text information is identified using the shooting ID as a key. That is, the text information associated with the shooting location icon 201 is identified.

[0069] The communication unit 13 is a communication circuit that connects the information processing device 10 to a network.

[0070] 3 is a flowchart showing the flow of processing performed in the information processing system 1 according to embodiment 1. As shown in FIG. 3, in embodiment 1, the first processing is performed followed by the second processing, and the second processing is performed followed by the third processing. However, this is merely an example, and the order in which the processing is performed can be changed as appropriate.

[0071] In step S11 of the first process, the photographing device 20 uploads photographed data. Specifically, in step S11, the photographer performs a photographing operation. In response, the photographing device 20 generates multiple pieces of image information and photographing information corresponding to each of the multiple pieces of image information. Then, the photographing device 20 transmits the multiple pieces of photographing information and the photographing information corresponding to each of the multiple pieces of image information to the information processing device 10.

[0072] In step S12, the acquisition unit 111 acquires the uploaded series of shooting data. Specifically, the acquisition unit 111 acquires, via the communication unit 13, a plurality of pieces of shooting information and pieces of shooting information corresponding to each of the plurality of pieces of image information.

[0073] In step S13, the acquisition unit 111 maps the multiple shooting location icons 201 and the trajectory 202 onto the design drawing. In detail, the acquisition unit 111 maps the multiple shooting location icons 201 and the trajectory 202 onto the design drawing by using the received multiple pieces of image information and shooting information corresponding to each of the multiple pieces of image information.

[0074] In step S21 of the second process, the terminal device 30 accepts annotation information or additional text information for the photography location icon 201 mapped on the design drawing. More specifically, in step S21, the user selects one of the photography location icons 201. The user then inputs annotation information or additional text information to be associated with the selected photography location icon 201. The terminal device 30 that has accepted the annotation information or additional text information from the user transmits this information to the information processing device 10.

[0075] In step S22 , the acquisition unit 111 of the information processing device 10 acquires the annotation information or additional text information sent from the terminal device 30 .

[0076] In step S23, the acquisition unit 111 links (associates) the shooting location icon 201 selected by the user in step S21 with the annotation information or the additional text information.

[0077] In step S31 of the third process, the terminal device 30 receives, from the user, character information that the user wishes to search for. The terminal device 30 also transmits the character information received from the user to the information processing device 10.

[0078] In step S32, the acquisition unit 111 of the information processing device 10 acquires (receives) character information that the user wants to search for.

[0079] In step S33, the acquisition unit 111 performs a determination process. In step S34, the output unit 114 generates control information based on the determination result of the determination process and transmits the control information to the terminal device 30. In step S35, the terminal device 30 highlights the first icon based on the control information. Details of the processes in steps S33 to S35 will be described later using FIG. 6 .

[0080] FIG. 4 is a flowchart showing the details of the second process (steps S21 to S23 in FIG. 3) when the annotation information is associated with the shooting location icon 201.

[0081] In step S201, the terminal device 30 displays on the display 31 a display page including image information associated with a certain shooting location icon 201.

[0082] In detail, in step S201, the terminal device 30 receives a display instruction from the user. The terminal device 30 that has received the display instruction transmits the display instruction to the information processing device 10. The acquisition unit 111 of the information processing device 10 that has received the display instruction reads out the design drawing specified by the user from the memory 12. The output unit 114 of the information processing device 10 then transmits an instruction to the terminal device 30 to display the design drawing read out by the acquisition unit 111 on the display 31. The terminal device 30 that has received this instruction displays the design drawing on the display 31.

[0083] Next, the terminal device 30 accepts a selection instruction from the user. In particular, the user inputs into the terminal device 30 an operation to select a photography location icon 201 associated with image information for which the user wishes to input text information. The selection instruction includes location information of the photography location icon 201 selected by the user. In other words, it includes information indicating which photography location icon 201 has been selected from the multiple photography location icons 201 displayed on the blueprint. The terminal device 30 transmits the selection instruction to the acquisition unit 111. Having received the selection instruction, the acquisition unit 111 reads out image information associated with the photography location icon 201 selected by the user from the memory 12 and inputs this image information to the output unit 114. The output unit 114 transmits to the terminal device 30 an instruction to display on the display 31 the image information read out from the memory 12 by the acquisition unit 111. Upon receiving this instruction, the terminal device 30 displays the image information on the display 31. That is, the terminal device 30 transitions the screen of the display 31 from the display page displaying the blueprint to a display page displaying the image information associated with the shooting location icon 201 selected by the user.

[0084] In step S202, the terminal device 30 displays a rectangular frame D at a position selected by the user, and acquires position information of the rectangular frame D on the image information.

[0085] More specifically, in step S202, the user inputs an operation to the terminal device 30 to specify (select) a predetermined area in the image information. The operation to specify a predetermined area in the image information is, for example, a drag operation. Upon receiving this operation, the terminal device 30 displays a rectangular frame D that surrounds the area specified by the user, superimposed on the image information. At this time, the terminal device 30 acquires position information (coordinate values) of each vertex of the rectangular frame D on the image information. In other words, it acquires position information of the focus display.

[0086] In step S203, the terminal device 30 receives text information from the user. That is, the terminal device 30 receives annotation information associated with the rectangular frame D shown in Fig. 2 etc. For example, the terminal device 30 receives, as annotation information, a character string such as "Delivery will begin at 1:00 PM today. This location will be used" shown in the text information display field R3 in Fig. 2.

[0087] In step S204, the terminal device 30 transmits to the information processing device 10 the position information (coordinate values) of the rectangular frame D on the image information and the text information (annotation information) received from the user.

[0088] In detail, the terminal device 30 transmits to the information processing device 10 information notifying that the user has designated a specific area on the image information and added an attention mark (rectangular frame D) to the image information, the position information of the attention mark on the image information, and text information (annotation information) that the user has input in association with the attention mark.

[0089] In step S205 , the acquisition unit 111 of the information processing device 10 receives the position information on the image information of the rectangular frame D and the text information (annotation information) accepted by the terminal device 30 .

[0090] In step S206, the acquisition unit 111 links (corresponds) the position information of the shooting location icon 201 selected by the user in step S201 or the image information associated with the shooting location icon 201 with the position information on the image information of the rectangular frame D and the text information input by the user in association with the rectangular frame D. The acquisition unit 111 stores this correspondence in the memory 12.

[0091] FIG. 5 is a flowchart showing the details of the second process when the additional text information is associated with the shooting location icon 201 .

[0092] In step S211 shown in FIG. 5, the same process as in step S201 shown in FIG. 4 is performed.

[0093] In the flowchart shown in FIG. 5, the process corresponding to step S202 shown in FIG. 4 is not performed.

[0094] In steps S213 to S216 shown in Fig. 5, the same processes as in steps S203 to S206 shown in Fig. 4 are performed. However, in step S214, unlike step S204, the terminal device 30 does not transmit position information on the image information of the rectangular frame D. In addition, in step S215, unlike step S205, the acquisition unit 111 does not receive position information on the image information of the rectangular frame D. In addition, in step S216, the acquisition unit 111 links the text information to the position information of the shooting location icon 201 selected by the user in step S211 or the image information associated with the shooting location icon 201. In other words, in step S216, the process of linking the position information on the image information of the rectangular frame D to the position information of the shooting location icon 201 selected by the user in step S211 or the image information associated with the shooting location icon 201 is not performed.

[0095] By executing the processes relating to steps S201 to S206 in Figure 4 and the processes relating to steps S211 to S216 in Figure 5, the text information (annotation information or additional text information) entered by the user is accumulated in the text information storage unit 124 of the memory 12.

[0096] FIG. 6 is a flowchart showing the details of the third process shown in FIG.

[0097] In step S301, the terminal device 30 accepts text information from the user.

[0098] Specifically, in step S301, the output unit 114 transmits an instruction to the terminal device 30 requesting that the display screen G1 be displayed on the display 31. Upon receiving this instruction, the terminal device 30 displays the display screen G1 on the display 31. FIG. 7 is a diagram showing an example of the display screen G1 displayed in step S301. As shown in FIG. 7, the display screen G1 includes a search window R5. In step S301, the user inputs text information into the search window R5.

[0099] In step S302, the terminal device 30 transmits the character information input by the user in the search box R5 to the information processing device 10.

[0100] In step S303, the acquisition unit 111 receives the text information.

[0101] In step S304, the acquisition unit 111 performs a determination process to determine whether text information including the character information input by the user is stored in the text information storage unit 124. Specifically, the acquisition unit 111 sets a search range so that all text information stored in the text information storage unit 124 of the memory 12 is the search target, sets the character information input by the user as a keyword, and executes a search process. The acquisition unit 111 then determines whether text information including a character string matching the keyword exists in the text information storage unit 124. If text information including the character information input by the user is stored in the text information storage unit 124, the acquisition unit 111 identifies the shooting location icon 201 associated with the text information. In other words, the acquisition unit 111 identifies the first icon. The acquisition unit 111 then acquires various information related to the first icon from the memory 12, such as the location information of the first icon on the blueprint and image information associated with the first icon. If the acquisition unit 111 cannot identify the first icon, the process ends.

[0102] In step S305, the output unit 114 transmits the control information to the terminal device 30. Specifically, the output unit 114 generates the control information based on the result of the determination process performed by the acquisition unit 111 in step S304. The control information includes an instruction to display the first icon in an emphasized manner. The output unit 114 transmits the control information to the terminal device 30.

[0103] In step S306, the terminal device 30 receives the control information.

[0104] In step S307, the terminal device 30 displays, on the display 31, a display screen G1 on which the first icon is highlighted, based on the control information.

[0105] The third process will be described in detail with reference to FIG. 7 . First, the user inputs text information into the search window R5 (step S301). In the example shown in FIG. 7 , the user inputs the text information "delivery" into the search window R5. Hereinafter, the text information input by the user into the search window R5 will be referred to as a keyword. The terminal device 30 transmits the keyword to the information processing device 10 (step S302). The keyword "delivery" input by the user is acquired by the acquisition unit 111 (step S303), and the acquisition unit 111 identifies a photography location icon 201 associated with text information including the keyword "delivery," i.e., a first icon (step S304). Here, it is assumed that, of the multiple photography location icons 201 shown in FIG. 7 , photography location icons 201A, 201B, and 201C are identified as first icons. Based on the identification result by the acquisition unit 111, the output unit 114 generates control information including an instruction to highlight the first icon (step S305). That is, the output unit 114 generates control information including an instruction to request that the shooting location icons 201A, 201B, and 201C be highlighted on the display screen G1. The terminal device 30 receives this control information from the output unit 114 (step S306) and highlights the first icon based on the control information (step S307).

[0106] An instruction to highlight the first icon is, for example, an instruction to make the display mode of the first icon different from the display mode of the second icon. In this example, the photography location icon 201 associated with text information that does not include the keyword "delivery" corresponds to the second icon. Specifically, of the multiple photography location icons 201 shown in FIG. 7 , the photography location icons 201 other than the photography location icons 201A, 201B, and 201C correspond to the second icon.

[0107] The instruction to change the display mode of the first icon from the display mode of the second icon is, for example, an instruction to change the color of the first icon from the color of the second icon. Specifically, the instruction is to display the first icon in, for example, red and the second icon in, for example, yellow. The instruction to change the display mode of the first icon from the display mode of the second icon may be to make the radius of the circular image constituting the first icon relatively larger than the radius of the circular image constituting the second icon. The instruction may also be to display only the first icon among the multiple shooting location icons 201 in the blueprint display field R1 and not display the second icon in the blueprint display field R1 (for example, by making it transparent). Additionally, the output unit 114 may transmit various instructions to the terminal device 30 to highlight the first icon.

[0108] The output unit 114 may also generate an instruction requesting that the highlighted first icon and image information associated with the highlighted first icon be displayed in the image information display field R2. For example, the output unit 114 may generate control information including an instruction requesting that the shooting location icon 201A (first icon) be displayed in red, the second icon be displayed in yellow, and image information associated with the shooting location icon 201A be displayed in the image information display field R2. This allows the user to check the image information associated with the first icon, thereby improving the convenience of the application.

[0109] If there are multiple first icons, the output unit 114 selects one of the multiple first icons. Then, it generates control information including an instruction to display image information associated with the selected first icon in the image information display field R2. For example, the output unit 114 selects the first icon with the most recent shooting date and time from among the multiple first icons. However, this is merely an example, and the output unit 114 may select one first icon in any appropriate manner.

[0110] When one first icon is selected from among a plurality of first icons, the output unit 114 may generate control information including an instruction to change the display mode of the selected first icon from that of the other first icons. For example, suppose that the photographing location icon 201A is selected from among the plurality of first icons. In this case, the output unit 114 may generate control information including an instruction to display the photographing location icon 201A in red, the photographing location icon 201B and the photographing location icon 201C in orange, and the second icon in yellow.

[0111] According to the information processing system 1 described above, the first icon, which is the photography location icon 201 associated with the text information including the character information input by the user, is displayed in an emphasized manner. In this way, the user can easily grasp the location of the first icon, which is the photography location icon 201 associated with the desired text information. As a result, an increase in the number of processing steps of the information processing device 10 and the like is suppressed, and therefore an increase in the burden on the information processing device 10 and the like is suppressed.

[0112] The present disclosure can employ the following modifications.

[0113] (1) When multiple attention displays are associated with image information, the control information generated by the output unit 114 may include an instruction to display only the attention display associated with the text information in the image information display field R2.

[0114] 8 is a diagram showing an example of a display screen G1 according to this modification. Multiple attention displays are associated with image information IM2 displayed in the image information display field R2 according to this modification. As a result, rectangular frames D1, D2, and D3 are drawn in the image information IM2. Assume here that text information (annotation information) is associated with rectangular frame D1, while no text information is associated with rectangular frames D2 and D3. In this case, the output unit 114 generates control information including an instruction requesting that only rectangular frame D1 be drawn in the image information IM2.

[0115] According to this modification, when a plurality of attention indicators are associated with image information, the user can more reliably understand which attention indicator is associated with the text information.

[0116] (2) When multiple attention displays are associated with image information, the control information generated by the output unit 114 may include instructions to display attention displays associated with text information in the image information display field R2 in a manner different from attention displays not associated with text information.

[0117] This will be described with reference to Figure 8. As in variant example (1), it is assumed that text information (annotation information) is linked to rectangular frame D1 in Figure 8, and that no text information is linked to rectangular frames D2 and D3. In this case, the output unit 114 generates control information including an instruction to make the display mode of rectangular frame D1 different from the display modes of rectangular frames D2 and D3. For example, the control information includes an instruction to display rectangular frame D1 in red and rectangular frames D2 and D3 in pink. This variant example can achieve the same effect as variant example (1).

[0118] (3) The control information generated by the output unit 114 may include an instruction to change the display mode of the photographing location icon 201 between a case where text information is associated with the featured display and a case where the text information is not associated with the featured display. In other words, the control information may include an instruction to change the display mode of the photographing location icon 201 between a case where annotation information is associated with a photographing location icon 201 among the multiple photographing location icons 201 and a case where additional text information is associated with the photographing location icon 201.

[0119] For example, the output unit 114 generates control information including an instruction to place a first mark M1 around a photography location icon 201 associated with annotation information and a second mark M2 around a photography location icon 201 associated with additional text information. FIG. 9 is a diagram showing an example of a display screen G1 according to this modified example. As shown in FIG. 9, the blueprint display field R1 of the display screen G1 includes a first mark M1 and a second mark M2. The first mark M1 is composed of an exclamation mark and a circular image surrounding the exclamation mark. The second mark M2 is composed of an image resembling a speech bubble. The first mark M1 is placed around a photography location icon 201 associated with annotation information. The second mark M2 is placed around a photography location icon 201 associated with additional text information. Note that when both annotation information and additional text information are associated with one photography location icon 201, the first mark M1 and the second mark M2 may be displayed around the photography location icon 201.

[0120] According to the configuration of this modified example, it is possible to present to the user a shooting location icon 201 associated with annotation information and a shooting location icon 201 associated with additional text information in a distinguished manner.

[0121] (4) The terminal device 30 may accept, from the user, input of a narrowing-down condition for narrowing down the search range when performing the above-described search process. For example, the terminal device 30 may accept the narrowing-down condition after accepting a keyword from the user and before transmitting the keyword to the information processing device 10. Then, the terminal device 30 may transmit the narrowing-down condition together with the keyword to the information processing device 10. In this modification, the control information generated by the output unit 114 may include an instruction to highlight only first icons that match the narrowing-down condition.

[0122] FIG. 10 is a diagram showing an example of a display screen G1 according to this modification. As shown in FIG. 10, the display screen G1 according to this modification includes a period input field R6. The user inputs information specifying a period into the period input field R6. The period specified by the user is an example of a filtering condition. The period input by the user is acquired by the acquisition unit 111. After acquiring the filtering condition, the acquisition unit 111 first identifies first icons based on the keywords input by the user, as described in the previous embodiment. Here, it is assumed that the shooting location icons 201A, 201B, and 201C are identified as first icons. Next, the acquisition unit 111 determines whether the identified first icons match the filtering condition. For example, it is assumed that the user inputs the filtering condition "April 1, 2024 - May 1, 2024" into the period input field R6. In this case, the acquisition unit 111 checks the shooting date and time of the image information associated with each of the identified first icons. Here, it is assumed that the shooting date and time of the image information associated with the shooting location icon 201A is April 1, 2024, and the shooting date and time of the image information associated with the shooting location icons 201B and 201C is March 1, 2024. In this case, the acquisition unit 111 determines that the shooting location icon 201A is the first icon that matches the narrowing-down condition. The acquisition unit 111 inputs this determination result to the output unit 114. Having acquired the above determination result, the output unit 114 transmits control information to the terminal device 30, including an instruction to request that only the shooting location icon 201A be highlighted.

[0123] The user may also input information specifying a specific space at the construction site (hereinafter referred to as space designation information) as a narrowing-down condition. FIG. 11 is a diagram showing another example of a display screen G1 according to this modified example. The display screen G1 shown in FIG. 11 includes a space designation button R7. When the user selects (e.g., clicks) the space designation button R7, the terminal device 30 switches the application mode from normal mode to space designation mode. In this state, the user inputs an operation to designate a specific space at the construction site. For example, the user inputs a drag operation to move the mouse cursor from point P1 to point P2 in FIG. 11. In this case, the area AR1 shown in FIG. 11 is designated as the specific space. The terminal device 30 then acquires position information on the blueprint of the space designated by the user. In the example shown in FIG. 11, the coordinates of the vertices (four corners) of area AR1 are acquired.

[0124] In addition, buttons (not shown) corresponding to spaces present at the construction site, such as "Conference Room A," "Conference Room B," "Room A," and "Elevator Hall," may be displayed on the display screen G1, and the user may select (click) one of these buttons to specify a specific space at the construction site.

[0125] The acquisition unit 111, which has received the space designation information, first identifies a first icon based on the keyword entered by the user, as described in the previous embodiment. For example, assume that the photography location icons 201A, 201B, and 201C are identified as first icons. The acquisition unit 111 then determines whether the identified first icons match the narrowing-down conditions. That is, the acquisition unit 111 determines whether the first icons are located within the space designated by the user based on the position information on the blueprint of each photography location icon 201 and the position information on the blueprint of the space designated by the user. In the example shown in FIG. 11 , of the photography location icons 201A, 201B, and 201C, only the photography location icon 201A is located within the area AR1. In this case, the acquisition unit 111 determines that the photography location icon 201A is the first icon that matches the narrowing-down conditions. The acquisition unit 111 inputs this determination result to the output unit 114. The output unit 114, which has acquired the above-mentioned determination result, transmits to the terminal device 30 control information including an instruction to highlight only the shooting location icon 201A.

[0126] Additionally, the output unit 114 may display buttons (not shown) such as "First Step," "Second Step," and "Third Step" on the display screen G1. Then, when the user selects the "First Step" button, for example, the output unit 114 may highlight only the first icon captured while the first step is being performed at the site.

[0127] According to the configuration of this modified example, only the first icons that match the narrowing conditions are highlighted, so that the location of the first icons that match the conditions desired by the user can be made clear.

[0128] The present disclosure is useful for remotely checking the status of a site.

Claims

1. An information processing method for a computer, comprising: displaying a bird's-eye view on a display of a terminal device; displaying a plurality of photography location icons indicating photography locations on the bird's-eye view; the plurality of photography location icons are associated with text information; accepting input of the text information from a user, identifying a first icon from the plurality of photography location icons associated with text information including the text information; and displaying the first icon in an emphasized manner.

2. The information processing method of claim 1, wherein image information is associated with the multiple shooting location icons, and highlighting the first icon includes displaying on the display image information associated with the first icon together with the highlighted first icon.

3. The information processing method according to claim 1 or 2, wherein displaying the first icon in an emphasized manner includes not displaying in the bird's-eye view a second icon among the plurality of shooting location icons that is associated with text information that does not include the character information.

4. The information processing method of claim 2, wherein the image information is associated with at least one attention indicator that is superimposed on the image information and displayed on the display, the text information is associated with at least one of the attention indicators, and further, when a plurality of the attention indicators are associated with the image information, only the attention indicator associated with the text information is displayed on the display.

5. The information processing method of claim 2, wherein the image information is associated with at least one attention indicator that is displayed on the display superimposed on the image information, the text information is associated with at least one of the attention indicators, and further, when a plurality of the attention indicators are associated with the image information, the attention indicator associated with the text information is displayed on the display in a manner different from the attention indicator not associated with the text information.

6. An information processing method as described in claim 4 or 5, wherein displaying the multiple shooting location icons on the bird's-eye view includes changing the display manner of the shooting location icons between a case where the text information is associated with the attention display and a case where the text information is not associated with the attention display.

7. The information processing method according to claim 1 or 2, further comprising: accepting input of a filtering condition; and highlighting the first icons comprises highlighting only the first icons that match the filtering condition.

8. An information processing device including a processor, wherein the processor executes the following operations: displaying a bird's-eye view on a display of a terminal device; displaying a plurality of shooting location icons indicating shooting locations on the bird's-eye view; the plurality of shooting location icons are associated with text information, accepting input of text information from a user, identifying a first icon from the plurality of shooting location icons associated with text information including the text information, and displaying the first icon in an emphasized manner.

9. An information processing program that causes a computer to execute the following operations: displaying a bird's-eye view on a display of a terminal device; displaying a plurality of photography location icons indicating photography locations on the bird's-eye view; the plurality of photography location icons being associated with text information, accepting input of text information from a user, identifying a first icon from the plurality of photography location icons to which text information including the text information is associated, and displaying the first icon in an emphasized manner.

Images