Information processing method, information processing device, and information processing program

Abstract

The information processing method includes: displaying a bird's-eye view on a display of a terminal device; displaying a plurality of shooting site icons representing shooting sites in the aerial view; receiving input of text information from a user by associating the text information with the plurality of shooting site icons; specifying, from among the plurality of imaging site icons, a first icon associated with text information including the text information; and displaying the first icon in an emphasized manner.

Description

Technical Field

[0001] This disclosure relates to techniques for remotely verifying the condition of a site. Background Technology

[0002] Patent Document 1 discloses a captured image management system. This system includes an image input unit, a shooting location designation unit, and a captured image display unit. The image input unit accepts input of captured images of the scene. The shooting location designation unit displays a floor map on the screen. The shooting location designation unit accepts the clicked location on the floor map as the shooting location for the captured image. The shooting location designation unit displays a marker indicating the shooting location on the floor map. The captured image display unit displays the captured image corresponding to the shooting location indicated by the marker selected by the user on the screen. The captured image display unit displays an icon overlapping the captured image. When the user selects an icon, the captured image display unit displays related documents such as descriptions and detailed views of the target captured in the captured image.

[0003] In the technology described in Patent Document 1, the computer's processing steps are increased in order to confirm the shooting location icon associated with the desired text information, thus increasing the computer's burden.

[0004] Prior art literature

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent No. 6908953 Summary of the Invention

[0007] This disclosure was made to address such a problem, and its purpose is to provide a technique that can suppress the increase in computer load.

[0008] One aspect of this disclosure is an information processing method in a computer, comprising: displaying a bird's-eye view on a display of a terminal device; displaying multiple shooting location icons representing shooting locations on the bird's-eye view; establishing a correspondence between text information and the multiple shooting location icons, receiving text information input from a user; determining a first icon from the multiple shooting location icons that corresponds to the text information containing the text information; and highlighting the first icon.

[0009] According to this disclosure, users can easily determine the location of the shooting location icon corresponding to the desired text information. As a result, the increase in computer processing steps is suppressed, thus reducing the increase in computer load. Attached Figure Description

[0010] Figure 1This is a block diagram illustrating an example of the overall structure of the information processing system in Embodiment 1 of this disclosure.

[0011] Figure 2 This is an example of a display screen shown on a terminal device's monitor.

[0012] Figure 3 This is a flowchart illustrating the process performed in the information processing system according to Embodiment 1.

[0013] Figure 4 This is a flowchart showing the details of the second process when establishing a correspondence between annotation information and shooting location icons.

[0014] Figure 5 This is a flowchart showing the details of the second process when additional text information is mapped to the shooting location icon.

[0015] Figure 6 It is shown Figure 3 The flowchart shows the detailed process of the third step.

[0016] Figure 7 It is shown in Figure 6 A diagram showing an example of the display screen shown on the monitor in step S301.

[0017] Figure 8 This is a diagram showing an example of the display screen shown on the monitor in the first and second variations.

[0018] Figure 9 This is a diagram showing an example of the display screen shown on the monitor in the third variation.

[0019] Figure 10 This is a diagram showing an example of the display screen shown on the monitor in the fourth variation.

[0020] Figure 11 This is a diagram showing another example of the display screen shown on the monitor in the fourth variation. Detailed Implementation

[0021] (A method for realizing this disclosure)

[0022] As for issues at construction sites and other on-site locations, there are problems such as specific instructions not being communicated to workers, time spent explaining the instructions, the need for personnel to patrol the entire site, and time spent moving to the site.

[0023] To address these issues, one approach is to install multiple cameras on-site, allowing remote users to refer to images from these cameras and issue instructions to workers. However, this is impractical. Specifically, in the aforementioned methods, as on-site work progresses, tasks such as dismantling the installed cameras and relocating them to another location arise. Since such tasks are time-consuming, installing multiple cameras on the construction site is not feasible. Therefore, the inventors have researched a technique that allows for remote monitoring of the construction site without the need for multiple cameras.

[0024] The inventors realized that if a user interface displays multiple shooting location icons overlaid on a bird's-eye view corresponding to the site, the user can remotely confirm the site's condition. Specifically, they realized that by establishing a correspondence between text information representing on-site personnel's comments, notes, etc., and the aforementioned multiple shooting location icons, and by outputting text information corresponding to the selected shooting location icon when the user selects any one of the icons, the user can remotely confirm the site's condition.

[0025] However, in the aforementioned user interface, users are sometimes forced to find the location icons that correspond to the desired text information. Specifically, when a large number of location icons are displayed in the bird's-eye view and text information is associated with these icons, it becomes difficult for users to determine which text information corresponds to which location icon. In this situation, users must select each location icon one by one to find the one that corresponds to the desired text information. This increases the computer's processing load due to the increased number of steps involved.

[0026] To this end, the inventors conducted further research and found that if the shooting location icon corresponding to the user's desired text information can be explicitly shown to the user, the task of finding the shooting location icon can be omitted, thus reducing the increase in computer processing steps.

[0027] This disclosure is based on the following understanding.

[0028] (1) An information processing method in one aspect of the present disclosure is an information processing method in a computer, the information processing method comprising: displaying a bird's-eye view on a display of a terminal device; displaying a plurality of shooting location icons representing shooting locations in the bird's-eye view; establishing a correspondence between text information and the plurality of shooting location icons, receiving text information input from a user; determining a first icon from the plurality of shooting location icons that corresponds to the text information containing the text information; and highlighting the first icon.

[0029] According to this structure, the location icon corresponding to the text information containing user-inputted text information, i.e., the first icon, is prominently displayed. In this way, the user can easily locate the location icon corresponding to the desired text information, i.e., the first icon. As a result, the increase in computer processing steps is suppressed, thus reducing the computer's workload.

[0030] (2) In the information processing method described in (1) above, the image information is also associated with the multiple shooting location icons, and highlighting the first icon includes: together with the highlighted first icon, displaying the image information associated with the first icon on the display.

[0031] According to this structure, convenience is improved because the image information corresponding to the first icon, along with the first icon that is highlighted, is displayed on the screen.

[0032] (3) In the information processing method described in (1) or (2) above, the first icon may be emphasized to include: the second icon corresponding to the text information that does not contain the text information among the multiple shooting location icons may be displayed in the bird's-eye view.

[0033] According to this structure, since the second icon is not displayed on the screen, users can more reliably determine the location of the first icon.

[0034] (4) In the information processing method described in (2) or (3) above, at least one attention display that overlaps with the image information on the display is established in correspondence with the image information, and the text information is established in correspondence with at least one of the attention displays. The information processing method further includes: when multiple attention displays are established in correspondence with the image information, only the attention display that is established in correspondence with the text information is displayed on the display.

[0035] According to this structure, since only the attention display corresponding to the text information is displayed on the monitor, even when multiple attention displays correspond to image information, it enables the user to more reliably understand which attention display corresponds to the text information.

[0036] (5) In the information processing method described in (2) or (3) above, at least one attention display that overlaps with the image information on the display is established in correspondence with the image information, and the text information is established in correspondence with at least one of the attention displays. The information processing method further includes: when multiple attention displays are established in correspondence with the image information, displaying the attention display that is established in correspondence with the text information on the display in a manner different from the attention display that is not established in correspondence with the text information.

[0037] Based on this structure, the display method can differ between attention displays that correspond to text information and attention displays that do not. Thus, when multiple attention displays correspond to image information, users can more reliably understand which attention display corresponds to the text information.

[0038] (6) In the information processing method described in (4) or (5) above, the display of the multiple shooting location icons in the bird's-eye view may include: making the display method of the shooting location icons different when the text information and the attention display are established, and when the text information and the attention display are not established.

[0039] Based on this structure, it is possible to distinguish between text information that has a corresponding shooting location icon displayed for attention and text information that has not a corresponding shooting location icon displayed for attention, and to provide users with a clear indication of the difference.

[0040] (7) In any of the information processing methods described in (1) to (6) above, the information processing method may further include: accepting the input of the filtering conditions, and highlighting the first icon includes: highlighting only the first icon that meets the filtering conditions.

[0041] According to this structure, since only the first icon that meets the filtering criteria is highlighted, the location of the first icon that meets the user's expected criteria can be clearly identified.

[0042] Furthermore, this disclosure can be implemented not only as an information processing method that performs the characteristic processes described above, but also as an information processing apparatus or the like that having a characteristic structure corresponding to the characteristic processes performed by the information processing method. It can also be implemented as a computer program that causes a computer to execute the characteristic processes included in such an information processing method. Therefore, the following other methods can also achieve the same effect as the information processing method described above.

[0043] (8) Another aspect of the present disclosure relates to an information processing apparatus that includes a processor, the processor performing the following processing: displaying a bird's-eye view on a display of a terminal device; displaying multiple shooting location icons representing shooting locations in the bird's-eye view; establishing a correspondence between text information and the multiple shooting location icons, receiving text information input from a user; determining a first icon from the multiple shooting location icons that corresponds to the text information containing the text information; and highlighting the first icon.

[0044] (9) The information processing program involved in another aspect of this disclosure causes a computer to perform the following processing: displaying a bird's-eye view on the display of a terminal device; displaying multiple shooting location icons representing shooting locations in the bird's-eye view; establishing a correspondence between text information and the multiple shooting location icons, receiving text information input from a user; determining a first icon from the multiple shooting location icons that corresponds to the text information containing the text information; and highlighting the first icon.

[0045] This disclosure can also be implemented as an information processing system operating according to such an information processing program. Furthermore, it goes without saying that such a program can be distributed via non-transitory recording media such as computer-readable CD-ROMs or communication networks such as the Internet.

[0046] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Furthermore, the embodiments described below represent specific examples of the present disclosure. The numerical values, shapes, constituent elements, steps, and order of steps shown in the following embodiments are examples and are not intended to limit the present disclosure. Additionally, any constituent elements in the following embodiments that are not described in the independent claim representing the highest-level concept are described as arbitrary constituent elements. Furthermore, in all embodiments, the various contents can be combined.

[0047] (Implementation Method 1)

[0048] Figure 1 This is a block diagram illustrating an example of the structure of the information processing system 1 in Embodiment 1 of this disclosure. The information processing system 1 includes an information processing device 10, an imaging device 20, and a terminal device 30. The information processing device 10, the imaging device 20, and the terminal device 30 are communicatively connected via a network. An example of such a network is the Internet. The information processing device 10 is, for example, a cloud server composed of one or more computers. However, this is just one example; the information processing device 10 may also be composed of an edge server or may be installed on the terminal device 30.

[0049] The shooting device 20, for example, is an omnidirectional camera that captures images at a given frame rate. The shooting device 20 is, for example, a portable shooting device held by a user. The user is, for example, a site worker or site supervisor. The site is, for example, a construction site. The user moves within the construction site while shooting at it using the shooting device 20. Specifically, at the starting point of the shooting, the user orients the shooting device 20 towards a reference direction, presses the shooting button on the shooting device 20 to begin shooting, and presses the shooting button again to end shooting when reaching the ending point. The reference direction is, for example, north, but is not limited to this. The shooting device 20 sends image information representing the captured images to the information processing device 10 via a network. Thus, the information processing device 10 can obtain image information from multiple shooting locations within the construction site. Here, the shooting device 20 captures images at a given frame rate, therefore the shooting locations are defined in units of frame periods. However, this is just one example; shooting locations can also be defined at given intervals (e.g., 1 second, 1 minute, etc.). The location of the shooting location and the shooting date and time are correlated with the image information. The location of the shooting location is obtained, for example, by a position sensor such as a magnetic sensor or GPS sensor equipped on the shooting device 20, and expressed in latitude and longitude. The shooting date and time are obtained, for example, by a clock equipped on the shooting device 20. The shooting device 20 includes an image sensor, an operating device, a communication circuit, a signal processing circuit, etc. The shooting device 20 may also be a portable computer such as a smartphone or a tablet computer.

[0050] The terminal device 30 is held by the user. The terminal device 30 can be a portable computer, such as a smartphone or tablet, or a fixed-position computer. The terminal device 30 displays image information on the display 31 under the control of the information processing device 10. Figure 1 The example shows one terminal device 30, but multiple terminal devices can also be connected via a network in the information processing device 10. The terminal device 30 includes: a central processing unit (CPU), memory, display 31, touch panel, mouse and keyboard and other operating devices, and communication circuitry.

[0051] The information processing device 10 includes a processor 11, a memory 12, and a communication unit 13. The processor 11 is, for example, a central processing unit (CPU). The processor 11 includes an acquisition unit 111, a setting unit 113, and an output unit 114. The acquisition unit 111 to the output unit 114 can be implemented by the processor 11 executing an information processing program, or they can be constructed by dedicated hardware circuits such as ASICs.

[0052] The acquisition unit 111 acquires a display instruction from the terminal device 30 via the communication unit 13. The display instruction is an instruction input by the user to the terminal device 30, and is an instruction requesting that a given design drawing be displayed on the display 31. The display instruction includes a design drawing ID, which serves as an identifier for the design drawing. The acquisition unit 111 reads the design drawing represented by this design drawing ID from the memory 12. The design drawing is image data representing a site. A design drawing is an example of a bird's-eye view. The acquisition unit 111 may also acquire a top view, blueprint, map, or perspective view of the construction site instead of a design drawing. Top views, blueprints, maps, and perspective views are another example of bird's-eye views. In the design drawing, latitude and longitude are pre-established to correspond to locations that become key points. Locations that become key points are, for example, the four corners of the design drawing.

[0053] Additionally, the acquisition unit 111 acquires text information input by the user into the terminal device 30 via the communication unit 13. Based on the text information, the acquisition unit 111 determines at least one first icon from a plurality of shooting location icons 201. The first icon is a shooting location icon 201 that corresponds to text information including the text information input by the user. The acquisition unit 111 inputs the determination result of the first icon to the output unit 114. At the same time, the acquisition unit 111 reads the image information corresponding to the first icon from the memory 12 and inputs it to the output unit 114.

[0054] Additionally, the acquisition unit 111 acquires a selection instruction from the terminal device 30 via the communication unit 13. The selection instruction is sent from the terminal device 30 to the acquisition unit 111 when the user selects one of the multiple shooting location icons 201 displayed in the design drawing. Upon receiving the selection instruction, the acquisition unit 111 acquires shooting information corresponding to the shooting location icon 201 selected by the user. The acquisition unit 111 then inputs the shooting information to the output unit 114.

[0055] The setting unit 113 decides to display the image information corresponding to the first icon or the image information corresponding to the shooting location icon 201 selected by the user in the image information display bar R2 (described later). Figure 2The first display range is determined by the setting unit 113. As described above, in this embodiment, a 360-degree camera is used for shooting. It is difficult to display the entire image captured by the 360-degree camera in the image information display bar R2 at once. Therefore, the setting unit 113 determines a portion of the entire image information as the first display range. This first display range is displayed in the image information display bar R2. For example, the setting unit 113 determines the first display range such that the reference direction (north direction in this example) is located at the center of the image information display bar R2. That is, the setting unit 113 determines the first display range such that the area around the reference direction is displayed in the image information display bar R2. In addition, the setting unit 113 determines the magnification of the first display range when the first display range is displayed in the image information display bar R2.

[0056] The output unit 114 generates control information containing instructions for displaying various display screens on the display 31. The output unit 114 generates control information based on various information received from the acquisition unit 111 and the setting unit 113.

[0057] Figure 2 This diagram illustrates an example of a terminal device 30 receiving control information from the output unit 114 and displaying it on a display screen G1 on the monitor 31. Display screen G1 is the basic screen of the application (user interface) provided by the information processing system 1. For example... Figure 2 As shown, the display screen G1 includes: a design drawing display bar R1, an image information display bar R2, a text information display bar R3, and a message display bar R4.

[0058] The design drawing display panel R1 displays the design drawing. For example... Figure 2 As shown, in the design drawing displayed in the design drawing display bar R1, multiple shooting location icons 201 and tracks 202 are displayed overlappingly.

[0059] Multiple shooting location icons 201 represent the shooting locations of multiple image information captured by the shooting device 20. In this example, each of the multiple shooting location icons 201 is composed of a circular image. Text information and image information are respectively associated with the multiple shooting location icons 201.

[0060] Multiple shooting location icons 201 represent a group of shooting location icons from the start point to the end point of the shooting. It includes icon 1 and icon 2. Icon 1 is a shooting location icon corresponding to text information containing keywords entered by the user. Icon 2 is a shooting location icon corresponding to text information that does not contain keywords entered by the user.

[0061] When the first icon is determined by the acquisition unit 111, the output unit 114 sends control information for emphasizing the first icon to the terminal device 30. Details will be described later.

[0062] Furthermore, when the acquisition unit 111 acquires a selection instruction, that is, when the user selects any one of the multiple shooting location icons 201, the output unit 114 sends control information to the terminal device 30 to make the display mode of the user-selected shooting location icon 201 (hereinafter referred to as the selection icon) different from the display modes of the other shooting location icons 201. For example, the output unit 114 sends control information including an instruction to make the color of the selection icon different from the colors of the other shooting location icons 201 to the terminal device 30.

[0063] Track 202 represents the movement path trajectory of the user who captured the image information. In this example, track 202 consists of lines connecting adjacent shooting location icons 201 to each other. Shooting location icons 201 located at the beginning and end of track 202 are displayed at a larger size than the other shooting location icons 201. The icons located at the beginning of track 202 (e.g., at the end of track 202) are larger than the others. Figure 2 The shooting location icon 201 (on the right end) indicates the shooting start position, located at the rear end of trajectory 202 (e.g., Figure 2 The shooting location icon 201 (on the left) indicates the end position of the shooting.

[0064] The image information display bar R2 displays image information corresponding to the shooting location icon 201. For example, when the acquisition unit 111 determines the first icon, the image information display bar R2 displays the image information corresponding to the first icon. Additionally, when the user selects a given shooting location icon 201, the image information display bar R2 may also display the image information corresponding to the selected icon. Furthermore, if the first icon is not determined and the user does not select a shooting location icon 201, the image information display bar R2 may be empty.

[0065] The attention display shown on display 31, which overlaps with image information, sometimes corresponds to the image information. In this case, the image information display bar R2 draws the attention display within the image information. For example, in... Figure 2 In the image information display bar R2, a rectangular frame D is drawn within the image information IM1. The rectangular frame D is an example of an area of ​​focus. For instance, the rectangular frame D is assigned to the image information IM1 by on-site personnel or others to emphasize a given target or area captured in the image information IM1. In other words, sometimes bounding boxes or similar frames are displayed overlapping in the image information.

[0066] The text information display bar R3 shows the text information corresponding to the first icon. The text information consists of text (strings).

[0067] In this embodiment, the text information is distinguished into annotation information and supplementary text information. Annotation information is text information that corresponds to the bounding box D (focus display) in the image information. That is, when a focus display such as a bounding box is displayed in the image information display bar R2 and strings such as notes, comments, and captions are associated with (corresponding to) the bounding box, the information processing device 10 processes these strings as annotation information.

[0068] On the other hand, text information that corresponds to the first icon but not to the attention display is supplementary text information. Sometimes, strings such as notes, comments, and captions are associated with the first icon without being related to the attention display. For example, sometimes a string describing the overall characteristics of an image that corresponds to the first icon is associated with the first icon. In addition, for example, sometimes a string describing the shooting location where the first icon is displayed is associated with the first icon. Furthermore, sometimes the exchange of messages between users is associated with the first icon. The information processing device 10 processes these strings as supplementary text information.

[0069] exist Figure 2 In the example shown, the text information display bar R3 displays a list of text information C1 entered by multiple users in response to the image information displayed in the image information display bar R2. In the text information display bar R3, the left side displays text information C1 entered by users other than the user themselves, and the right side displays the text information C1 entered by the user themselves.

[0070] As an example, when the text information is comment information, output unit 114 displays the attention display ID at the beginning of the text information. The attention display ID is an identifier for attention display. For example, such as... Figure 2 As shown, suppose a user inputs the text message "A new occupancy began at 1 PM today, and this location is being used." Then, suppose the user associates this text message with the rectangular frame D displayed in the image information display bar R2. In this case, the output unit 114 obtains the attention display ID of the rectangular frame D and appends this attention display ID to the beginning of the user-input text message. That is, as... Figure 2 As shown, the text message "ID001: Someone moved in and started using this place starting at 13:00 today" is displayed.

[0071] In addition, the output unit 114 can also display the attention display ID as a link text in the text information display bar R3. When the user clicks the attention display ID, the attention display corresponding to the attention display ID will be highlighted. For example, the rectangular frame D of the image information display bar R2 can also be made to blink.

[0072] Furthermore, if the first icon is not selected and the user does not select the shooting location icon 201, the text information display bar R3 can also be empty.

[0073] The message display bar R4 displays various messages related to the scene. For example, messages indicating that a text message has been associated with a given shooting location icon 201 are displayed in the message display bar R4.

[0074] Return to Figure 1 The memory 12 is composed of a non-volatile, rewritable storage device such as a hard disk drive or a solid-state drive. The memory 12 includes: a design information storage unit 121, a photographic information storage unit 122, an image information storage unit 123, and a text information storage unit 124.

[0075] The design drawing information storage unit 121 stores at least one design drawing. Additionally, the design drawing information storage unit 121 stores the design drawing ID assigned to each design drawing. Furthermore, the design drawing information storage unit 121 stores the latitude and longitude of key points (e.g., the four corners) of each design drawing.

[0076] The shooting information storage unit 122 stores shooting information related to a single shooting action using the shooting device 20. Shooting information is generated each time a shooting action is performed. A single shooting action refers to a series of actions performed by the operator holding the shooting device 20 on-site from the start to the end of shooting. Multiple image information items are generated through a single shooting action.

[0077] The shooting information includes: design drawing ID, shooting ID, shooting date and time, representative value of shooting date and time, location information of shooting location, and location information of shooting location icon 201. The design drawing ID is an identifier for the design drawing at the site where the shooting action took place. The shooting ID is an identifier used to identify each shot included in a single shooting action. The shooting date and time indicates the year, month, day, and time of the shot represented by the shooting ID. The representative value of the shooting date and time, for example, indicates the year, month, day, and time when the shooting began. The shooting location indicates the location (latitude and longitude) of the shot represented by the shooting ID. The location information of shooting location icon 201 indicates the display position (coordinates) of the shooting location icon 201 corresponding to the shot represented by the shooting ID on the design drawing. Furthermore, based on the latitude and longitude of the key points (four corners) of the design drawing stored in the design drawing information storage unit 121, and the location information (latitude and longitude) of the shooting location, the shooting location is mapped onto the design drawing for each shot represented by the shooting ID, thereby calculating the display position of shooting location icon 201 on the design drawing.

[0078] Image information storage unit 123 stores image information. Image information refers to a single image generated by each shot taken in a single shooting action. As described above, in this embodiment, the shooting device 20 shoots at a given frame rate (e.g., 30fps), and therefore stores multiple images in image information storage unit 123. A shooting ID is established to correspond to each image stored in image information storage unit 123. Using the shooting ID as a key, the information processing device 10 can determine the shooting information that corresponds to the image information. In addition, when image information is given attention display, attention display ID used to identify the attention display is established to correspond to the image information. When multiple attention displays are established to correspond to one image, multiple attention display IDs are established to correspond to one image. In addition, when image information is given attention display, image information storage unit 123 stores attention display position information indicating the display position on the image information of attention display. The attention display position information includes, for example, the coordinate values ​​of the image information of each vertex of the rectangular frame D.

[0079] The text information storage unit 124 stores text information corresponding to each shooting location icon 201. Additionally, a shooting ID is associated with each piece of text information. Using this shooting ID as a key, the information processing device 10 determines the display position of the shooting location icon 201 corresponding to the text information on the design drawing.

[0080] Furthermore, the text information storage unit 124 stores differentiation information associated with each piece of text information. This differentiation information indicates whether the text information is comment information or supplementary text information.

[0081] Additionally, the text information storage unit 124 stores a attention display ID used to determine which attention display is associated with the annotation information. When multiple attention displays are assigned to image information, the information processing device 10 uses this attention display ID to determine which attention display is associated with the annotation information.

[0082] In this way, since a correspondence is established between the shooting ID, the shooting information, and the image information, the location information of the shooting location icon 201 can be determined using the shooting ID as the key. That is, the image information corresponding to the shooting location icon 201 can be identified. Furthermore, since a correspondence is established between the shooting ID and the text information, the location information of the shooting location icon 201 corresponding to the text information can be determined using the shooting ID as the key. That is, the text information corresponding to the shooting location icon 201 can be identified.

[0083] The communication unit 13 is a communication circuit that connects the information processing device 10 to the network.

[0084] Figure 3 This is a flowchart illustrating the processing flow performed in the information processing system 1 according to Embodiment 1. For example... Figure 3 As shown, in Implementation 1, the second process is performed after the first process, and the third process is performed after the second process. However, this is just one example, and the order of the processes can be changed appropriately.

[0085] In step S11 of the first processing step, the imaging device 20 uploads the imaging data. Specifically, in step S11, the photographer performs the shooting action. Correspondingly, the imaging device 20 generates multiple image information and shooting information corresponding to each of the multiple image information. Furthermore, the imaging device 20 sends the multiple shooting information and the shooting information corresponding to each of the multiple image information to the information processing device 10.

[0086] In step S12, the acquisition unit 111 acquires a series of uploaded shooting data. Specifically, the acquisition unit 111 acquires multiple shooting information and shooting information corresponding to the multiple image information through the communication unit 13.

[0087] In step S13, the acquisition unit 111 maps multiple shooting location icons 201 and tracks 202 onto the design drawing. Specifically, the acquisition unit 111 uses multiple received image information and shooting information corresponding to each of the multiple image information to map multiple shooting location icons 201 and tracks 202 onto the design drawing.

[0088] In step S21 of the second processing step, the terminal device 30 receives annotation information or additional text information for the shooting location icons 201 mapped on the design drawing. Specifically, in step S21, the user selects any one of the multiple shooting location icons 201. The user then inputs annotation information or additional text information corresponding to the selected shooting location icon 201. The terminal device 30, having received the annotation information or additional text information from the user, sends this information to the information processing device 10.

[0089] In step S22, the acquisition unit 111 of the information processing device 10 acquires the annotation information or additional text information sent from the terminal device 30.

[0090] In step S23, the acquisition unit 111 associates the shooting location icon 201 selected by the user in step S21 with the annotation information or additional text information (establishes a correspondence).

[0091] In step S31 of the third processing step, the terminal device 30 receives the text information that the user wants to retrieve. Additionally, the terminal device 30 sends the received text information to the information processing device 10.

[0092] In step S32, the acquisition unit 111 of the information processing device 10 acquires (receives) the text information that the user wants to retrieve.

[0093] In step S33, the acquisition unit 111 performs a determination process. In step S34, the output unit 114 generates control information based on the determination result of the determination process and sends the control information to the terminal device 30. In step S35, the terminal device 30 emphasizes the first icon based on the control information. For details regarding the processing of steps S33 to S35, please refer to... Figure 6 To be discussed later.

[0094] Figure 4 This illustrates the second process where a correspondence is established between the annotation information and the shooting location icon 201. Figure 3 The flowchart details steps S21 to S23.

[0095] In step S201, the terminal device 30 displays a display page containing image information corresponding to a certain shooting location icon 201 on the display screen 31.

[0096] Specifically, in step S201, the terminal device 30 receives a display instruction from the user. Upon receiving the display instruction, the terminal device 30 sends the display instruction to the information processing device 10. The acquisition unit 111 of the information processing device 10, having received the display instruction, reads the design drawing specified by the user from the memory 12. Furthermore, the output unit 114 of the information processing device 10 sends an instruction to the terminal device 30 to display the design drawing read by the acquisition unit 111 on the display 31. The terminal device 30, having received this instruction, displays the design drawing on the display 31.

[0097] Next, the terminal device 30 receives a selection instruction from the user. Specifically, the user inputs an operation to select a shooting location icon 201 associated with the image information for which they want to input text information into the terminal device 30. The selection instruction includes the location information of the shooting location icon 201 selected by the user. That is, it includes information indicating whether any one of the multiple shooting location icons 201 displayed in the design drawing has been selected. The terminal device 30 sends the selection instruction to the acquisition unit 111. The acquisition unit 111, having received the selection instruction, reads the image information corresponding to the shooting location icon 201 selected by the user from the memory 12 and inputs the image information to the output unit 114. The output unit 114 sends an instruction to the terminal device 30 to display the image information read from the memory 12 by the acquisition unit 111 on the display 31. Upon receiving this instruction, the terminal device 30 displays the image information on the display 31. That is, the terminal device 30 transitions the screen of the display 31 from the display page showing the design drawing to the display page showing the image information corresponding to the shooting location icon 201 selected by the user.

[0098] In step S202, the terminal device 30 displays a rectangular frame D at the location selected by the user and obtains the position information of the image information of the rectangular frame D.

[0099] Specifically, in step S202, the user inputs an operation to specify (select) a given area in the image information into the terminal device 30. Specifying a given area in the image information can be, for example, a drag operation. The terminal device 30, receiving this operation, displays a rectangular frame D containing the user-specified area overlapping the image information. At this time, the terminal device 30 acquires the position information (coordinate values) of each vertex of the rectangular frame D in the image information. In other words, it acquires the position information of the area to be displayed.

[0100] In step S203, the terminal device 30 receives text information from the user. That is, it receives and... Figure 2 The annotation information associated with the rectangular frame D shown. For example, terminal device 30 receives... Figure 2The text information display bar R3 shows strings such as "Movers-in began at 13:00 today. Using this venue" as annotation information.

[0101] In step S204, the terminal device 30 sends the position information (coordinate values) on the image information of the rectangular frame D and the text information (annotation information) received from the user to the information processing device 10.

[0102] In detail, the terminal device 30 sends the information that notifies the user of a given area on the image information and assigns attention display (rectangular box D) to the image information, the position information on the image information of the attention display, and the text information (annotation information) input by the user corresponding to the attention display to the information processing device 10.

[0103] In step S205, the acquisition unit 111 of the information processing device 10 receives position information on the image information of the rectangular frame D and text information (annotation information) received by the terminal device 30.

[0104] In step S206, the acquisition unit 111 associates the position information on the image information of the rectangular frame D, the text information input by the user corresponding to the rectangular frame D, and the position information of the shooting location icon 201 selected by the user in step S201, or the image information corresponding to the shooting location icon 201 (establishing a correspondence). The acquisition unit 111 stores this correspondence in the memory 12.

[0105] Figure 5 This is a flowchart showing the details of the second process when the additional text information is associated with the shooting location icon 201.

[0106] exist Figure 5 In step S211 shown, the following is performed: Figure 4 The same process applies to step S201 shown.

[0107] exist Figure 5 In the flowchart shown, no equivalent to Figure 4 The process shown in step S202.

[0108] exist Figure 5 In steps S213 to S216 shown, the following steps are performed: Figure 4The same processing is performed in steps S203 to S206. However, in step S214, unlike step S204, the terminal device 30 does not send the position information on the image information of the rectangular frame D. Furthermore, in step S215, unlike step S205, the acquisition unit 111 does not receive the position information on the image information of the rectangular frame D. Additionally, in step S216, the acquisition unit 111 associates the text information with the position information of the shooting location icon 201 selected by the user in step S211, or with the corresponding image information established with that shooting location icon 201. That is, in step S216, the processing of associating the position information on the image information of the rectangular frame D with the position information of the shooting location icon 201 selected by the user in step S211, or with the corresponding image information established with that shooting location icon 201, is not performed.

[0109] By execution Figure 4 The processes involved in steps S201 to S206, and Figure 5 The processing involved in steps S211 to S216 involves storing the text information (comment information or additional text information) input by the user in the text information storage unit 124 of the memory 12.

[0110] Figure 6 It is shown Figure 3 The flowchart shows the detailed process of the third step.

[0111] In step S301, the terminal device 30 receives text information from the user.

[0112] Specifically, in step S301, the output unit 114 sends an instruction to the terminal device 30 requesting that the display screen G1 be displayed on the monitor 31. Upon receiving the instruction, the terminal device 30 displays the display screen G1 on the monitor 31. Figure 7 This is a diagram illustrating an example of the display screen G1 shown in step S301. For example... Figure 7 As shown, the display screen G1 includes a search window R5. In step S301, the user enters text information into the search window R5.

[0113] In step S302, the terminal device 30 sends the text information entered by the user into the search window R5 to the information processing device 10.

[0114] In step S303, the acquisition unit 111 receives text information.

[0115] In step S304, the acquisition unit 111 performs a determination process to determine whether text information containing user-inputted text information is stored in the text information storage unit 124. Specifically, the acquisition unit 111 sets a search range so that all text information stored in the text information storage unit 124 of the memory 12 becomes the search object, sets the user-inputted text information as a keyword, and performs a search process. Thus, the acquisition unit 111 determines whether text information containing a string matching the aforementioned keyword exists in the text information storage unit 124. Here, if text information containing user-inputted text information is stored in the text information storage unit 124, the acquisition unit 111 determines that a shooting location icon 201 corresponding to that text information has been established. That is, the first icon is determined. Furthermore, the acquisition unit 111 retrieves various information associated with the first icon from the memory 12, such as the location information on the design drawing of the first icon and the image information corresponding to the first icon. Furthermore, if the acquisition unit 111 cannot determine the first icon, the process ends.

[0116] In step S305, the output unit 114 sends control information to the terminal device 30. Specifically, the output unit 114 generates control information based on the result of the determination process of the acquisition unit 111 in step S304. The control information includes an indication to emphasize the first icon. The output unit 114 then sends the control information to the terminal device 30.

[0117] In step S306, the terminal device 30 receives control information.

[0118] In step S307, the terminal device 30 displays the display screen G1, in which the first icon is highlighted, on the display screen 31 based on the control information.

[0119] Reference Figure 7 The third process will be explained in detail. First, the user enters text information into the search window R5 (step S301). Figure 7 In the example shown, the user inputs the text information "moved in" into the search window R5. Hereinafter, the text information input by the user into the search window R5 will be referred to as a keyword. The terminal device 30 sends the keyword to the information processing device 10 (step S302). The keyword "moved in" entered by the user is acquired by the acquisition unit 111 (step S303), and the acquisition unit 111 determines that a corresponding shooting location icon 201, i.e., the first icon, has been established with the text information containing the keyword "moved in" (step S304). Here, it is assumed that... Figure 7Among the multiple shooting location icons 201 shown, shooting location icons 201A, 201B, and 201C are determined as the first icon. Based on the determination result of the acquisition unit 111, the output unit 114 generates control information containing an instruction to emphasize the first icon (step S305). That is, the output unit 114 generates control information containing an instruction to emphasize the shooting location icons 201A, 201B, and 201C in the display screen G1. The terminal device 30, which receives the control information from the output unit 114 (step S306), emphasizes the first icon based on the control information (step S307).

[0120] The instruction to emphasize the first icon is, for example, an instruction to display the first icon differently from the second icon. In this example, the shooting location icon 201, which corresponds to text information that does not contain keywords such as "moved in," is equivalent to the second icon. Specifically, Figure 7 Of the multiple shooting location icons 201 shown, the shooting location icons 201 other than shooting location icons 201A, 201B, and 201C are equivalent to the second icon.

[0121] The instruction to display the first icon differently from the second icon could be, for example, an instruction to display the first icon in a different color than the second icon. Specifically, it could be an instruction to display the first icon in red, for example, and the second icon in yellow, for example. Alternatively, the instruction to display the first icon differently from the second icon could also be an instruction to make the radius of the circle constituting the first icon relatively larger than the radius of the circle constituting the second icon. Alternatively, it could be an instruction to display only the first icon among the multiple shooting location icons 201 in the design view display bar R1, and not display the second icon in the design view display bar R1 (e.g., through). Furthermore, the output unit 114 can also send various instructions to the terminal device 30 to emphasize the display of the first icon.

[0122] Alternatively, the output unit 114 can also generate an instruction requesting that image information corresponding to the highlighted first icon be displayed in the image information display bar R2. For example, the output unit 114 can also generate control information including the following instruction: requesting that the shooting location icon 201A (first icon) be displayed in red, the second icon in yellow, and that image information corresponding to the shooting location icon 201A be displayed in the image information display bar R2. In this way, the user can confirm that image information corresponding to the first icon has been established, thus improving the convenience of the application.

[0123] Furthermore, when multiple first icons exist, the output unit 114 selects any one of the multiple first icons. It also generates control information that indicates the image information display bar R2 should display image information corresponding to the selected first icon. For example, the output unit 114 selects the first icon with the most recent shooting date and time from the multiple first icons. However, this is just one example; the output unit 114 only needs to select one first icon using an appropriate method.

[0124] When any one of the multiple first icons is selected, the output unit 114 can also generate control information that includes instructions to make the display mode of the selected first icon different from the other first icons. For example, suppose that the shooting location icon 201A is selected from the multiple first icons. In this case, the output unit 114 can also generate control information that includes instructions to display the shooting location icon 201A in red, the shooting location icons 201B and 201C in orange, and the second icon in yellow.

[0125] According to the information processing system 1 described above, a corresponding shooting location icon 201, i.e., the first icon, is established with the text information, including the text information entered by the user, and is prominently displayed. In this way, the user can easily grasp the location of the shooting location icon 201, i.e., the first icon, which corresponds to the desired text information. As a result, the increase in processing steps of the information processing device 10, etc., is suppressed, thus reducing the burden on the information processing device 10, etc.

[0126] The following variations are possible with respect to this disclosure.

[0127] (1) When multiple attention displays are associated with image information, the control information generated by the output unit 114 may also include an instruction to display only the attention display associated with the text information in the image information display bar R2.

[0128] Figure 8 This diagram illustrates an example of the display screen G1 involved in this variation. In the image information IM2 displayed in the image information display bar R2 involved in this variation, multiple attention displays are correspondingly established. As a result, rectangular frames D1, D2, and D3 are drawn in the image information IM2. Here, it is envisioned that text information (annotation information) is associated with the rectangular frames D1, and that the text information is not associated with the rectangular frames D2 and D3. In this case, the output unit 114 generates control information containing an instruction requesting that only the rectangular frames D1 be drawn on the image information IM2.

[0129] According to this variation, when multiple attention displays are associated with image information, the user can more reliably understand which attention display corresponds to the text information.

[0130] (2) When multiple attention displays are associated with image information, the control information generated by the output unit 114 may also include an instruction to display attention displays associated with text information in a different manner than attention displays not associated with text information in the image information display bar R2.

[0131] Reference Figure 8 To illustrate. Similar to variation (1), imagine combining text information (annotation information) with... Figure 8 The rectangular frame D1 is associated with the text information, but the text information is not associated with the rectangular frames D2 and D3. In this case, the output unit 114 generates control information containing an indication that the display mode of the rectangular frame D1 is different from that of the rectangular frames D2 and D3. For example, control information is generated containing an indication that the rectangular frame D1 is displayed in red and the rectangular frames D2 and D3 are displayed in pink. According to this modified example, the same effect as modified example (1) can be achieved.

[0132] (3) The control information generated by the output unit 114 may also include the following instructions: when the text information corresponds to the attention display and when the text information does not correspond to the attention display, the display mode of the shooting location icon 201 shall be different. That is to say, the control information may also include the following instructions: when the annotation information among multiple shooting location icons 201 corresponds to the shooting location icon 201, and when the shooting location icon 201 corresponds to the additional text information, the display mode of the shooting location icon 201 shall be different.

[0133] For example, the output unit 114 generates control information containing the following instructions: requesting that a first marker M1 be configured around the shooting location icon 201 corresponding to the annotation information, and a second marker M2 be configured around the shooting location icon 201 corresponding to the additional text information. Figure 9 This is a diagram illustrating an example of the display screen G1 involved in this variation. For example... Figure 9As shown, the design display bar R1 of the display screen G1 includes a first marker M1 and a second marker M2. The first marker M1 consists of an exclamation mark and an image of a circle surrounding the exclamation mark. The second marker M2 consists of an image mimicking a speech bubble. The first marker M1 is positioned around the shooting location icon 201 that corresponds to the annotation information. The second marker M2 is positioned around the shooting location icon 201 that corresponds to the additional text information. Furthermore, when both the annotation information and the additional text information are associated with a single shooting location icon 201, the first marker M1 and the second marker M2 can also be displayed around that shooting location icon 201.

[0134] Based on the structure involved in this variation, the shooting location icon 201 corresponding to the annotation information and the shooting location icon 201 corresponding to the additional text information can be distinguished and presented to the user.

[0135] (4) The terminal device 30 may also receive input from the user for filtering conditions used to filter the search scope during the above-mentioned search processing. For example, the terminal device 30 may receive filtering conditions at a time after the user receives the keywords and before sending the keywords to the information processing device 10. Furthermore, the terminal device 30 may send the filtering conditions to the information processing device 10 together with the keywords. In this modified example, the control information generated by the output unit 114 may also include an instruction to highlight only the first icon that meets the filtering conditions.

[0136] Figure 10 This is a diagram illustrating an example of the display screen G1 involved in this variation. For example... Figure 10As shown, the display screen G1 involved in this modified example includes a period input field R6. The user inputs information specifying a period into the period input field R6. The period specified by the user is an example of a filter condition. The period input by the user is acquired by the acquisition unit 111. The acquisition unit 111, having acquired the filter condition, first determines a first icon based on the keywords input by the user, as described in the previous embodiment. Here, it is assumed that the shooting location icons 201A, 201B, and 201C are determined as the first icon. Next, the acquisition unit 111 determines whether the determined first icon meets the filter condition. For example, it is assumed that the user inputs a filter condition such as "April 1, 2024 - May 1, 2024" into the period input field R6. In this case, the acquisition unit 111 confirms that the shooting date and time of the corresponding image information have been established with the determined first icon. Here, it is assumed that the shooting date and time corresponding to the image information established with shooting location icon 201A is April 1, 2024, and the shooting date and time corresponding to the image information established with shooting location icons 201B and 201C is March 1, 2024. In this case, the acquisition unit 111 determines that shooting location icon 201A is the first icon that meets the filtering criteria. The acquisition unit 111 inputs this determination result to the output unit 114. The output unit 114, having acquired the above determination result, sends control information containing an instruction to emphasize shooting location icon 201A only to the terminal device 30.

[0137] Alternatively, users can also input information about a given space at a specified building site (hereinafter referred to as space specification information) as a filter criterion. Figure 11 This is a diagram showing another example of the display screen G1 involved in this variation. Figure 11 The displayed screen G1 includes a space assignment button R7. When the user selects (e.g., clicks) the space assignment button R7, the terminal device 30 switches the application mode from the normal mode to the space assignment mode. In this state, the user inputs an operation to assign a given space on the building site. For example, the user inputs to move the mouse cursor from... Figure 11 A drag operation that moves point P1 toward point P2. In this case, Figure 11 The area AR1 shown is designated as a given space. Furthermore, the terminal device 30 acquires the location information of the user-specified space on the design drawing. Figure 11 In the example shown, the coordinates of the vertices (4 corners) of region AR1 are obtained.

[0138] In addition, buttons that are not shown, such as "Meeting Room A", "Meeting Room B", "Room A", and "Elevator Lobby", which correspond to spaces existing on the building site, can be displayed on the display screen G1. Users can select (click) any of these buttons to specify a given space on the building site.

[0139] The acquisition unit 111, upon receiving space-specific information, first determines a first icon based on the keywords entered by the user, as described in the previous embodiment. For example, suppose shooting location icons 201A, 201B, and 201C are determined as the first icon. Next, the acquisition unit 111 determines whether the determined first icon meets the filtering criteria. That is, the acquisition unit 111 determines whether the first icon is located within the space specified by the user based on the position information on the design drawing of each shooting location icon 201 and the position information on the design drawing of the space specified by the user. Figure 11 In the example shown, only location icon 201A among location icons 201A, 201B, and 201C is located within area AR1. In this case, the acquisition unit 111 determines that location icon 201A is the first icon that meets the filtering criteria. The acquisition unit 111 inputs this determination result to the output unit 114. The output unit 114, having acquired the above determination result, sends control information containing an instruction to highlight only location icon 201A to the terminal device 30.

[0140] In addition, the output unit 114 can also display buttons such as "Step 1", "Step 2", and "Step 3" (not shown) on the display screen G1. Furthermore, if the user selects the "Step 1" button, the output unit 114 can also highlight only the first icon captured during the first step on-site.

[0141] According to the structure involved in this variation, since only the first icon that meets the filtering criteria is highlighted, the location of the first icon that meets the user's expected criteria can be clearly identified.

[0142] Industrial availability

[0143] This disclosure is useful in remotely confirming the condition of a site.

Claims

1. An information processing method, specifically an information processing method in a computer. The information processing method includes: A bird's-eye view is displayed on the terminal device's screen; The bird's-eye view displays multiple location icons indicating the shooting locations; The text information is mapped to the icons of the multiple shooting locations. From the user's input of text information; A first icon was established from the plurality of shooting location icons that corresponds to the text information containing the text information; and The first icon is highlighted.

2. The information processing method according to claim 1, wherein, The image information was correlated with the icons of the multiple shooting locations. Emphasizing the first icon includes displaying image information corresponding to the first icon on the display screen, along with the emphasized first icon.

3. The information processing method according to claim 1 or 2, wherein, The first icon is highlighted to include: a second icon corresponding to the text information that does not contain the text information is not displayed in the bird's-eye view among the multiple shooting location icons.

4. The information processing method according to claim 2, wherein, At least one focus display on the display, which overlaps with the image information, establishes a correspondence with the image information. The text information corresponds to at least one of the displayed attention items. The information processing method further includes: when multiple attention displays correspond to the image information, displaying only the attention displays that correspond to the text information on the display.

5. The information processing method according to claim 2, wherein, At least one focus display on the display, which overlaps with the image information, establishes a correspondence with the image information. The text information corresponds to at least one of the displayed attention items. The information processing method further includes: when multiple attention displays correspond to the image information, displaying the attention displays that correspond to the text information in a manner different from the attention displays that do not correspond to the text information on the display.

6. The information processing method according to claim 4 or 5, wherein, The display of the multiple shooting location icons in the bird's-eye view includes: different display methods for the shooting location icons when the text information corresponds to the attention display and when the text information does not correspond to the attention display.

7. The information processing method according to claim 1 or 2, wherein, The information processing method further includes: inputting acceptance screening criteria. The first icon highlighting includes: highlighting only the first icon that meets the filtering criteria.

8. An information processing apparatus, comprising a processor, The processor performs the following processing: A bird's-eye view is displayed on the terminal device's screen; The bird's-eye view displays multiple location icons indicating the shooting locations; Establish a correspondence between the text information and the icons of the multiple shooting locations; From the user's input of text information; A first icon was established from the plurality of shooting location icons that corresponds to the text information containing the text information; and The first icon is highlighted.

9. An information processing program that causes a computer to perform the following processing: A bird's-eye view is displayed on the terminal device's screen; The bird's-eye view displays multiple location icons indicating the shooting locations; The text information is mapped to the icons of the multiple shooting locations. From the user's input of text information; A first icon was established from the plurality of shooting location icons that corresponds to the text information containing the text information; and The first icon is highlighted.

Images