Information processing method, information processing device, and information processing program

Abstract

When a selection of a first imaging site icon, which is one imaging site icon among a plurality of imaging site icons, is sensed, the information processing device displays a first image corresponding to the first imaging site icon on the display. The display of the first image on the display includes: determining whether or not the annotation information is associated with a second imaging site icon located within a predetermined range with respect to the first imaging site icon; and displaying the first image and the annotation information on a display when it is determined that the annotation information is associated with the second imaging site icon.

Description

Technical Field

[0001] The present disclosure relates to a technology for displaying a selected image. Background Art

[0002] Patent document 1 discloses that an image (e.g., a contour line) representing a designated area designated by a worker is displayed on an overhead image of a work site, and a chat associated with the designated area input by the worker is displayed in a display bar provided next to the overhead image.

[0003] However, the prior art shown in Patent Document 1 does not disclose that the image captured in the scene shown in the bird's-eye view image is displayed on the display according to the user's selection instruction. Therefore, the prior art does not disclose that annotation information is displayed in association with the first image displayed on the display according to the user's selection instruction. Therefore, the prior art cannot allow the user to confirm the annotation information set in the second image without inputting an operation to select the second image different from the first image during the browsing of the first image.

[0004] Prior Art Literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2021-86224 Summary of the invention

[0007] The present disclosure is made to solve such a problem, and an object of the present disclosure is to provide a technology that allows a user to confirm annotation information set in a second image without inputting an operation to select a second image different from the first image during browsing of the first image.

[0008] An information processing method in one aspect of the present disclosure is an information processing method in a computer, wherein a bird's-eye view image superimposed with a plurality of shooting location icons representing shooting locations is displayed on a display of a terminal device, and when a selection of a first shooting location icon that is one of the plurality of shooting location icons is sensed, a first image corresponding to the first shooting location icon is displayed on the display, and displaying the first image on the display includes: determining whether annotation information is associated with a second shooting location icon that is within a given range relative to the first shooting location icon; and when it is determined that the annotation information is associated with the second shooting location icon, displaying the first image and the annotation information on the display.

[0009] According to such a configuration, the user can confirm the annotation information set in the second image without inputting an operation to select the second image different from the first image during browsing of the first image. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a block diagram showing an example of the configuration of an information processing system in an embodiment of the present disclosure.

[0011] Figure 2 This is a diagram showing an example of a display screen displayed on the display of a terminal device.

[0012] Figure 3 FIG. 1 is a diagram showing an example of a menu screen for selecting a shooting date and time.

[0013] Figure 4 This is a diagram showing an example of a first image displayed on the display.

[0014] Figure 5 This is a diagram showing another example of the first image displayed on the display.

[0015] Figure 6 It is a diagram showing the detailed structure of the design image display section.

[0016] Figure 7 It is shown Figure 1 A flowchart of an example of processing of the information processing device is shown.

[0017] Figure 8 It is shown Figure 7 The flowchart shown shows the details of the processing.

[0018] Fig. 9 This is a diagram showing still another example of the first image displayed on the display.

[0019] Fig.10 This is a diagram showing another example of the display form of the first image.

[0020] Fig.11 This is a diagram showing a first image in which a second annotation area is displayed in association with an annotation icon. DETAILED DESCRIPTION

[0021] (Process of realizing one mode of the present disclosure)

[0022] Problems at the construction site include communication problems where specific instructions are not conveyed to workers and explanation of the instructions takes time, and problems with confirming the construction site where manpower is required to inspect the entire construction site and movement to the construction site takes time.

[0023] In order to solve such a problem, it is also considered to install multiple cameras at the construction site, and the remote site supervisor gives instructions to the workers while referring to the images obtained from the multiple cameras. However, at the construction site, as the construction progresses, there will be operations such as removing the installed sensors or installing the removed sensors in other places. Since such operations are time-consuming, it is not practical to install sensors at the construction site. For this reason, the inventors have studied a technology that can confirm the conditions of the construction site in detail from a remote location without installing sensors.

[0024] It is then recognized that when an operation is input to select a certain location in the design drawing of the construction site displayed on the display, if there is a user interface with an image of the construction site taken in advance at that location and the user can specify an area with respect to the image and enter annotations, the conditions of the construction site can be confirmed in detail remotely.

[0025] Furthermore, suppose that while viewing the first image thus selected and displayed, the user wishes to confirm the annotation set in the second image captured at a shooting location near the first image. In this case, the user is required to close the first image and input an operation to select the shooting location of the second image from the user interface. Such an operation takes time for the user.

[0026] To this end, the inventor obtained the understanding that, when an annotation is set in a second image captured near a first image, if the annotation is displayed together with the first image, the annotation set in the second image can be confirmed while browsing the first image without performing such an operation, thereby conceiving of the present disclosure.

[0027] (1) An information processing method in one aspect of the present disclosure is an information processing method in a computer, the method comprising: displaying an overhead image on a display of a terminal device on which a plurality of shooting location icons representing shooting locations are superimposed, and upon sensing selection of a first shooting location icon that is one of the plurality of shooting location icons, displaying a first image corresponding to the first shooting location icon on the display, wherein displaying the first image on the display includes: determining whether annotation information is associated with a second shooting location icon that is within a given range relative to the first shooting location icon; and upon determining that the annotation information is associated with the second shooting location icon, displaying the first image and the annotation information on the display.

[0028] According to this structure, when the annotation information is associated with the second shooting location icon located within a given range relative to the first shooting location icon corresponding to the first image, the first image is displayed in association with the annotation information. Therefore, even if an operation of closing the first image and displaying the second image is not input separately during the viewing of the first image, the annotation information set in the second image can be confirmed. In addition, since the annotation information can be confirmed without such an operation, the number of computer processing steps consumed for such an operation is reduced.

[0029] (2) In the information processing method described in (1) above, the annotation information may also include a position of an annotation area in which an annotation is set in a second image corresponding to the second shooting location icon, and the display of the first image on the display includes: based on the second image, sensing a corresponding area corresponding to the annotation area from the first image; and displaying the first image with the annotation area overlapped in the corresponding area on the display.

[0030] According to this configuration, since the annotation information set on the second image can be displayed at a position on the first image corresponding to the annotation information set on the second image, it is possible to easily check on which subject the annotation information set on the second image is set.

[0031] (3) In the information processing method described in (1) or (2) above, the display of the first image on the display may also include: when a correspondence is established between the first shooting location icon and the annotation information, further displaying the annotation information corresponding to the first shooting location icon and the annotation information corresponding to the second shooting location icon on the display.

[0032] According to this configuration, it is possible to simultaneously check the comment information associated with the first shooting location icon and the comment information associated with the second shooting location icon.

[0033] (4) In the information processing method described in any one of (1) to (3) above, the annotation information may also include: a position of a first annotation area, indicating an area in which annotations are set in the first image; and a position of a second annotation area, indicating an area in which annotations are set in the second image corresponding to the second shooting location icon, and displaying the first image on the display includes: based on the second image, sensing a corresponding area corresponding to the second annotation area from the first image; and when the first annotation area is set in the first image, displaying the first image on the display with the first annotation area overlapping and the second annotation area overlapping in the corresponding area.

[0034] According to such a configuration, the first annotation region and the second annotation region can be simultaneously confirmed in the first image.

[0035] (5) In the information processing method described in any one of (1) to (4) above, the display of the first image on the display may also include: using different display objects to display the annotation information associated with the first shooting location icon and the annotation information associated with the second shooting location icon.

[0036] According to this configuration, it is possible to easily distinguish between the comment information associated with the first shooting location icon and the comment information associated with the second shooting location icon.

[0037] (6) In the information processing method described in any one of (1) to (5) above, a selection icon configured to be movable may be further displayed in the bird's-eye view image, wherein the first shooting location icon is determined based on a distance between a movement end position of the selection icon and each of the plurality of shooting location icons.

[0038] According to such a configuration, since the shooting location icon can be selected by moving the selection icon on the bird's-eye view image, the shooting location icon can be selected intuitively.

[0039] (7) In the information processing method described in any one of (1) to (6) above, when a selection of the annotation information displayed on the display is detected, an annotation message represented by the annotation information may be displayed on the display.

[0040] According to this configuration, the comment message represented by the comment information can be confirmed as needed.

[0041] (8) In another embodiment of the present disclosure, an information processing device comprises a processor, which performs the following processing: displaying an overhead image on a display of a terminal device on which a plurality of shooting location icons representing shooting locations are superimposed, and when a selection of a first shooting location icon that is one of the plurality of shooting location icons is sensed, displaying a first image corresponding to the first shooting location icon on the display, wherein the display of the first image on the display includes: determining whether annotation information is set in a second shooting location icon that is located within a given range relative to the first shooting location icon; and when it is determined that the annotation information corresponds to the second shooting location icon, displaying the first image and the annotation information on the display.

[0042] According to such a configuration, it is possible to provide an information processing device that allows confirmation of annotation information set in a second image without inputting an operation to close the first image and display the second image during browsing of the first image.

[0043] (9) An information processing program according to still another aspect of the present disclosure causes a computer to execute the information processing method described in any one of (1) to (7) above.

[0044] According to such a configuration, it is possible to provide an information processing program that allows confirmation of annotation information set in a second image without inputting an operation for closing the first image and displaying the second image during browsing of the first image.

[0045] The present disclosure can also be realized as an information processing system that operates by such an information processing program. In addition, of course, such a computer program can be distributed via a computer-readable non-transitory recording medium such as a CD-ROM or a communication network such as the Internet.

[0046] In addition, the embodiments described below all represent a specific example of the present disclosure. The numerical values, shapes, structural elements, steps, the order of steps, etc. shown in the following embodiments are examples and are not intended to limit the present disclosure. In addition, among the structural elements in the following embodiments, the structural elements that are not recorded in the independent claims representing the highest concept are described as arbitrary structural elements. In addition, it is also possible to combine the various contents in all embodiments.

[0047] (Implementation Method)

[0048] Figure 1 1 is a block diagram showing an example of the structure of the information processing system 1 in the embodiment of the present disclosure. The information processing system 1 includes an information processing device 10, a shooting device 20, and a terminal device 30. The information processing device 10, the shooting device 20, and the terminal device 30 are communicatively connected via a network. An example of a network is the Internet. The information processing device 10 is, for example, a cloud server composed of one or more computers. However, this is an example, and the information processing device 10 may also be composed of an edge server or may be installed in the terminal device 30.

[0049] The photographing device 20 is, for example, composed of an omnidirectional camera, and photographs images at a given frame rate. The photographing device 20 is, for example, a portable photographing device held by a user. The user is, for example, a worker or a site supervisor at a construction site. The user moves within the construction site while photographing the construction site with the photographing device 20. The photographing device 20 sends image information representing the photographed image to the information processing device 10 via a network. The image information is associated with the position of the photographing location representing the photographing location and the photographing date and time. The position of the photographing location is obtained, for example, by a position sensor such as a magnetic sensor or a GPS sensor provided by the photographing device 20, and is expressed as latitude and longitude. The photographing date and time is obtained, for example, by a clock provided by the photographing device 20. Thus, the information processing device 10 can obtain image information at multiple photographing locations at the construction site. Here, the photographing device 20 photographs images at a given frame rate, so the photographing location is specified in units of frame periods. However, this is an example, and the photographing location can also be specified for each given time (for example, 1 second, 1 minute, etc.). The imaging device 20 includes an image sensor, an operating device, a communication circuit, a signal processing circuit, etc. The imaging device 20 may be constituted by a portable computer such as a smartphone or a tablet computer.

[0050] The terminal device 30 is held by the user. The terminal device 30 may be a portable computer such as a smartphone or a tablet computer, or may be a fixed computer. The terminal device 30 displays image information on a display according to the control of the information processing device 10. Figure 1 In the example of FIG. 1 , one terminal device 30 is shown, but a plurality of terminal devices may be connected to the information processing device 10 via a network. The terminal device 30 includes a central processing unit (CPU), a memory, an operating device such as a display, a touch panel, and a keyboard, and a communication circuit.

[0051] The information processing device 10 includes a processor 11, a memory 12, and a communication unit 13. The processor 11 is composed of, for example, a central processing unit (CPU). The processor 11 includes a display control unit 111. The display control unit 111 can be implemented by the processor 11 executing an information processing program, or can be composed of a dedicated hardware circuit such as an ASIC.

[0052] The display control unit 111 obtains an instruction from the terminal device 30 for selecting a design drawing by the user, reads out the design drawing information indicated by the instruction from the memory 12, and displays the read design drawing information on the display of the terminal device 30. The design drawing information is information of the design drawing representing the construction site. The design drawing information is an example of an overhead image. The display control unit 111 displays the shooting location icon for selecting any position of the design drawing in an overlapping manner on the design drawing. When the display control unit 111 obtains an instruction from the terminal device 30 for selecting the shooting date and time by the user, it displays multiple shooting location icons representing the shooting location under the shooting date and time in an overlapping manner on the design drawing. These displays are realized by the display control unit 111 sending the display instruction to the terminal device 30 using the communication unit 13. The selection icon is configured to be movable on the design drawing. The position that becomes a key point in the design drawing is previously associated with the latitude and longitude. The position that becomes a key point is, for example, the position of the four corners of the design drawing.

[0053] The display control unit 111 senses the selection of the first shooting location icon as one shooting location icon by acquiring an instruction to select one shooting location icon from among a plurality of shooting location icons from the terminal device 30. When the display control unit 111 senses the selection of the first shooting location icon, it displays the first image corresponding to the first shooting location icon on the display of the terminal device 30. The first image is an image captured at the shooting location represented by the first shooting location icon. The first image is an omnidirectional image. However, this is an example, and the first image may also be a panoramic image.

[0054] The display control unit 111 performs the following processing when displaying the first image. The display control unit 111 determines whether the annotation information is associated with the second shooting location icon located within a given range relative to the first shooting location icon. And, when it is determined that the annotation information is associated with the second shooting location icon, the display control unit 111 associates the first image with the annotation information and displays it on the display of the terminal device 30. The so-called second shooting location icon refers to a shooting location icon located within a certain distance range with the position (coordinates) of the first shooting location icon as the center. In the following description, for the sake of convenience, it is assumed that the second shooting location icon refers to two shooting location icons adjacent to the left and right of the first shooting location. However, this is an example, and the second shooting location icon may also be n (an integer greater than or equal to 2) shooting location icons located on the left side of the first shooting location icon and n shooting location icons located on the right side. As a certain distance range, for example, the closer the shooting location of the first image and the shooting location of the second image are, the closer the shooting location of the first image and the second image are to each other.

[0055] The annotation information includes the position on the image of the annotation area where the annotation is set. Hereinafter, the annotation area set in the first image is referred to as the first annotation area, and the annotation area set in the second image is referred to as the second annotation area. Based on the second image, the display control unit 111 senses a corresponding area corresponding to the second annotation area from the first image, and displays the first image with the second annotation area overlapped in the corresponding area on the display of the terminal device 30. In this case, the display control unit 111 sends a display instruction of the first image to the terminal device 30 via the communication unit 13. The first annotation area and the second annotation area are displayed using different display objects. For example, the display object of the first annotation area and the display object of the second annotation area are displayed by changing color, shape, etc.

[0056] The annotation area is an area set by the user by moving and deforming a rectangular frame displayed in an overlapping manner in the first image or the second image. The display control unit 111 can sense the corresponding area from the first image by applying pattern matching using the second annotation area as a template to the first image. Alternatively, first, the display control unit 111 uses an object identifier to identify the object included in the second annotation area, and the object identifier is pre-produced using machine learning such as deep learning. Next, the display control unit 111 uses the object identifier to identify the object included in the first image. Finally, the display control unit 111 can also sense an area on the first image that displays the same object as the object included in the second annotation area as the corresponding area.

[0057] When the first shooting location icon and the annotation information are associated with each other, the display control unit 111 may further display the annotation information associated with the first shooting location icon on the display of the terminal device 30 .

[0058] Specifically, when the first annotation region is set in the first image, the display control unit 111 displays the first image on which the first annotation region is further superimposed on the display of the terminal device 30 .

[0059] The memory 12 is composed of a non-volatile rewritable storage device such as a hard disk drive and a solid state drive. The memory 12 stores design information, shooting information, annotation information, and image information. The design information is image information representing the design. The design information is associated with a design ID for identifying the design. In the design, the latitude and longitude of the actual construction site are set at key points as described above.

[0060] The photographing information indicates information related to one photographing action using the photographing device 20. The photographing information is generated each time one photographing action is performed. One photographing action refers to a series of actions from the start of photographing by the worker holding the photographing device 20 to the end of photographing at the construction site. A plurality of images are photographed by one photographing action. The photographing information includes a design drawing ID, a photographing ID, a photographing date and time, a representative value of the photographing date and time, a position of a photographing location, and a position of a photographing location icon. The photographing ID is an identifier for identifying each photographing included in one photographing action. The photographing date and time is the photographing date and time of the photographing indicated by the photographing ID. The representative value of the photographing date and time is the photographing date and time when the photographing is started. The photographing location indicates the position (latitude and longitude) at which the photographing indicated by the photographing ID is performed. The position of the photographing location icon indicates the display position (coordinates) of the photographing location icon corresponding to the photographing ID on the design drawing. Based on the photographing position (latitude and longitude) at the key point of the design drawing indicated by the design drawing information and the photographing location (latitude and longitude) corresponding to the photographing ID, the photographing position is mapped onto the design drawing, thereby calculating the position of the photographing location icon.

[0061] The annotation information is information indicating an annotation. One annotation information corresponds to one annotation. The annotation information includes a photographing ID, the position of the annotation area, and an annotation message. The annotation message is text information input by the user as an annotation.

[0062] The image information indicates one image captured by each capture included in one capture operation. That is, the image information indicates the first image or the second image described above. The image information is associated with the capture ID.

[0063] In this way, since the shooting information and image information are associated with the shooting ID, the shooting ID is used as a key to determine the image corresponding to the shooting location icon. Since the shooting information and annotation information are associated with the shooting ID, the shooting ID is used as a key to determine the annotation information corresponding to the shooting location icon. The annotation information includes the shooting ID, the position of the annotation area, and the annotation message. Since the image information is associated with the shooting ID, the shooting ID is used as a key to determine the position of the annotation area set in the image and the annotation message.

[0064] The communication unit 13 is a communication circuit that connects the information processing device 10 to a network.

[0065] Figure 2 1 is a diagram showing an example of a display screen G1 displayed on the display of the terminal device 30. The display screen G1 is a basic screen of an application provided by the information processing device 10. The display screen G1 includes an image information display column R1, an annotation information display column R2, a design drawing display column R3, and a message display column R4.

[0066] The image information display field R1 displays a first image associated with a first shooting location icon.

[0067] The comment information display column R2 displays the comment information associated with the first shooting location icon. Here, for the first image displayed in the image information display column R1, a plurality of comment messages C1 input by users are displayed at a glance. In the comment information display column R2, the comment messages C1 input by users other than the user are displayed on the left, and the comment messages C1 input by the user are displayed on the right.

[0068] In the default display screen G1 immediately after the application is started, the shooting location icon has not been selected by the user. Therefore, in the default display screen G1, the image information display field R1 and the comment information display field R2 are empty.

[0069] The design drawing display area R3 displays the design drawing of the construction site. On the design drawing displayed in the design drawing display area R3, a selection icon 201, a shooting location icon 202, and a track 203 are superimposed and displayed.

[0070] The selection icon 201 is configured to be movable by a drag-and-drop operation. In this example, the selection icon 201 is configured by an image simulating a person.

[0071] The shooting location icon 202 is an icon indicating the shooting location, and corresponds to the image captured at the shooting location. In this example, the shooting location icon 202 is composed of a circular image. The trajectory 203 indicates the trajectory of the user who captured the image. In this example, the trajectory 203 is composed of a line connecting adjacent shooting location icons 202. The shooting location icon 202 located at the front end of the trajectory 203 and the shooting location icon 202 located at the rear end of the trajectory 203 are displayed in a larger size than the other shooting location icons. The shooting location icon 202 located at the front end (for example, the right end) of the trajectory 203 indicates the shooting start position, and the shooting location icon 202 located at the rear end (for example, the left end) of the trajectory 203 indicates the shooting end position.

[0072] When an operation (for example, tap or click) for selecting the image displayed in the image information display field R1 is input, the design display field R3 displays the image displayed in the image information display field R1 in the design display field R3 .

[0073] The message display section R4 displays various messages related to the construction site, that is, messages input by the user.

[0074] For example, when the selection icon 201 is placed in a given area of ​​any of the shooting location icons 202, the shooting location icon 202 is determined as the first shooting location icon, and the image corresponding to the first shooting location icon is sensed as the first image. And the first image is displayed in the image information display column R1. In this case, if the first shooting location icon is associated with annotation information, the annotation information corresponding to the first shooting location icon is displayed in the annotation information display column R2.

[0075] For example, it is assumed that the selection icon 201 is not placed in a given area in any of the shooting location icons 202. In this case, the shooting location icon that is the shortest distance from the placement position and that is associated with the annotation information is determined as the first shooting location icon, and the image corresponding to the first shooting location icon is sensed as the first image. Furthermore, the first image is displayed in the image information display column R1, and the annotation information corresponding to the first image is displayed in the annotation information display column R2.

[0076] Figure 3 1 is a diagram showing an example of a menu screen 300 for selecting a shooting date and time. When a given operation for displaying the menu screen 300 is performed on the display screen G1, the menu screen 300 is displayed. In the menu screen 300, a list of a plurality of shooting dates and times taken at a certain construction site is displayed. One shooting date and time included in the menu screen 300 represents a representative value of the shooting date and time of one shooting action.

[0077] When the user inputs an instruction to select a shooting date and time from the menu screen 300, the display control unit 111 obtains the instruction from the terminal device 30. Then, the display control unit 111 sends a display instruction to the terminal device 30 to overlap and display the trajectory 203 corresponding to the shooting date and time indicated by the obtained instruction in the design drawing display bar R3. Thus, the user can display the trajectory 203 corresponding to the selected shooting date and time in the design drawing display bar R3. In addition, in the display screen G1 that is first displayed after the application is started, the trajectory 203 corresponding to the latest shooting date and time is displayed in the design drawing display bar R3. Figure 3 In the figure, for the sake of convenience, the shooting location icon 202 is omitted, but in fact Figure 2 As shown, a shooting location icon 202 is also displayed.

[0078] Figure 4 FIG. 1 is a diagram showing an example of the first image G31 displayed on the display. Figure 4In the figure, the left side represents the first image G31, and the right side represents the second image G32. The quadrilateral frame in the first image G31 represents the first annotation area 301. The circular area in the first image G31 and the second image G32 represents the second annotation area 302. In the first image G31, in addition to the first annotation area 301 set in itself, the second annotation area 302 set in the second image G32 is also displayed. The second annotation area 302 displayed in the first image G31 is displayed in the corresponding area on the first image G31 of the second annotation area 302 included in the second image G32. Therefore, even if the user does not input an operation to display the second image G32 while browsing the first image G31, the second annotation area 302 can be confirmed on the first image G31. Figure 4 In the example of FIG. 1 , since the first annotation area and the second annotation area are set for the same subject, the first annotation area and the second display area are displayed overlappingly. In addition, since the first image G31 and the second image G32 are omnidirectional images, Figure 4 In the example of , the first image G31 displayed on the display is a region that is a part of the entire region of the first image.

[0079] Figure 5 FIG. 4 is a diagram showing another example of the first image G31 displayed on the display. Figure 5 In the example of , the expanded images obtained by expanding the entire area of ​​the omnidirectional image are used as the first image G31 and the second image G32. Figure 5 In the example of , the first image G31 and the second image G32 are distorted, but the entire area of ​​the first image G31 and the second image G32 is displayed on the display. Figure 5 In the example, Figure 4 Similarly, in the first image G31, in addition to the first annotation area 301 set in the first image G31 itself, the second annotation area 302 set in the second image G32 is also displayed. The display control unit 111 generates the expanded images of the first image G31 and the second image G32 using the equidistant azimuthal projection method or the fisheye projection method.

[0080] Figure 6 2 is a diagram showing the detailed structure of the design drawing display section R3. Figure 6 In the example of FIG. 1 , the shooting location icon 202 corresponding to the annotation information is associated with the annotation icon 204 and displayed. The annotation icon 204 uses a dialog box mark starting from the corresponding shooting location icon 202. The annotation icon 204 only indicates that the annotation information is associated, and the annotation message has not yet been displayed. Figure 6In the example, the shooting location icon 202 indicated by the arrow is determined to be the first shooting location icon 202_1. Therefore, the shooting location icon 202 adjacent to the left side of the first shooting location icon 202_1 and the shooting location icon 202 adjacent to the right side become the second shooting location icon 202_2. The second shooting location icon 202_2 on the left side of the two second shooting location icons 202_2 is associated with the annotation information. Therefore, the first image corresponding to the first shooting location icon 202_1 is displayed with the second annotation area corresponding to the second shooting location icon 202_2 on the left side overlapped. In addition, the first shooting location icon 202_1 is also associated with the annotation information. Therefore, the first image is also displayed with the first annotation area corresponding to the first shooting location icon 202_1 overlapped.

[0081] Figure 7 It is shown Figure 1 Flowchart of an example of processing of the information processing device 10 shown in FIG. 1 . The display control unit 111 obtains an instruction from a user who selects a design drawing (step S1). In this case, a menu screen for selecting a design drawing is displayed on the display of the terminal device 30, and an instruction to select one design drawing from the menu screen is input. The input instruction is sent to the information processing device 10 via the network and received by the communication unit 13. The display control unit 111 obtains the instruction via the communication unit 13. Since the instruction includes a design drawing ID, the display control unit 111 can obtain the design drawing information representing the instructed design drawing from the design drawing information stored in the memory 12.

[0082] Next, the display control unit 111 transmits a display instruction of the display screen G1 to the terminal device 30 via the communication unit 13, thereby displaying the display screen G1 on the display of the terminal device 30 (step S2). The display instruction of the display screen G1 displayed by default includes the design drawing information indicating the design drawing selected in step S1 and the shooting information corresponding to the latest shooting date and time. Therefore, in the display screen G1 displayed by default, as shown in FIG. Figure 2 As shown, the design drawing includes a selection icon 201, a shooting location icon 202 corresponding to the latest shooting date and time, and a track 203. Since the first shooting location icon has not been determined at this time point, the image information display column R1 and the annotation information display column R2 are blank. In addition, when an instruction to select the shooting date and time is input in step S3 described later, the display control unit 111 displays a display screen G1 including the design drawing in which the shooting location icon 202 corresponding to the selected shooting date and time and the track 203 are superimposed and displayed on the display.

[0083] Next, the display control unit 111 determines whether an instruction from the user to select the shooting date and time is obtained (step S3). In this case, a menu screen 300 for selecting the shooting date and time is displayed on the display of the terminal device 30. The user inputs an instruction to select one shooting date and time from the menu screen 300. The shooting date and time displayed on the menu screen 300 is a representative value of the shooting date and time included in the shooting information stored in the memory 12. When the shooting date and time is selected, one shooting action corresponding to the shooting date and time is selected. The input instruction is sent to the information processing device 10 via the network and received by the communication unit 13. The display control unit 111 obtains the instruction via the communication unit 13. Since the instruction includes the representative value of the shooting date and time, the display control unit 111 can determine one shooting information stored in the memory 12. When the instruction to select the shooting date and time is input (yes in step S3), the process proceeds to step S4. When the instruction to select the shooting date and time is not input (no in step S3), the process returns to step S2.

[0084] Next, the display process is performed (step S4). The details of the display process are as follows: Figure 8 The display process is a process of displaying the first image in the image information display area R1.

[0085] Next, the display control unit 111 determines whether an instruction to input an annotation is obtained (step S5). The instruction to input an annotation is an instruction input when the user intends to input an annotation for the image displayed in the image information display field R1. The instruction is input, for example, by selecting an annotation input instruction button (not shown) displayed on the display screen G1. The input instruction is sent to the information processing device 10 via the network and received by the communication unit 13. The display control unit 111 obtains the instruction via the communication unit 13.

[0086] Next, when an instruction to input annotations is obtained (yes in step S5), the display control unit 111 obtains the position of the annotation area (step S6). The position of the annotation area is input by, for example, performing an operation of moving and deforming a rectangular frame in the image information display bar R1. The position of the annotation area is, for example, the coordinates of a key point (vertex) of the frame. The input position of the annotation area is sent to the information processing device 10 via the network and received by the communication unit 13. The display control unit 111 obtains the position of the annotation area via the communication unit 13. The display control unit 111 generates annotation information including the obtained position of the annotation area, establishes a correspondence between the generated annotation information and the shooting ID, and stores it in the memory 12. Thus, as shown in FIG. Figure 3 As shown, the comment area D1 is set.

[0087] Next, when an instruction to input a comment is not acquired (NO in step S5 ), the process proceeds to step S8 .

[0088] Next, the display control unit 111 obtains the annotation message (step S7). Figure 3 As shown, the comment message is input into the comment information display field R2. The input comment message is sent to the information processing device 10 via the network and received by the communication unit 13. The display control unit 111 acquires the comment message via the communication unit 13. The display control unit 111 includes the acquired comment message in the comment information generated in step S6 and stores it in the memory 12.

[0089] Next, the display control unit 111 determines whether an end instruction is obtained (step S8). The so-called end instruction is an instruction to close the display screen G1 displayed in step S3. The instruction is input by pressing an end button (not shown) displayed on the display screen G1. When the end instruction is obtained (yes in step S8), the process ends. When the end instruction is not obtained (no in step S8), the process returns to step S3. In this case, the display of the display screen G1 is maintained. The end instruction is sent to the information processing device 10 via the network and received by the communication unit 13. The display control unit 111 obtains the end instruction via the communication unit 13.

[0090] Figure 8 It is shown Figure 7 The display control unit 111 obtains an instruction to select the first image (step S21). Figure 2 As shown, the instruction is performed by dragging and dropping the selection icon 201 on the design drawing. As described above, the display control unit 111 determines one shooting location icon selected by the user based on the drop position of the selection icon 201 and the position of the shooting location icon 202 as the first shooting location icon, and senses the image corresponding to the first shooting location icon as the first image.

[0091] Next, the display control unit 111 determines whether the annotation information is associated with the second shooting location icon (step S22). If the annotation information is associated with the second shooting location icon (yes in step S22), the display control unit 111 detects the corresponding area of ​​the second annotation area set in the second image from the first image by the above-mentioned pattern matching (step S23). If the annotation information is not associated with the second shooting location icon (no in step S22), the process proceeds to step S25.

[0092] Next, in step S24 , the display control unit 111 superimposes the second annotation area on the corresponding area (step S24 ).

[0093] Next, the display control unit 111 determines whether the annotation information is associated with the first shooting location icon (step S25). If the annotation information is associated with the first shooting location icon (yes in step S25), the display control unit 111 overlaps the first annotation area with the first image (step S26). On the other hand, if the annotation information is not associated with the first shooting location icon (no in step S25), the process proceeds to step S27.

[0094] Next, the display control unit 111 displays the first image with the first and second annotation regions superimposed on the display of the terminal device 30 (step S27 ). If step S25 is No, the second annotation region is superimposed on the first image without superimposing the first annotation region.

[0095] Fig. 9 FIG. 4 is a diagram showing another example of the first image G31 displayed on the display. Fig. 9 In the example of , an expanded image in which the entire area of ​​the omnidirectional image is expanded is used as the first image G31. Fig. 9 In the example of FIG. 3 , the first annotation area 301 and the second annotation area 302 are set for different subjects. Fig. 9 In the example, Figure 5 Unlike the example of , the first annotation area 301 and the second annotation area 302 are displayed separately.

[0096] Thus, according to the present embodiment, when annotation information is associated with a second shooting location icon located within a given range relative to a first shooting location icon corresponding to a first image, the first image is displayed in association with the annotation information. Therefore, even when the first image is viewed without inputting an operation to close the first image and display the second image, the annotation information set in the second image can be confirmed. In addition, since the annotation information can be confirmed without such an operation, the number of computer processing steps required for such an operation is reduced.

[0097] The present disclosure can adopt the following modified examples.

[0098] (1) Figure 6In the case where annotation information is also set in the second shooting location icon 202_2 on the right side of the first shooting location icon 202_1, the display control unit 111 may further display the annotation information in the first image. In this case, in the first image, in addition to the second annotation area set in the second image corresponding to the second shooting location icon 202_2 on the left side, the second annotation area set in the second image corresponding to the second shooting location icon 202_2 on the right side is also displayed overlapping on the first image. In the case where the two second annotation areas are not accommodated in the display area of ​​the first image, the display control unit 111 may also reduce the first image so that the two annotation areas are accommodated in the display area at the same time. Alternatively, the display control unit 111 may display an expanded image of the first image in order to display the entire area of ​​the first image.

[0099] (2) When the first annotation area and the second annotation area are not contained within the display area of ​​the first image, the display control unit 111 may reduce the size of the first image so that the first annotation area and the second annotation area are contained within the display area of ​​the first image. Alternatively, the display control unit 111 may display an expanded image of the first image in order to display the entire area of ​​the first image.

[0100] (3) When a plurality of annotation areas are set in the second image, the display control unit 111 may sense corresponding areas of the plurality of annotation areas set in the second image from the first image by pattern matching, and display the first image with the plurality of annotation areas set in the second image superimposed on the sensed plurality of corresponding areas.

[0101] (4) The display control unit 111 may not overlap the second annotation area on the first image. For example, the display control unit 111 may display a message indicating that annotation information is set in the second image together with the first image. In this case, the display control unit 111 may display an annotation icon with a dialog box mark in the outer frame of the first image. In this case, the user may not set the second annotation area in the second image.

[0102] (5) The display control unit 111 may not overlap the first annotation area on the first image. For example, the display control unit 111 may display a message indicating that annotation information is set in the first image together with the first image. In this case, the display control unit 111 may display an annotation icon with a dialog box mark in the outer frame of the first image. Furthermore, in the case of adopting this method, the user may not set the first annotation area in the first image.

[0103] (6) When the annotation message corresponding to the first image is displayed in the annotation information display field R2, if an instruction to select the second annotation area is input (for example, tapping or clicking), the display control unit 111 may display the annotation message corresponding to the second annotation area in the annotation information display field R2. In addition, in this state, if an instruction to select the first annotation area is input, the display control unit 111 may display the annotation message corresponding to the first annotation area in the annotation information display field R2.

[0104] (7) In the above embodiments, a construction site is exemplified as the site, but the present disclosure is not limited thereto, and a manufacturing site, a logistics site, a distribution site, a farmland, a construction site, a retail site, an office, a hospital, a commercial facility, a nursing facility, etc. may also be used as the site.

[0105] (8) Figure 4 In the example of the first image G31, the first image G31 is displayed overlapping the second annotation area 302, but the present disclosure is not limited to this. For example, the display control unit 111 may replace the display of the second annotation area 302 in the first image G31 with Fig.10 As shown, an annotation icon 303 indicating that an annotation comment is attached to the second image G32 is displayed in the first image G31. Fig.10 FIG. 4 is a diagram showing another example of the display method of the first image G31. Fig.10 In FIG. 3 , the first image G31 is shown on the left side, and the second image G32 is shown on the right side. A second comment area 302 is set in the second image G32 adjacent to the first image G31. Furthermore, the second comment area 302 is accompanied by a comment message.

[0106] In this case, the first image G31 displays an annotation icon 303 indicating that an annotation message is attached to the second image G32 in association with the first annotation area 301. This allows a user viewing the first image G31 to confirm at a glance that an annotation message is set in an adjacent image.

[0107] (9) When the second annotation area 302 is superimposed on the first image G31 , the annotation icon 304 may be displayed in association with the second annotation area 302 . Fig.11 3 is a diagram showing a first image G31 in which a second annotation area 302 and an annotation icon 303 are displayed in association with each other. Fig.11 In the figure, the first annotation area 301 is omitted. The display control unit 111 is as follows Fig.11 As shown on the left side of Fig.11As shown on the right side of the image G31, the comment message display box 305 is displayed overlapping the first image G31. The comment message display box 305 displays the comment messages attached to the second comment area 302 at a glance. Figure 2 The annotation messages of the second annotation area 302 are listed in the annotation information display field R2 shown in FIG. 1. Thus, the user can check the annotation messages attached to the second annotation area 302 by simply selecting the annotation icon 304 in the first image G31 without displaying the second image G32.

[0108] Industrial Applicability

[0109] The present disclosure enables remote confirmation of the status of a construction site and is therefore useful in managing the construction site.

Claims

1. An information processing method is an information processing method in a computer. A bird's-eye view image on which a plurality of shooting location icons indicating shooting locations are superimposed is displayed on a display of the terminal device. When a selection of a first shooting location icon, which is one of the plurality of shooting location icons, is sensed, a first image corresponding to the first shooting location icon is displayed on the display, The displaying of the first image on the display includes: determining whether the annotation information is associated with a second shooting location icon located within a given range relative to the first shooting location icon; and When it is determined that the comment information is associated with the second shooting location icon, the first image and the comment information are displayed on the display.

2. The information processing method according to claim 1, wherein: The annotation information includes a position of an annotation area in which an annotation is set in the second image corresponding to the second shooting location icon. The displaying of the first image on the display includes: Based on the second image, sensing a corresponding area corresponding to the annotation area from the first image; and The first image in which the annotation area is superimposed on the corresponding area is displayed on the display.

3. The information processing method according to claim 1, wherein: The displaying of the first image on the display includes: When the comment information is associated with the first shooting location icon, the comment information associated with the first shooting location icon and the comment information associated with the second shooting location icon are further displayed on the display.

4. The information processing method according to claim 1, wherein: The annotation information includes: a position of a first annotation area indicating an area in the first image where annotations are set; and a position of a second annotation area indicating an area in the second image corresponding to the second shooting location icon where annotations are set. The displaying of the first image on the display includes: Based on the second image, sensing a corresponding area corresponding to the second annotation area from the first image; and When the first annotation region is set in the first image, the first image on which the first annotation region is superimposed and the second annotation region is superimposed in the corresponding region is displayed on the display.

5. The information processing method according to claim 3 or 4, wherein: The displaying of the first image on the display includes: The comment information associated with the first shooting location icon and the comment information associated with the second shooting location icon are displayed using different display objects.

6. The information processing method according to claim 1, wherein: Furthermore, a selection icon configured to be movable is displayed in the bird's-eye view image, The first shooting location icon is determined based on the distance between the movement end position of the selection icon and each of the plurality of shooting location icons.

7. The information processing method according to claim 1, wherein: Further, in case that the selection of the annotation information displayed on the display is sensed, the annotation message represented by the annotation information is displayed on the display.

8. An information processing device comprising a processor, The processor performs the following processing: A bird's-eye view image on which a plurality of shooting location icons indicating shooting locations are superimposed is displayed on a display of the terminal device. When a selection of a first shooting location icon, which is one of the plurality of shooting location icons, is sensed, a first image corresponding to the first shooting location icon is displayed on the display, The displaying of the first image on the display includes: determining whether annotation information is set in a second shooting location icon located within a given range with respect to the first shooting location icon; and When it is determined that the comment information is associated with the second shooting location icon, the first image and the comment information are displayed on the display.

9. An information processing program causing a computer to execute the information processing method according to claim 1 or 2.

Images