Information processing method, information processing device, and information processing program

Abstract

This information processing method includes: displaying a bird's-eye view on the display of an information terminal; displaying, in the bird's-eye view, a plurality of first icons indicating a plurality of imaging points in accordance with an imaging time; acquiring recorded content information; and displaying, in the bird's-eye view, information indicating that the content information has been associated with a first icon of the plurality of first icons whose imaging time is close to the recording time of the content information.

Description

Information processing method, information processing device, and information processing program

[0001] The present disclosure relates to techniques for displaying images.

[0002] For example, the technology of Patent Document 1 discloses that a mobile terminal combines photographic identification information, which combines the photographic location, photographic direction, and time information, with a photographic image of a construction site, and transmits the photographic image with the combined photographic identification information to a server.The server then generates icons indicating the photographic direction and photographic location, saves them for each time information, and displays all of the generated icons on a map of the area where the photograph was taken or on a digital aerial photograph image.

[0003] However, in the above-mentioned conventional technology, an icon representing an image captured by a mobile terminal is displayed on a map at a position corresponding to the location where the image was captured, but an icon representing additionally recorded content information that is different from the captured image is not displayed on the map, and further improvement is needed. Also, there is a risk that accessing the content information increases the number of processing steps of the computer, which increases the burden on the computer.

[0004] Japanese Patent Application Laid-Open No. 2006-331260

[0005] The present disclosure has been made to solve the above problems, and aims to provide a technology that can notify a user that there is recorded content information other than images taken at the shooting location, can reduce the processing steps required to access the content information, can reduce the computer processing steps required to access the content information, and can reduce the burden on the computer.

[0006] An information processing method according to one aspect of the present disclosure is an information processing method executed by a computer, and includes displaying a bird's-eye view on a display of an information terminal, displaying a plurality of first icons indicating a plurality of shooting locations on the bird's-eye view according to the shooting time, acquiring recorded content information, and displaying information on the bird's-eye view indicating that the content information is associated with a first icon, among the plurality of first icons, whose shooting time is close to the recording time of the content information.

[0007] According to the present disclosure, it is possible to notify a user that there is recorded content information other than images captured at a shooting location, thereby reducing the processing steps required to access the content information. Furthermore, it is possible to reduce the processing steps required by a computer to access the content information, thereby reducing the burden on the computer.

[0008] FIG. 1 is a diagram showing an overall configuration of an information processing system in the first embodiment. FIG. 2 is a first flowchart showing an example of processing of a second information terminal in the first embodiment. FIG. 3 is a second flowchart showing an example of processing of the second information terminal in the first embodiment. FIG. 4 is a diagram showing an example of a display screen displayed on the second information terminal when a second image is captured in the first embodiment. FIG. 5 is a first flowchart showing an example of processing of a server in the first embodiment. FIG. 6 is a second flowchart showing an example of processing of the server in the first embodiment. FIG. 7 is a diagram showing an example of a design drawing screen displayed on the first information terminal in the first embodiment. FIG. 8 is a diagram showing an overall configuration of an information processing system in the second embodiment. FIG. 9 is a first flowchart showing an example of processing of a second information terminal in the second embodiment. FIG. 10 is a second flowchart showing an example of processing of the second information terminal in the second embodiment. FIG. 11 is a diagram showing an example of a display screen displayed on the second information terminal when recording starts in the second embodiment. FIG. 12 is a diagram showing an example of a display screen displayed on the second information terminal when recording ends in the second embodiment. FIG. 13 is a first flowchart showing an example of processing of a server in the second embodiment. FIG. 1 is a diagram showing the overall configuration of an information processing system in Embodiment 3. FIG. 2 is a first flowchart showing an example of processing by a second information terminal in Embodiment 3. FIG. 3 is a second flowchart showing an example of processing by a second information terminal in Embodiment 3. FIG. 4 is a diagram showing the overall configuration of an information processing system in Embodiment 4. FIG. 5 is a first flowchart showing an example of processing by a second information terminal in Embodiment 4. FIG. 6 is a second flowchart showing an example of processing by a server in Embodiment 4. FIG. 7 is a diagram showing the overall configuration of an information processing system in Embodiment 4. FIG. 8 is a first flowchart showing an example of processing by a second information terminal in Embodiment 4. FIG. 9 is a second flowchart showing an example of processing by a server in Embodiment 4.Fig. 10 is a diagram showing an example of a design drawing screen displayed on the first information terminal in the present embodiment 4. Fig. 11 is a first flowchart showing an example of a process of the server in the present embodiment 5. Fig. 12 is a second flowchart showing an example of a process of the server in the present embodiment 5. Fig. 13 is a diagram showing an example of a design drawing screen displayed on the first information terminal in the present embodiment 5.

[0009] (Findings that form the basis of the present disclosure) A user interface is being developed that displays multiple photography location icons indicating locations where actual photographs of a construction site were taken on a two-dimensional blueprint of the construction site, etc., and when one photography location icon is selected, displays an image taken at the photography location indicated by that icon. This allows a construction site manager to understand the situation at the construction site without visiting the construction site.

[0010] In addition, at a construction site, the construction site may be photographed by a first image capturing device, and a portion of the construction site may be photographed by a second image capturing device different from the first image capturing device. In this case, the first image captured by the first image capturing device and the second image captured by the second image capturing device are managed separately. Therefore, the user interface may require an increased number of processing steps for the user to access the second image.

[0011] In the technology of Patent Document 1, an icon representing an image taken by a mobile terminal is displayed on a map at a position corresponding to the location where the image was taken, but an icon representing additionally recorded content information that is different from the image that was taken is not displayed on the map.

[0012] Therefore, it is not possible to notify the user that there is content information recorded other than the image captured at the shooting location, making it difficult to reduce the processing steps required to access the content information.Furthermore, there is a risk that the processing steps required to access the content information will increase, which may increase the burden on the computer.

[0013] In order to solve the above problems, the following techniques are disclosed.

[0014] (1) An information processing method according to one aspect of the present disclosure is an information processing method executed by a computer, and includes displaying a bird's-eye view on a display of an information terminal, displaying a plurality of first icons indicating a plurality of shooting locations on the bird's-eye view according to the shooting times, acquiring recorded content information, and displaying information on the bird's-eye view indicating that the content information is associated with a first icon, among the plurality of first icons, whose shooting time is close to the recording time of the content information.

[0015] According to this configuration, a plurality of first icons indicating a plurality of shooting locations are displayed on the bird's-eye view according to the shooting time, and information indicating that the content information is associated with a first icon among the plurality of first icons whose shooting time is close to the recording time of the content information is displayed on the bird's-eye view.

[0016] Therefore, the user can be notified that there is recorded content information other than the image taken at the shooting location, and the processing steps for accessing the content information can be reduced. Also, because the user is notified that there is recorded content information other than the image taken at the shooting location, the processing steps of the computer required to access the content information can be reduced, and the burden on the computer can be reduced.

[0017] (2) In the information processing method described in (1) above, the information may be a second icon that is different from the plurality of first icons.

[0018] According to this configuration, the second icon can notify the user that there is recorded content information in addition to the image captured at the shooting location.

[0019] (3) In the information processing method described in (1) above, the information may be the first icon that has a display mode different from that of other first icons.

[0020] According to this configuration, the first icon, which has a display mode different from other first icons, can notify the user that there is content information recorded in addition to images taken at the shooting location.

[0021] (4) In the information processing method described in any one of (1) to (3) above, the plurality of first icons may be associated with a plurality of first images taken at the plurality of shooting locations by a first photographing device, and the content information may include a second image taken by a second photographing device while the plurality of first images are being taken by the first photographing device.

[0022] According to this configuration, a second image captured by a second image capturing device is acquired while a plurality of first images are being captured by a first image capturing device, and information indicating that the second image is associated with a first icon among the plurality of first icons whose capture time is close to the recording time of the second image is displayed on the bird's-eye view. Thus, it is possible to notify the user that there is a second image that was captured near the capture location of the first image.

[0023] (5) In the information processing method described in any one of (1) to (3) above, the plurality of first icons may be associated with a plurality of images taken at the plurality of shooting locations by a photographing device, and the content information may include audio information recorded by a recording device while the plurality of images were being taken by the photographing device.

[0024] According to this configuration, audio information recorded by the audio recording device is acquired while the multiple images are being captured by the imaging device, and information indicating that the audio information is associated with a first icon among the multiple first icons whose imaging time is close to the recording time of the audio information is displayed on the bird's-eye view. Therefore, it is possible to notify the user that audio information is recorded near the location where the first images were captured.

[0025] (6) In the information processing method described in any one of (1) to (3) above, the plurality of first icons may be associated with a plurality of images taken at the plurality of shooting locations by a photographing device, and the content information may include text information input by an input device while the plurality of images are being taken by the photographing device.

[0026] According to this configuration, text information input by the input device while the image capture device is capturing multiple images is acquired, and information indicating that the text information is associated with a first icon among the multiple first icons whose capture time is close to the recording time of the text information is displayed on the bird's-eye view. Therefore, it is possible to notify the user that there is text information recorded near the capture location of the first image.

[0027] (7) In the information processing method described in any one of (1) to (3) above, the plurality of first icons may be associated with a plurality of images taken at the plurality of shooting locations by a photographing device, and the content information may include text information converted from audio information recorded by a recording device while the plurality of images were being taken by the photographing device.

[0028] According to this configuration, text information is obtained by converting audio information recorded by the recording device while the multiple images are being captured by the imaging device, and information indicating that the text information is associated with a first icon among the multiple first icons whose capture time is close to the recording time of the text information is displayed on the bird's-eye view. Therefore, it is possible to notify the user that there is text information recorded near the location where the first images were captured.

[0029] (8) In the information processing method described in (2) above, displaying the information may include associating the second icon with the first icon by a frame line of a predetermined shape.

[0030] According to this configuration, the second icon and the first icon are associated with each other by a frame line of a predetermined shape, so that the user can easily recognize which first icon the text information is associated with.

[0031] (9) In the information processing method described in (8) above, the plurality of first icons may be associated with the shooting direction of the camera that captured the images at the plurality of shooting locations, and a point on the outer edge of the first icon identified by the shooting direction associated with the first icon may be connected to the frame line.

[0032] According to this configuration, the user can know the shooting direction of the camera that captured the image from the connection point between the outer edge of the first icon and the frame line.

[0033] (10) In the information processing method described in (2) above, displaying the information may include displaying the second icon in a location that does not overlap with characters or graphics in the bird's-eye view.

[0034] According to this configuration, the second icon is displayed in a location that does not overlap with the text or graphics in the bird's-eye view, thereby improving the visibility of the bird's-eye view and the second icon.

[0035] (11) In the information processing method described in (10) above, the bird's-eye view may include a blueprint of a building, and the display of the information may include displaying the second icon in a margin within the building on the blueprint, in a margin outside the building on the blueprint, or in a portion outside an area on the blueprint that includes the plurality of first icons.

[0036] According to this configuration, the second icon is displayed in the margins within the building in the blueprint, the margins outside the building in the blueprint, or in the area outside the area containing multiple first icons in the blueprint, thereby improving the visibility of the blueprint and the second icon.

[0037] (12) In the information processing method described in (5) above, the method may further include determining whether the audio information includes speech from multiple people and converting the audio information into text information, and the display of the information may further include displaying the text information identifiable for each of the multiple people.

[0038] According to this configuration, identifiable text information is further displayed for each of the plurality of people, so that the content of what was said by the plurality of people at the location where the image was taken can be confirmed from the text information.

[0039] Furthermore, the present disclosure can be realized not only as an information processing method that executes the characteristic processes described above, but also as an information processing device having a characteristic configuration corresponding to the characteristic processes executed by the information processing method. Furthermore, the present disclosure can also be realized as a computer program that causes a computer to execute the characteristic processes included in such an information processing method. Therefore, the same effects as those of the above information processing method can also be achieved in the following other aspects.

[0040] (13) Another aspect of the present disclosure is an information processing device including a processor, which displays a bird's-eye view on a display of an information terminal, displays a plurality of first icons indicating a plurality of shooting locations on the bird's-eye view according to the shooting time, acquires recorded content information, and displays information on the bird's-eye view indicating that the content information has been associated with a first icon among the plurality of first icons whose shooting time is close to the recording time of the content information.

[0041] (14) An information processing program according to another aspect of the present disclosure causes a computer to function in the following manner: display a bird's-eye view on a display of an information terminal; display a plurality of first icons indicating a plurality of shooting locations on the bird's-eye view according to the shooting time; acquire recorded content information; and display information indicating that the content information is associated with a first icon among the plurality of first icons whose shooting time is close to the recording time of the content information on the bird's-eye view.

[0042] (15) A non-transitory computer-readable recording medium according to another aspect of the present disclosure records the information processing program described in (14) above.

[0043] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept are described as optional components. Furthermore, in all embodiments, the respective contents can be combined.

[0044] First Embodiment FIG. 1 is a diagram showing the overall configuration of an information processing system 1 according to a first embodiment.

[0045] The information processing system 1 displays a bird's-eye view, displays a plurality of first icons representing a plurality of shooting locations on the bird's-eye view according to shooting times, and displays information on the bird's-eye view indicating that content information is associated with a first icon, among the plurality of first icons, whose shooting time is close to the recording time of the content information. The information processing system 1 includes a server 10, a first information terminal 20, an imaging device 30, and a second information terminal 40.

[0046] The server 10 is an example of an information processing device and a computer. The server 10, the first information terminal 20, and the second information terminal 40 are connected to each other so as to be able to communicate with each other via a network NT. An example of the network NT is the Internet. The server 10 is, for example, a cloud server configured with one or more computers. However, this is just one example, and the server 10 may be configured as an edge server or may be implemented in the first information terminal 20. An aspect in which the server 10 is implemented in the first information terminal 20 is an example of an aspect in which the first information terminal 20 is configured as an information processing device.

[0047] The first information terminal 20 is carried by a user. The user is, for example, a manager of a predetermined space to be photographed by the imaging device 30. The predetermined space is, for example, a construction site. However, this is just one example, and the predetermined space may be a construction site, factory, store, office, etc. The first information terminal 20 may be configured as a portable computer such as a smartphone or tablet computer, or as a stationary computer. The first information terminal 20 displays images on a display 23 under control of the server 10. Although one first information terminal 20 is illustrated in the example of FIG. 1, multiple first information terminals may be connected to the server 10 via a network NT. The first information terminal 20 includes a communication unit 21, a processor 22, a display 23, and an operation unit 24.

[0048] The communication unit 21 is a communication interface that connects the first information terminal 20 to the network NT. The communication unit 21 transmits instruction signals indicating various instructions received from the user by the operation unit 24 to the server 10. The communication unit 21 receives display data for displaying various display screens from the server 10.

[0049] The processor 22 is configured by, for example, a central processing unit (CPU), and displays a display screen indicated by the display data received by the communication unit 21 on the display 23 .

[0050] The display 23 is configured with various display devices such as a liquid crystal display or an organic EL (Electro-Luminescence) display, and displays various display screens under the control of the processor 22.

[0051] The operation unit 24 is composed of, for example, a keyboard, a touch panel, a mouse, and the like, and accepts various instructions input by the user.

[0052] The second information terminal 40 is configured as a portable information terminal such as a smartphone or a tablet computer, and connects the photographing device 30 to the network NT. The second information terminal 40 and the photographing device 30 are connected to each other so that they can communicate with each other via a short-range wireless communication path such as Bluetooth (registered trademark). The photographing device 30 is an example of a first photographing device, and the second information terminal 40 is an example of a second photographing device.

[0053] The image capturing device 30 is, for example, an omnidirectional camera, and captures a first image at a predetermined frame rate. An omnidirectional camera is also called a 360-degree camera, and is a camera that can capture images in all directions of 360 degrees. The image capturing device 30 is, for example, a portable image capturing device carried by a photographer. The photographer is, for example, a worker or a site supervisor at a construction site. The image capturing device 30 may be held in the photographer's hand, or may be worn on the photographer's body (for example, on the head). The image capturing device 30 may also be a regular camera.

[0054] The photographer moves through the construction site while photographing the construction site with the camera device 30. At the start point of photography, the photographer points the camera device 30 in a northerly direction and presses the camera button on the camera device 30 to start photographing, and when the photographer reaches the end point of photography, presses the camera button again to stop photographing. When photography ends, the camera device 30 transmits the series of first images that have been photographed to the second information terminal 40.

[0055] First, the photographing device 30 displays a photographing button for instructing the photographer to start or stop photographing, and accepts the photographer's pressing of the photographing button. Next, the photographing device 30 determines whether the photographing button for instructing the photographer to start photographing has been pressed. If the photographing button has been pressed, the photographing device 30 starts photographing a first image and stores the photographed first image in its internal memory. Next, the photographing device 30 determines whether the photographing button for instructing the photographer to stop photographing has been pressed. If the photographing button has been pressed, the photographing device 30 stops storing the first image and transmits the multiple first images stored in its internal memory to the second information terminal 40.

[0056] The first image includes a shooting date and time. The shooting date and time is obtained, for example, by a clock provided in the image capture device 30. In this example, the image capture device 30 captures images at a predetermined frame rate (e.g., 10 frames per second), so the shooting location is defined in frame cycle units. However, this would result in an enormous amount of data, so the shooting location may be defined every predetermined time period (e.g., 1 second, 5 seconds, or 10 seconds).

[0057] The second information terminal 40 receives the plurality of first images transmitted by the image capturing device 30. The second information terminal 40 also displays a blueprint of the construction site and accepts designation by the photographer of the position of the image capturing start point and the position of the image capturing end point on the blueprint.

[0058] The start and end points of photography are identified by the photographer inputting instructions specifying the positions on the blueprint screen of the site displayed on the display of the second information terminal 40. Two-dimensional coordinate axes are defined on the blueprint screen. Therefore, the start and end points of photography are each defined by two-dimensional coordinate values. The start and end points of photography are used by the server 10 to identify the position of each photography point and the photography direction of each image.

[0059] The second information terminal 40 transmits first photographing information including the captured multiple first images to the server 10 via the network NT. The first photographing information is generated each time a single photographing operation is performed. One photographing operation refers to a series of operations from when a worker holding the photographing device 30 starts photographing at a construction site until when the photographing ends. Multiple first images are photographed by one photographing operation. The first photographing information includes the captured multiple first images and meta information for each of the multiple first images. The meta information includes a photographing ID, a photographing date and time, and a blueprint ID. The photographing ID is an identifier for identifying the photographing operation. The photographing date and time is the photographing date and time of the first image. The blueprint ID is an identifier for identifying a blueprint of a specified space corresponding to the image.

[0060] The second information terminal 40 also includes a camera that captures the second image. The photographer moves around the construction site while photographing the construction site with the photographing device 30. At this time, the photographer individually photographs desired locations within the construction site with the camera of the second information terminal 40. The processor of the second information terminal 40 records the second image captured by the camera in the memory of the second information terminal 40. The second image is an example of content information.

[0061] The second information terminal 40 transmits second shooting information including a second image captured by the second information terminal 40 while the multiple first images are being captured by the imaging device 30 to the server 10 via the network NT. The second shooting information includes at least one captured second image and meta information for each of the at least one second image. The meta information includes a shooting ID and a shooting date and time. The shooting ID is an identifier for identifying the shooting operation. The shooting ID is the same as the shooting ID associated with the multiple first images. The shooting date and time is the shooting date and time of the second image, and is obtained, for example, by a clock provided in the second information terminal 40.

[0062] In the first embodiment, the camera included in the second information terminal 40 captures the second image, but the present disclosure is not particularly limited to this, and the second image may be captured by a different imaging device than the imaging device 30. The second information terminal 40 may be communicably connected to the different imaging device, receive the second image transmitted by the different imaging device, and transmit second imaging information including the received second image to the server 10.

[0063] The server 10 includes a processor 11, a memory 12, and a communication unit 13. The processor 11 is configured, for example, by a central processing unit (CPU). The processor 11 includes an acquisition unit 111 and a display control unit 112. The acquisition unit 111 and the display control unit 112 may be realized by the processor 11 executing an information processing program, or may be configured by a dedicated hardware circuit such as an ASIC. The information processing program may be recorded on a non-transitory computer-readable recording medium.

[0064] The communication unit 13 is a communication interface that connects the server 10 to the network NT. The communication unit 13 receives first photographing information transmitted by the second information terminal 40. The communication unit 13 receives second photographing information transmitted by the second information terminal 40. As a result, the communication unit 13 acquires content information recorded by the second information terminal 40, i.e., second images photographed by the second information terminal 40. The communication unit 13 receives instruction signals from the first information terminal 20 that indicate various instructions accepted from the user. The communication unit 13 transmits display data for displaying various display screens to the first information terminal 20.

[0065] The memory 12 is configured as a non-volatile rewritable storage device such as a hard disk drive or a solid state drive, and includes a design drawing information storage unit 121, a first image information storage unit 122, and a second image information storage unit 123.

[0066] The blueprint information storage unit 121 stores blueprint information. Blueprint information is image information showing a blueprint of a specified space. A blueprint ID that identifies the blueprint is associated with the blueprint information. A blueprint is an example of a bird's-eye view. A blueprint is a drawing showing the design of a construction site, and may be a floor plan, blueprint, map, perspective view, or the like of the construction site. A bird's-eye view can also be called a bird's-eye view, and may be a view from above or a view from a high place.

[0067] The processor 11 uses Visual SLAM (Simultaneous Localization and Mapping) technology to identify the location of each of the first images from the shooting start point, shooting end point, and the first images included in the first shooting information received by the communication unit 13. The location of the shooting point is expressed as two-dimensional coordinate values ​​on the design drawing. The processor 11 also uses Visual SLAM technology to identify the shooting direction of each of the first images. The shooting direction of each of the first images is expressed, for example, as a three-dimensional polar coordinate vector.

[0068] The processor 11 creates first image information based on the first shooting information received by the communication unit 13 and stores the created first image information in the first image information storage unit 122 .

[0069] The first image information storage unit 122 stores first image information. The first image information is information that associates multiple first images captured by the imaging device 30 at multiple imaging locations with meta information for each first image. The first image information is generated each time the above-mentioned first imaging information is received. The meta information includes an imaging ID, imaging date and time, imaging direction, imaging location, and blueprint ID. The imaging direction is the imaging direction of the imaging device 30 that captured the first image. The imaging location is location information (two-dimensional coordinate values) that indicates the imaging location of the first image.

[0070] The processor 11 creates second image information based on the second shooting information received by the communication unit 13 and stores the created second image information in the second image information storage unit 123 .

[0071] The second image information storage unit 123 stores second image information. The second image information is information that associates at least one second image captured by the second information terminal 40 with meta information for each second image. The second image information is generated each time the above-mentioned second capture information is received. The meta information includes a capture ID and a capture date and time.

[0072] The communication unit 13 receives an instruction to display a design drawing from the first information terminal 20. The operation unit 24 of the first information terminal 20 accepts a design drawing selection by the user, and the communication unit 21 transmits an instruction to display the design drawing selected by the user to the server 10. The instruction to display the design drawing includes a design drawing ID.

[0073] Furthermore, the communication unit 13 receives an instruction to specify the shooting date and time from the first information terminal 20. The operation unit 24 of the first information terminal 20 accepts the selection of the shooting date and time by the user, and the communication unit 21 transmits an instruction to specify the shooting date and time selected by the user to the server 10. The instruction to specify the shooting date and time includes the shooting ID.

[0074] The acquisition unit 111 acquires, from the blueprint information storage unit 121, a blueprint corresponding to a blueprint ID included in the instruction to display the blueprint received by the communication unit 13. The acquisition unit 111 also acquires, from the first image information storage unit 122, a plurality of shooting locations corresponding to a shooting ID included in the instruction to specify a shooting date and time received by the communication unit 13. The acquisition unit 111 also acquires, from the first image information storage unit 122, a plurality of shooting dates and times of a plurality of first images corresponding to the shooting ID, and acquires, from the second image information storage unit 123, a shooting date and time of a second image corresponding to the shooting ID.

[0075] The display control unit 112 displays a blueprint of the workspace on the display 23 of the first information terminal 20. The display control unit 112 displays the blueprint acquired by the acquisition unit 111 on the display 23 of the first information terminal 20. The display control unit 112 transmits display data of the blueprint to the first information terminal 20 via the communication unit 13. The communication unit 21 of the first information terminal 20 receives the display data transmitted by the server 10. The display 23 displays the blueprint received by the communication unit 21.

[0076] The display control unit 112 displays a plurality of first icons indicating a plurality of shooting locations on the blueprint according to the shooting times. The plurality of first icons are associated with a plurality of first images captured by the imaging device 30 at the plurality of shooting locations, respectively.

[0077] The display control unit 112 displays information indicating that the content information is associated with a first icon, among the multiple first icons, whose shooting time is close to the recording time of the content information, on the blueprint. In the first embodiment, the shooting time is the shooting date and time of the first image, the content information is a second image captured by the second information terminal 40 while the multiple first images were being captured by the imaging device 30, and the recording time is the shooting date and time of the second image. Furthermore, the information indicating that the second image is associated with the first icon is a second icon different from the multiple first icons. That is, the display control unit 112 displays a second icon indicating that the second image is associated with a first icon, among the multiple first icons, whose shooting date and time is close to the shooting date and time of the second image, on the blueprint.

[0078] First, the processing of the second information terminal 40 in the first embodiment will be described.

[0079] FIG. 2 is a first flowchart showing an example of processing by the second information terminal 40 in the present embodiment 1, and FIG. 3 is a second flowchart showing an example of processing by the second information terminal 40 in the present embodiment 1.

[0080] First, in step S11, the second information terminal 40 displays the design drawing selected by the photographer. At this time, the second information terminal 40 accepts the selection of the design drawing by the photographer.

[0081] Next, in step S12, the second information terminal 40 determines whether the start point registration button has been pressed. At this time, the second information terminal 40 displays the start point registration button together with the design drawing. The second information terminal 40 accepts the photographer's input of the start point position on the design drawing, and also accepts the photographer's pressing of the start point registration button. For example, the photographer touches the start point position on the displayed design drawing. This inputs the start point position. The photographer then presses the start point registration button.

[0082] After pressing the start point registration button, the photographer presses the photographing button on the photographing device 30 to start photographing the first image with the photographing device 30 .

[0083] If it is determined that the start point registration button has not been pressed (NO in step S12), the process of step S12 is repeated.

[0084] On the other hand, if it is determined that the start point registration button has been pressed (YES in step S12), in step S13, the second information terminal 40 stores the photographing start point on the design drawing. The second information terminal 40 stores the coordinates of the photographing start point on the design drawing.

[0085] Next, in step S14, the second information terminal 40 determines whether or not the capture button has been pressed. After the start point registration button has been pressed, the second information terminal 40 displays a capture button for accepting capture of a second image.

[0086] FIG. 4 is a diagram showing an example of a display screen displayed on second information terminal 40 when a second image is captured in the first embodiment.

[0087] 4, a photographing button 401 is displayed on the display screen of the second information terminal 40. While the photographer is moving around taking photographs using the photographing device 30, if there is a location that the photographer wants to photograph individually, the photographer presses the photographing button 401.

[0088] Returning to FIG. 2, if it is determined that the capture button has been pressed (YES in step S14), the second information terminal 40 captures a second image in step S15.

[0089] Next, in step S16, the second information terminal 40 stores the second image, and the process returns to step S14.

[0090] On the other hand, if it is determined that the capture button has not been pressed (NO in step S14), in step S17, the second information terminal 40 determines whether or not multiple first images have been received from the photographing device 30. When the photographer reaches the photographing end point, the photographer presses the photographing button on the photographing device 30 again to end photographing. When the photographing button is pressed, the photographing device 30 transmits the captured multiple first images to the second information terminal 40.

[0091] If it is determined that a plurality of first images have not been received from the image capturing device 30 (NO in step S17), the process returns to step S14.

[0092] On the other hand, if it is determined that a plurality of first images have been received from the photographing device 30 (YES in step S17), the second information terminal 40 stores the received plurality of first images in step S18.

[0093] Next, in step S19, the second information terminal 40 determines whether the end point registration button has been pressed. At this time, the second information terminal 40 displays the end point registration button together with the design drawing. The second information terminal 40 accepts the photographer's input of the end point position on the design drawing, and also accepts the photographer's pressing of the end point registration button. For example, the photographer touches the end point position on the displayed design drawing. This inputs the end point position. The photographer then presses the end point registration button.

[0094] If it is determined that the end point registration button has not been pressed (NO in step S19), the process of step S19 is repeated.

[0095] On the other hand, if it is determined that the end point registration button has been pressed (YES in step S19), in step S20, the second information terminal 40 stores the photographing end point on the design drawing. The second information terminal 40 stores the coordinates of the photographing end point on the design drawing.

[0096] Next, in step S21, the second information terminal 40 transmits the first photographing information including the plurality of first images to the server 10.

[0097] Next, in step S22, the second information terminal 40 accepts the selection of a second image to be transmitted to the server 10. The photographer may take multiple second images while traveling. When multiple second images are taken, the photographer selects a second image to be transmitted to the server 10 from the multiple second images taken while traveling.

[0098] Next, in step S23, the second information terminal 40 determines whether the send button has been pressed. The second information terminal 40 accepts the photographer's selection of the second image to be sent and also accepts the photographer's pressing of the send button. After selecting the second image to be sent, the photographer presses the send button.

[0099] If it is determined that the send button has not been pressed (NO in step S23), the process returns to step S22.

[0100] On the other hand, if it is determined that the send button has been pressed (YES in step S23), in step S24, the second information terminal 40 transmits second shooting information including at least one selected second image to the server 10.

[0101] Note that if a second image is not captured while the photographing device 30 is capturing a plurality of first images, the processes of steps S22 to S24 may not be performed. Also, even if a second image is captured while the photographing device 30 is capturing a plurality of first images, if the second image is not selected, the second photographing information may not be transmitted. Furthermore, in the first embodiment, after the process of step S21 is performed, the processes of steps S22 to S24 are performed consecutively in time, but the present disclosure is not particularly limited to this, and the processes of steps S22 to S24 may be performed some time after the process of step S21 is performed.

[0102] Next, the processing of the server 10 in the first embodiment will be described.

[0103] FIG. 5 is a first flowchart showing an example of the processing of the server 10 in the first embodiment, and FIG. 6 is a second flowchart showing an example of the processing of the server 10 in the first embodiment.

[0104] First, in step S31 , the communication unit 13 receives the first photographing information transmitted by the second information terminal 40 .

[0105] Next, in step S32, the acquisition unit 111 acquires a design drawing from the design drawing information storage unit 121. The acquisition unit 111 reads out from the design drawing information storage unit 121 a design drawing corresponding to a design drawing ID included in the first photographing information received by the communication unit 13.

[0106] Next, in step S33, the acquisition unit 111 creates first image information based on the first shooting information and the blueprint. The acquisition unit 111 calculates the shooting location and shooting direction of each of the multiple first images on the blueprint using Visual SLAM technology. The first image information is information that associates the multiple first images with meta information for each first image. The meta information includes the shooting ID, shooting date and time, shooting direction, shooting location, and blueprint ID. The acquisition unit 111 stores the created first image information in the first image information storage unit 122.

[0107] Next, in step S34, the acquisition unit 111 determines whether the communication unit 13 has received the second shooting information.

[0108] If it is determined that the communication unit 13 has not received the second photographing information (NO in step S34), the process proceeds to step S36.

[0109] On the other hand, if it is determined that the communication unit 13 has received the second shooting information (YES in step S34), in step S35, the acquisition unit 111 creates second image information based on the second shooting information received by the communication unit 13. The second image information is information that associates at least one second image with meta information for each second image. The meta information includes a shooting ID and a shooting date and time. The acquisition unit 111 stores the created second image information in the second image information storage unit 123.

[0110] Next, in step S36, the communication unit 13 receives an instruction to display a blueprint from the first information terminal 20. In this case, a menu screen for selecting a blueprint is displayed on the display 23 of the first information terminal 20, and the operation unit 24 accepts a user instruction to select one blueprint from the menu screen. Also, a menu screen for selecting a photography ID (photographing date and time) is displayed on the display 23 of the first information terminal 20, and the operation unit 24 accepts a user instruction to select one photography ID (photographing date and time) from the menu screen. The input instruction is transmitted to the server 10 via the network NT and received by the communication unit 13. The instruction to display the blueprint includes the blueprint ID and the photography ID.

[0111] Next, in step S37, the display control unit 112 calculates the shooting period, which is the period between the shooting date and time at the first shooting location (shooting start point) of the multiple first images and the shooting date and time at the last shooting location (shooting end point).

[0112] Next, in step S38, the display control unit 112 determines whether the shooting date and time of the second image is within the shooting period. If it is determined that the shooting date and time of the second image is not within the shooting period (NO in step S38), in step S39, the display control unit 112 transmits display data of the design drawing to the first information terminal 20 via the communication unit 13, thereby displaying the design drawing on the display 23 of the first information terminal 20.

[0113] Next, in step S40, the display control unit 112 sends a display instruction to the first information terminal 20 via the communication unit 13 to display multiple first icons indicating multiple shooting locations on the design drawing, thereby displaying multiple first icons indicating multiple shooting locations on the design drawing displayed on the display 23 of the first information terminal 20.

[0114] On the other hand, if it is determined that the shooting date and time of the second image falls within the shooting period (YES in step S38), in step S41, the display control unit 112 selects, from among the multiple shooting locations, a shooting location whose shooting date and time is closest to the shooting date and time of the second image. The display control unit 112 selects a shooting location whose shooting date and time is the same as the shooting date and time of the second image. If there is no shooting location whose shooting date and time is the same as the shooting date and time of the second image, the display control unit 112 selects a shooting location whose shooting date and time is closest to the shooting date and time of the second image.

[0115] Next, in step S42, the display control unit 112 searches for a blank area within a predetermined range from the selected shooting location in which the second icon can be displayed. The predetermined range is, for example, an area within a circle of a predetermined radius centered on the selected shooting location.

[0116] Next, in step S43, the display control unit 112 determines whether or not there is a blank area within the predetermined range.

[0117] If it is determined that there is no blank area within the predetermined range (NO in step S43), the display control unit 112 expands the predetermined range in step S44. Thereafter, the process returns to step S42, and the display control unit 112 repeats the processes of steps S42 to S44 until a blank area is found.

[0118] On the other hand, if it is determined that there is a blank area within the specified range (YES in step S43), in step S45, the display control unit 112 displays the design drawing on the display 23 of the first information terminal 20 by transmitting display data of the design drawing to the first information terminal 20 via the communication unit 13.

[0119] Next, in step S46, the display control unit 112 sends a display instruction to the first information terminal 20 via the communication unit 13 to display multiple first icons indicating multiple shooting locations on the design drawing, thereby displaying multiple first icons indicating multiple shooting locations on the design drawing displayed on the display 23 of the first information terminal 20.

[0120] Next, in step S47, the display control unit 112 displays a second icon indicating that the second image is associated with a first icon, among the multiple first icons, whose shooting date and time is close to the shooting date and time of the second image, in a margin area of ​​the blueprint by sending a display instruction to the first information terminal 20 via the communication unit 13, thereby displaying the second icon in a margin area of ​​the blueprint.

[0121] FIG. 7 is a diagram showing an example of the design drawing screen 200 displayed on the first information terminal 20 in the first embodiment.

[0122] The blueprint screen 200 displays a blueprint 201 of the workspace. A plurality of first icons 202 are superimposed on the blueprint screen 200. Furthermore, on the blueprint screen 200, a second icon 203 indicating that the second image is associated with a first icon 202 whose shooting date and time is close to that of the second image is displayed in a blank area of ​​the blueprint 201. The second icon 203 is, for example, rectangular and is an image resembling a camera.

[0123] The first icon 202 is an icon indicating a shooting location. An image captured at the shooting location is associated with the first icon 202. In this example, the first icon 202 is configured as a circular image. Note that the shape of the first icon 202 is just an example, and other shapes may be used. The multiple first icons 202 displayed here correspond to multiple shooting locations belonging to one shooting operation. The route from the first icon 202A at the shooting start location to the first icon 202B at the shooting end location indicates the trajectory followed by the photographer in this shooting operation.

[0124] A first image captured at the shooting location is associated with the first icon 202. The operation unit 24 may accept a user's selection of one first icon 202 from the plurality of first icons 202. When one first icon 202 from the plurality of first icons 202 is selected, the display control unit 112 may read out a first image corresponding to the selected one first icon 202 from the first image information storage unit 122 and display the read out first image on the display 23.

[0125] The display control unit 112 displays the second icon 203 in a location that does not overlap with any characters or figures in the blueprint 201. Note that the display control unit 112 may display the second icon 203 in a blank area within the building in the blueprint 201, a blank area outside the building in the blueprint 201, or a portion outside the area in the blueprint 201 that includes the plurality of first icons 202.

[0126] The display control unit 112 associates the second icon 203 with the first icon 202 using a frame line 204 of a predetermined shape. The display control unit 112 connects the second icon 203 with the first icon 202 using the frame line 204 of a predetermined shape. The predetermined shape is, for example, a triangle. Note that the tip of the frame line 204 may coincide with the outer periphery of the first icon 202. Furthermore, the frame line 204 may be displayed so as to indicate the correspondence between the first icon 202 and the second icon 203. For example, the tip of the frame line 204 may be located inside the outer periphery of the circular image of the first icon 202, or may be located within a predetermined range outside the outer periphery of the circular image of the first icon 202.

[0127] Furthermore, the multiple first icons 202 are associated with the shooting directions of the imaging device 30 that captured images at the multiple shooting locations. Points on the outer edge of the first icon 202 identified by the shooting directions associated with the first icon 202 are connected to a frame line 204. The position of the tip of the frame line 204 extending from the second icon 203 to the first icon 202 indicates the shooting direction associated with the first icon 202. That is, the frame line 204 is displayed so that the intersection of a line extending from the center of the circular image representing the first icon 202 in the shooting direction associated with the first icon 202 and the periphery of the circular image representing the first icon 202 coincides with the vertex of the triangular frame line 204 extending from the second icon 203. The user can know the shooting direction associated with the first icon 202 by checking where the tip of the frame line 204 is located on the outer edge of the circular image of the first icon 202. This allows the user to confirm the direction in which the first and second images were captured.

[0128] The second icon 203 is associated with a second image captured by the second information terminal 40. The operation unit 24 may accept a user selection of the second icon 203. When the second icon 203 is selected, the display control unit 112 may read out a second image corresponding to the selected second icon 203 from the second image information storage unit 123 and display the read out second image on the display 23.

[0129] When a plurality of second images are associated with the shooting ID, the display control unit 112 may display a plurality of second icons 203 indicating the shooting locations of the plurality of second images on the design drawing 201. Furthermore, the display control unit 112 may connect the first icon 202 and the second icon 203 with a straight line.

[0130] Furthermore, in the first embodiment, the display control unit 112 displays the second icon 203 that is different from the multiple first icons 202, but the present disclosure is not particularly limited to this. The display control unit 112 may change the display mode of a first icon whose shooting date and time is close to that of the second image from that of the other first icons. In this case, the display control unit 112 may change the color of a first icon whose shooting date and time is close to that of the second image from that of the other first icons, or may change the shape of a first icon whose shooting date and time is close to that of the second image from that of the other first icons.

[0131] In this way, a plurality of first icons 202 indicating the shooting locations of the plurality of first images captured by the imaging device 30 and a second icon 203 indicating the shooting location of the second image captured by the second information terminal 40 while the plurality of first images were being captured are displayed on the blueprint 201. Therefore, the user can know that there is a second image captured from a different viewpoint than the first image at the shooting location of the first image. Furthermore, the user can reduce the time required to organize the second images captured separately from the plurality of first images, thereby improving user convenience.

[0132] Furthermore, the user can be notified that there is recorded content information other than the images taken at the shooting location, which reduces the processing steps required to access the content information. Furthermore, since the user is notified that there is recorded content information other than the images taken at the shooting location, it is possible to reduce the processing steps required by the computer to access the content information, thereby reducing the burden on the computer.

[0133] (Embodiment 2) In embodiment 1, second images are captured by second information terminal 40 while multiple first images are being captured by imaging device 30. In contrast, in embodiment 2, audio information recorded by second information terminal 40A is acquired while multiple first images are being captured by imaging device 30.

[0134] FIG. 8 is a diagram showing the overall configuration of an information processing system 1A according to the second embodiment.

[0135] The information processing system 1A includes a server 10A, a first information terminal 20, an imaging device 30, and a second information terminal 40A. In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals and will not be described again. In addition, in the configuration of the second embodiment, the same functions as those in the first embodiment will not be described again.

[0136] The second information terminal 40A is configured as a portable information terminal such as a smartphone or a tablet computer, and connects the image capturing device 30 to the network NT. The second information terminal 40A and the image capturing device 30 are connected to each other so that they can communicate with each other via a short-range wireless communication path such as Bluetooth (registered trademark). The image capturing device 30 is an example of an image capturing device, and the second information terminal 40A is an example of an audio recording device.

[0137] The second information terminal 40A receives the plurality of first images transmitted by the photographing device 30. The second information terminal 40A transmits first photographing information including the plurality of photographed first images to the server 10A via the network NT.

[0138] The second information terminal 40A also includes a microphone that collects surrounding sounds. The photographer moves around the construction site while photographing the construction site with the photography device 30. At this time, the photographer speaks at a desired location within the construction site. The microphone of the second information terminal 40A collects the photographer's voice. The processor of the second information terminal 40A records the audio information collected by the microphone in the memory of the second information terminal 40A. The audio information is an example of content information.

[0139] The second information terminal 40A transmits recorded information, including audio information recorded by the second information terminal 40A while the photographing device 30 is photographing multiple first images, to the server 10A via the network NT. The recorded information includes at least one piece of recorded audio information and meta information for each piece of audio information. The meta information includes a photographing ID, a recording start date and time, and a recording end date and time. The photographing ID is an identifier for identifying the photographing operation. The photographing ID is the same as the photographing ID associated with the multiple first images. The recording start date and time is the date and time when recording of the audio information started, and the recording end date and time is the date and time when recording of the audio information ended. The recording start date and time and the recording end date and time are obtained, for example, by a clock provided in the second information terminal 40A.

[0140] In the second embodiment, the microphone included in the second information terminal 40A collects the audio, but the present disclosure is not limited to this, and an audio recording device separate from the second information terminal 40A may collect the audio. The second information terminal 40A may be communicably connected to the other audio recording device, receive audio information transmitted by the other audio recording device, and transmit recorded information including the received audio information to the server 10A. Alternatively, an audio recording unit included in the image capturing device 30 may record the audio information. In this case, the audio recording unit included in the image capturing device 30 is an example of an audio recording device.

[0141] The server 10A includes a processor 11A, a memory 12A, and a communication unit 13A. The processor 11A is configured, for example, by a central processing unit (CPU). The processor 11A includes an acquisition unit 111A and a display control unit 112A. The acquisition unit 111A and the display control unit 112A may be realized by the processor 11A executing an information processing program, or may be configured by a dedicated hardware circuit such as an ASIC. The information processing program may be recorded on a non-transitory computer-readable recording medium.

[0142] The communication unit 13A is a communication interface that connects the server 10A to the network NT. The communication unit 13A receives first shooting information transmitted by the second information terminal 40A. The communication unit 13A receives audio information transmitted by the second information terminal 40A. As a result, the communication unit 13A acquires content information recorded by the second information terminal 40A, i.e., audio information recorded by the second information terminal 40A. The communication unit 13A receives instruction signals from the first information terminal 20 that indicate various instructions received from the user. The communication unit 13A transmits display data for displaying various display screens to the first information terminal 20.

[0143] The memory 12A is configured as a non-volatile rewritable storage device such as a hard disk drive or a solid state drive, and includes a design drawing information storage unit 121, a first image information storage unit 122, and an audio information storage unit 124.

[0144] The audio information storage unit 124 stores recorded information including audio information. The recorded information is information that associates at least one piece of audio information collected by the second information terminal 40A with meta information for each piece of audio information. The recorded information is stored each time the above-mentioned recorded information is received. The meta information includes a shooting ID, a recording start date and time, and a recording end date and time.

[0145] The acquisition unit 111A acquires multiple shooting dates and times of multiple first images corresponding to the shooting ID from the first image information storage unit 122, and also acquires the recording start date and time of audio information corresponding to the shooting ID from the audio information storage unit 124.

[0146] The display control unit 112A displays information indicating that the content information is associated with a first icon, among the multiple first icons, whose shooting time is close to the recording time of the content information, on the blueprint. In the second embodiment, the shooting time is the shooting date and time of the first image, the content information is audio information recorded by the second information terminal 40A while the multiple first images were being captured by the imaging device 30, and the recording time is the recording start date and time or the recording end date and time of the audio information. Furthermore, the information indicating that the audio information is associated with a first icon is a second icon different from the multiple first icons. That is, the display control unit 112A displays a second icon indicating that the audio information is associated with a first icon, among the multiple first icons, whose shooting date and time is close to the recording start date and time of the audio information, on the blueprint.

[0147] First, the processing of the second information terminal 40A in the second embodiment will be described.

[0148] FIG. 9 is a first flowchart showing an example of processing by the second information terminal 40A in this embodiment 2, and FIG. 10 is a second flowchart showing an example of processing by the second information terminal 40A in this embodiment 2.

[0149] The processing in steps S51 to S53 is the same as the processing in steps S11 to S13 shown in FIG. 2, and therefore a description thereof will be omitted.

[0150] Next, in step S54, the second information terminal 40A determines whether the recording start button has been pressed. After the start point registration button has been pressed, the second information terminal 40A displays the recording start button to accept the start of audio recording.

[0151] FIG. 11 is a diagram showing an example of a display screen displayed on second information terminal 40A when recording starts in the second embodiment.

[0152] 11 , a recording start button 402 is displayed on the display screen of the second information terminal 40A. When the photographer wants to record audio while moving around and taking pictures using the camera device 30, the photographer presses the recording start button 402. For example, while taking pictures of a construction site, the photographer presses the recording start button 402 at a location where he or she wants to leave a comment and speaks. This starts recording the photographer's speech.

[0153] 9, if it is determined that the recording start button has been pressed (YES in step S54), the second information terminal 40A starts recording audio information in step S55. The audio information is stored in a memory provided in the second information terminal 40A.

[0154] Next, in step S56, the second information terminal 40A determines whether the recording stop button has been pressed. After the recording start button has been pressed, the second information terminal 40A displays a recording stop button to accept the end of voice recording.

[0155] FIG. 12 is a diagram showing an example of a display screen displayed on second information terminal 40A when recording ends in the second embodiment.

[0156] 12, a recording stop button 403 is displayed on the display screen of the second information terminal 40A. When the photographer wants to end recording, he or she presses the recording stop button 403. For example, when the photographer finishes speaking, the photographer presses the recording stop button 403. This ends the recording of the photographer's speech, and the photographer can leave an audio comment at the shooting location of the first image.

[0157] Returning to FIG. 9, if it is determined that the recording stop button has been pressed (YES in step S56), the second information terminal 40A ends recording of the audio information in step S57.

[0158] On the other hand, if it is determined that the recording stop button has not been pressed (NO in step S56), the process of step S56 is repeated.

[0159] If the recording of audio information has finished in step S57, or if it is determined that the recording start button has not been pressed (NO in step S54), in step S58, the second information terminal 40A determines whether multiple first images have been received from the photographing device 30.

[0160] The processing in steps S58 to S62 is the same as the processing in steps S17 to S21 in FIGS. 2 and 3, and therefore a description thereof will be omitted.

[0161] Next, in step S63, the second information terminal 40A transmits recorded information including at least one piece of audio information to the server 10A.

[0162] In the second embodiment, the second information terminal 40A may accept a selection of audio information to be transmitted to the server 10A. The photographer may record multiple pieces of audio information while traveling. When multiple pieces of audio information are recorded, the photographer may select audio information to be transmitted to the server 10A from the multiple pieces of audio information recorded while traveling. The second information terminal 40A may transmit recorded information including at least one selected piece of audio information to the server 10A. The selection of audio information and transmission of the recorded information may be performed some time after the processing of step S62 is performed. If no audio information is recorded while the photographing device 30 is capturing multiple first images, the processing of step S63 may not be performed.

[0163] Next, the processing of the server 10A in the second embodiment will be described.

[0164] FIG. 13 is a first flowchart showing an example of the processing of the server 10A in the second embodiment, and FIG. 14 is a second flowchart showing an example of the processing of the server 10A in the second embodiment.

[0165] The processing in steps S71 to S73 is the same as the processing in steps S31 to S33 in FIG. 5, and therefore a description thereof will be omitted.

[0166] Next, in step S74, the acquisition unit 111A determines whether or not the communication unit 13A has received the recording information.

[0167] If it is determined that the communication unit 13A has not received the recording information (NO in step S74), the process proceeds to step S76.

[0168] On the other hand, if it is determined that the communication unit 13A has received the recording information (YES in step S74), in step S75, the acquisition unit 111A stores the recording information received by the communication unit 13A in the audio information storage unit 124. The recording information is information that associates at least one piece of audio information with meta information for each piece of audio information. The meta information includes the shooting ID, the recording start date and time, and the recording end date and time.

[0169] The processes in steps S76 and S77 are the same as those in steps S36 and S37 in FIG. 5, and therefore will not be described further.

[0170] Next, in step S78, the display control unit 112A determines whether the recording start date and time of the audio information falls within the shooting period. If it is determined that the recording start date and time of the audio information does not fall within the shooting period (NO in step S78), in step S79, the display control unit 112A transmits display data of the design drawing to the first information terminal 20 via the communication unit 13A, thereby displaying the design drawing on the display 23 of the first information terminal 20.

[0171] The process of step S80 is the same as the process of step S40 in FIG. 5, and therefore a description thereof will be omitted.

[0172] On the other hand, if it is determined that the recording start date and time of the audio information falls within the shooting period (YES in step S78), in step S81, the display control unit 112A selects, from among the multiple shooting locations, a shooting location whose shooting date and time is closest to the recording start date and time of the audio information. The display control unit 112A selects a shooting location whose shooting date and time is the same as the recording start date and time of the audio information. If there is no shooting location whose shooting date and time is the same as the recording start date and time of the audio information, the display control unit 112A selects a shooting location whose shooting date and time is closest to the recording start date and time of the audio information.

[0173] The processing in steps S82 to S86 is the same as the processing in steps S42 to S46 in FIG. 6, and therefore a description thereof will be omitted.

[0174] Next, in step S87, the display control unit 112A sends a display instruction to the first information terminal 20 via the communication unit 13A to display a second icon indicating that audio information has been associated with a first icon among the multiple first icons whose shooting date and time is close to the recording start date and time of the audio information in a blank area within the blueprint, thereby displaying the second icon in a blank area within the blueprint.

[0175] FIG. 15 is a diagram showing an example of a design drawing screen 200A displayed on the first information terminal 20 in the second embodiment.

[0176] The blueprint screen 200A displays a blueprint 201 of the workspace. A plurality of first icons 202 are superimposed on the blueprint screen 200A. Also, on the blueprint screen 200A, a second icon 203A indicating that audio information is associated with a first icon 202 whose shooting date and time is close to the recording start date and time of the audio information is displayed in a blank area of ​​the blueprint 201. The second icon 203A is, for example, a rectangular image resembling a microphone.

[0177] The plurality of first icons 202 and the frame 204 in the second embodiment are the same as the plurality of first icons 202 and the frame 204 in the first embodiment.

[0178] The display control unit 112A displays the second icon 203A in a location that does not overlap with any characters or figures in the blueprint 201. Note that the display control unit 112A may display the second icon 203A in a blank area within the building in the blueprint 201, a blank area outside the building in the blueprint 201, or a portion outside the area in the blueprint 201 that includes the plurality of first icons 202.

[0179] The second icon 203A is associated with audio information recorded by the second information terminal 40A. The operation unit 24 may accept a user selection of the second icon 203A. When the second icon 203A is selected, the display control unit 112A may read audio information corresponding to the selected second icon 203A from the audio information storage unit 124 and output the read audio information from a speaker. Note that the first information terminal 20 further includes a speaker. In this case, the communication unit 13A may transmit the audio information corresponding to the second icon 203A to the first information terminal 20. The speaker of the first information terminal 20 may output the received audio information.

[0180] In addition, when a plurality of pieces of audio information are associated with the shooting ID, the display control unit 112A may display a plurality of second icons 203A on the design drawing 201, each of which indicates the recording start point of the plurality of pieces of audio information.

[0181] Furthermore, in the second embodiment, the display control unit 112A displays the second icon 203A, which is different from the multiple first icons 202. However, the present disclosure is not particularly limited to this. The display control unit 112A may cause the display mode of a first icon whose shooting date and time is close to the recording start date and time of the audio information to be different from that of the other first icons. In this case, the display control unit 112A may cause the color of the first icon whose shooting date and time is close to the recording start date and time of the audio information to be different from that of the other first icons, or may cause the shape of the first icon whose shooting date and time is close to the recording start date and time of the audio information to be different from that of the other first icons.

[0182] In this way, a plurality of first icons 202 indicating the shooting locations of the plurality of first images captured by the imaging device 30 and a second icon 203A indicating the recording start point of the audio information recorded by the second information terminal 40A while capturing the plurality of first images are displayed on the blueprint 201. Therefore, the user can know that there is an audio comment at the shooting location of the first image. Furthermore, the user can reduce the time required to organize the audio information recorded while capturing the plurality of first images, thereby improving user convenience.

[0183] (Embodiment 3) In embodiment 2, audio information recorded by second information terminal 40A is acquired while multiple first images are being captured by image capturing device 30. In contrast, in embodiment 3, audio information recorded by image capturing device 30B is acquired while multiple first images are being captured by image capturing device 30B.

[0184] FIG. 16 is a diagram showing the overall configuration of an information processing system 1B according to the third embodiment.

[0185] The information processing system 1B includes a server 10B, a first information terminal 20, an imaging device 30B, and a second information terminal 40B. In the third embodiment, the same components as those in the first and second embodiments are denoted by the same reference numerals and will not be described again. In addition, in the configuration of the third embodiment, the same functions as those in the first and second embodiments will not be described again.

[0186] The image capturing device 30B is an example of an image capturing device and an audio recording device.

[0187] The image capturing device 30B is configured, for example, as an omnidirectional camera, and captures first images at a predetermined frame rate. The image capturing device 30B also includes a microphone that collects surrounding sound. The image capturing device 30B records audio information while capturing multiple first images. The image capturing device 30B is, for example, a portable image capturing device carried by a photographer. The photographer may be, for example, a construction site worker or site supervisor. The image capturing device 30B may be held in the photographer's hand or worn on the photographer's body (for example, head). The image capturing device 30B may also be configured as a regular camera.

[0188] The photographer moves through the construction site while photographing and recording the construction site with the camera device 30B. At the start point of photography, the photographer points the camera device 30B in a northerly direction and presses the camera button on the camera device 30B to start photographing and recording. When the photographer reaches the end point of photography, he presses the camera button again to stop photographing and recording. When photographing and recording are finished, the camera device 30B transmits the series of captured first images and audio information to the second information terminal 40B. The camera device 30B records audio information from the start to the end of photographing the first images.

[0189] First, the photographing device 30B displays a photographing button for instructing the start or stop of photographing and accepts the photographer's pressing of the photographing button. Next, the photographing device 30B determines whether the photographing button for instructing the start of photographing has been pressed. If the photographing button has been pressed, the photographing device 30B starts photographing a first image and starts recording audio information, and stores the photographed first image and the recorded audio information in its internal memory. Next, the photographing device 30B determines whether the photographing button for instructing the stop of photographing has been pressed. If the photographing button has been pressed, the photographing device 30B stops storing the first image and stops recording audio information, and transmits the multiple first images and audio information stored in its internal memory to the second information terminal 40B.

[0190] The second information terminal 40B receives the plurality of first images and audio information transmitted by the image capturing device 30B. The second information terminal 40B also displays a blueprint of the construction site and accepts designation by the photographer of the position of the image capturing start point and the position of the image capturing end point on the blueprint.

[0191] The second information terminal 40B transmits first photographing information, including the captured multiple first images and collected audio information, to the server 10B via the network NT. The first photographing information is generated each time a single photographing operation is performed. One photographing operation refers to a series of operations from when a worker holding the photographing device 30B starts photographing at a construction site until when the photographing ends. A single photographing operation captures multiple first images and collects audio information. The first photographing information includes the captured multiple first images, meta information for each of the multiple first images, and collected audio information. The meta information includes a photographing ID, a photographing date and time, and a blueprint ID. The photographing ID is an identifier for identifying the photographing operation. The photographing date and time is the date and time when the first image was photographed. The blueprint ID is an identifier for identifying a blueprint of a specified space corresponding to the image.

[0192] The second information terminal 40B accepts the recording start date and time and the recording end date and time specified by the photographer. The recording start date and time is specified by the photographer pressing a recording start button displayed on the display of the second information terminal 40B. The recording end date and time is specified by the photographer pressing a recording stop button displayed on the display of the second information terminal 40B. The recording start date and time and the recording end date and time are used by the server 10B to identify only the necessary portions of the audio information recorded from the start to the end of capturing the first image.

[0193] The photographer moves around the construction site while photographing and recording the construction site with the camera device 30B. At this time, the photographer speaks at a desired location within the construction site. The microphone of the camera device 30B collects the photographer's voice. The second information terminal 40B accepts input by the photographer of the recording start date and time and recording end date and time of the audio information collected by the camera device 30B that should be stored in the server 10B. The second information terminal 40B does not store the audio information while photographing the first image, but stores only the recording start date and time and recording end date and time.

[0194] The second information terminal 40B transmits recording date and time information, including the recording start date and time and the recording end date and time specified by the second information terminal 40B while the multiple first images are being captured by the imaging device 30B, to the server 10B via the network NT. The recording date and time information includes the recording start date and time, the recording end date and time, and a shooting ID. The shooting ID is an identifier for identifying the shooting operation. The shooting ID is the same as the shooting ID associated with the multiple first images. The recording start date and time and the recording end date and time are obtained, for example, by a clock provided in the second information terminal 40B.

[0195] The server 10B includes a processor 11B, a memory 12B, and a communication unit 13B. The processor 11B is configured, for example, by a central processing unit (CPU). The processor 11B includes an acquisition unit 111B and a display control unit 112B. The acquisition unit 111B and the display control unit 112B may be realized by the processor 11B executing an information processing program, or may be configured by a dedicated hardware circuit such as an ASIC. The information processing program may be recorded on a non-transitory computer-readable recording medium.

[0196] The communication unit 13B is a communication interface that connects the server 10B to the network NT. The communication unit 13B receives first photographing information transmitted by the second information terminal 40B. The first photographing information includes a plurality of first images and audio information. The communication unit 13B receives recording date and time information transmitted by the second information terminal 40B. As a result, the communication unit 13B acquires content information recorded by the photographing device 30B, i.e., audio information recorded by the photographing device 30B. The communication unit 13B receives instruction signals from the first information terminal 20 that indicate various instructions received from the user. The communication unit 13B transmits display data for displaying various display screens to the first information terminal 20.

[0197] The memory 12B is configured as a non-volatile rewritable storage device such as a hard disk drive or a solid state drive, etc. The memory 12B includes a design drawing information storage unit 121, a first image information storage unit 122, and an audio information storage unit 124B.

[0198] The acquisition unit 111B creates recorded information based on the audio information included in the first shooting information received by the communication unit 13B and the recording date and time information received by the communication unit 13B, and stores the created recorded information in the audio information storage unit 124B. The acquisition unit 111B extracts audio information for a recording period specified by the recording start date and time and recording end date and time included in the recording date and time information from the audio information included in the first shooting information. The acquisition unit 111B then generates recorded information that associates the extracted audio information with meta information. The meta information includes the shooting ID, the recording start date and time, and the recording end date and time.

[0199] The audio information storage unit 124B stores recorded information including audio information. The recorded information is information in which at least one piece of audio information extracted from the audio information collected by the image capture device 30B is associated with meta information of each piece of audio information. The recorded information is stored each time the above-mentioned recorded information is received.

[0200] The acquisition unit 111B acquires multiple shooting dates and times of multiple first images corresponding to the shooting ID from the first image information storage unit 122, and also acquires the recording start date and time of the audio information corresponding to the shooting ID from the audio information storage unit 124B.

[0201] The display control unit 112B displays information indicating that the content information is associated with a first icon, among the plurality of first icons, whose shooting time is close to the recording time of the content information, on the blueprint. In the third embodiment, the shooting time is the shooting date and time of the first image, the content information is audio information recorded by the imaging device 30B while the imaging device 30B is capturing the plurality of first images, and the recording time is the recording start date and time or the recording end date and time of the audio information. Furthermore, the information indicating that the audio information is associated with a first icon is a second icon different from the plurality of first icons. That is, the display control unit 112B displays a second icon indicating that the audio information is associated with a first icon, among the plurality of first icons, whose shooting date and time is close to the recording start date and time of the audio information, on the blueprint.

[0202] First, the processing of the second information terminal 40B in the third embodiment will be described.

[0203] FIG. 17 is a first flowchart showing an example of processing of the second information terminal 40B in this embodiment 3, and FIG. 18 is a second flowchart showing an example of processing of the second information terminal 40B in this embodiment 3.

[0204] The processing in steps S91 to S93 is the same as the processing in steps S11 to S13 shown in FIG. 2, and therefore a description thereof will be omitted.

[0205] Next, in step S94, the second information terminal 40B determines whether the recording start button has been pressed. After the start point registration button has been pressed, the second information terminal 40B displays the recording start button for accepting the audio recording start date and time.

[0206] In the third embodiment, the display screen displayed on the second information terminal 40B when recording starts is the same as the display screen in the second embodiment shown in Fig. 11. When the photographer wants to record audio while moving around and taking pictures using the camera device 30B, the photographer presses the recording start button 402. For example, while photographing and recording a construction site, the photographer presses the recording start button 402 at a location where he or she wants to leave a comment and speaks. This causes the recording start date and time of the photographer's speech to be stored.

[0207] If it is determined that the recording start button has been pressed (YES in step S94), then in step S95, the second information terminal 40B stores a recording start date and time indicating the date and time when recording of the audio information will start. The recording start date and time is stored in a memory provided in the second information terminal 40B.

[0208] Next, in step S96, the second information terminal 40B determines whether the recording stop button has been pressed. After the recording start button has been pressed, the second information terminal 40B displays a recording stop button for receiving the audio recording end date and time.

[0209] In the third embodiment, the display screen displayed on the second information terminal 40B when recording ends is the same as the display screen in the second embodiment shown in FIG. 12. When the photographer wants to end recording, he or she presses the recording stop button 403. For example, when the photographer finishes speaking, the photographer presses the recording stop button 403. This causes the recording end date and time of the photographer's speech to be stored. The period from the recording start date and time to the recording end date and time indicates the period during which the photographer wants to keep the audio information collected by the camera device 30B during the shooting period. The photographer can leave audio comments for the desired period.

[0210] If it is determined that the recording stop button has been pressed (YES in step S96), then in step S97, the second information terminal 40B stores a recording end date and time indicating the date and time when recording of the audio information will end. The recording end date and time is stored in the memory of the second information terminal 40B.

[0211] On the other hand, if it is determined that the recording stop button has not been pressed (NO in step S96), the process of step S96 is repeated.

[0212] If the audio end date and time is stored in step S97, or if it is determined that the recording start button has not been pressed (NO in step S94), in step S98, the second information terminal 40B determines whether multiple first images have been received from the photographing device 30B.

[0213] The processing in steps S98 to S102 is the same as the processing in steps S17 to S21 in FIGS. 2 and 3, and therefore a description thereof will be omitted.

[0214] Next, in step S103, the second information terminal 40B transmits recording date and time information including at least one combination of the recording start date and time and the recording end date and time to the server 10B.

[0215] In the third embodiment, the second information terminal 40B may accept a selection of a combination of the recording start date / time and the recording end date / time to be transmitted to the server 10B. The photographer may specify multiple combinations of the recording start date / time and the recording end date / time while traveling. If multiple combinations of the recording start date / time and the recording end date / time are specified, the photographer may select a combination of the recording start date / time and the recording end date / time to be transmitted to the server 10B from the multiple combinations specified while traveling. The second information terminal 40B may transmit recording date / time information including at least one selected combination of the recording start date / time and the recording end date / time to the server 10B. The selection of the combination of the recording start date / time and the recording end date / time and the transmission of the recording date / time information may be performed some time after the processing of step S102 is performed. If the recording start date / time and the recording end date / time are not stored while the photographing device 30B is capturing the multiple first images, the processing of step S103 may not be performed.

[0216] Next, the processing of the server 10B in the third embodiment will be described.

[0217] FIG. 19 is a first flowchart showing an example of the processing of the server 10B in the third embodiment, and FIG. 20 is a second flowchart showing an example of the processing of the server 10B in the third embodiment.

[0218] The processing in steps S111 to S113 is the same as the processing in steps S31 to S33 in FIG. 5, and therefore a description thereof will be omitted.

[0219] Next, in step S114, the acquisition unit 111B determines whether or not the communication unit 13B has received the recording date and time information.

[0220] If it is determined that the communication unit 13B has not received the recording date and time information (NO in step S114), the process proceeds to step S116.

[0221] On the other hand, if it is determined that the communication unit 13B has received the recording date and time information (YES in step S114), in step S115, the acquisition unit 111B creates recording information based on the audio information included in the first shooting information received by the communication unit 13B and the recording date and time information received by the communication unit 13B, and stores the created recording information in the audio information storage unit 124B. The recording information is information that associates at least one piece of audio information with meta information for each piece of audio information. The meta information includes the shooting ID, the recording start date and time, and the recording end date and time.

[0222] The processing in steps S116 to S127 is the same as the processing in steps S76 to S87 in FIGS. 13 and 14, and therefore a description thereof will be omitted.

[0223] In addition, the design drawing screen displayed on the first information terminal 20 in the third embodiment is the same as the design drawing screen 200A in the second embodiment shown in FIG.

[0224] In the third embodiment, the second icon 203A is associated with audio information that is recorded by the imaging device 30B and in which the recording start date and time and the recording end date and time are specified by the second information terminal 40B. The operation unit 24 may accept a user selection of the second icon 203A. When the second icon 203A is selected, the display control unit 112B may read out audio information corresponding to the selected second icon 203A from the audio information storage unit 124B and output the read audio information from the speaker. In this case, the communication unit 13B may transmit the audio information corresponding to the second icon 203A to the first information terminal 20. The speaker of the first information terminal 20 may output the received audio information.

[0225] In addition, in the third embodiment, the acquisition unit 111B generates recorded information in which at least one piece of audio information extracted from audio information included in the first shooting information is associated with meta information for each piece of audio information, and stores the generated recorded information in the audio information storage unit 124B. However, the present disclosure is not particularly limited to this. The acquisition unit 111B may generate recorded information in which audio information included in the first shooting information is associated with at least one combination of a recording start date and time and a recording end date and time, and an imaging ID, and store the generated recorded information in the audio information storage unit 124B. In this case, the second icon 203A is associated with audio information recorded by the imaging device 30B and a combination of a recording start date and time and a recording end date and time specified by the second information terminal 40B. When the second icon 203A is selected, the display control unit 112B may extract audio information for a recording period specified by the recording start date and time and the recording end date and time corresponding to the selected second icon 203A from the audio information corresponding to the selected second icon 203A, and output the extracted audio information from a speaker.

[0226] Furthermore, in the second and third embodiments, the content information includes audio information recorded by the second information terminal 40A or the imaging device 30B while the imaging device 30 or 30B is capturing the multiple first images, but the present disclosure is not particularly limited to this, and the content information may include text information converted from audio information recorded by the second information terminal 40A or the imaging device 30B while the imaging device 30 or 30B is capturing the multiple first images. The text information may include the recording start date and time.

[0227] In this case, the memory 12A or 12B may include a text information storage unit that stores text information. The acquisition unit 111A or 111B may convert the voice information into text information using existing voice recognition technology and store the converted text information in the text information storage unit. The display control unit 112A or 112B may display a second icon indicating that the text information is associated with a first icon, among the multiple first icons, whose shooting date and time is close to the recording start date and time of the text information, in a blank area within the blueprint. Note that the second information terminal 40A may convert the voice information into text information and transmit the converted text information to the server 10A. The acquisition unit 111A may store the received text information in the text information storage unit.

[0228] (Embodiment 4) In embodiment 1, a second image captured by second information terminal 40 is acquired while a plurality of first images are being captured by imaging device 30. In contrast, in embodiment 4, text information input by second information terminal 40C is acquired while a plurality of first images are being captured by imaging device 30.

[0229] FIG. 21 is a diagram showing the overall configuration of an information processing system 1C according to the fourth embodiment.

[0230] The information processing system 1C includes a server 10C, a first information terminal 20, an imaging device 30, and a second information terminal 40C. In the fourth embodiment, the same components as those in the first embodiment are denoted by the same reference numerals and will not be described again. In addition, in the configuration of the fourth embodiment, the same functions as those in the first embodiment will not be described again.

[0231] The second information terminal 40C is configured as a portable information terminal such as a smartphone or a tablet computer, and connects the photographing device 30 to the network NT. The second information terminal 40C and the photographing device 30 are connected to each other so that they can communicate with each other via a short-range wireless communication path such as Bluetooth (registered trademark). The photographing device 30 is an example of a photographing device, and the second information terminal 40C is an example of an input device.

[0232] The second information terminal 40C receives the plurality of first images transmitted by the photographing device 30. The second information terminal 40C transmits first photographing information including the plurality of photographed first images to the server 10C via the network NT.

[0233] The second information terminal 40C also has an operation unit that accepts input of text information by the photographer. The operation unit is composed of, for example, a keyboard, a touch panel, a mouse, etc. The photographer moves around the construction site while photographing the construction site with the photography device 30. At this time, the photographer inputs text information at a desired location within the construction site. The operation unit of the second information terminal 40C accepts input of text information by the photographer. The processor of the second information terminal 40C records the text information input by the operation unit in the memory of the second information terminal 40C. The text information is an example of content information.

[0234] The second information terminal 40C transmits at least one piece of text information input by the second information terminal 40C while the photographing device 30 is photographing a plurality of first images to the server 10C via the network NT. The text information includes meta information. The meta information includes a photographing ID and the input date and time of the text information. The photographing ID is an identifier for identifying the photographing operation. The photographing ID is the same as the photographing ID associated with the plurality of first images. The input date and time is the date and time when the text information was input. The input date and time is obtained, for example, by a clock provided in the second information terminal 40C.

[0235] In the fourth embodiment, an operation unit included in the second information terminal 40C accepts input of text information, but the present disclosure is not particularly limited to this, and an input device different from the second information terminal 40C may accept input of text information. The second information terminal 40C may be communicably connected to the other input device, receive text information transmitted by the other input device, and transmit the received text information to the server 10C. Alternatively, an input unit included in the imaging device 30 may accept input of text information. In this case, the input unit included in the imaging device 30 is an example of an input device.

[0236] The server 10C includes a processor 11C, a memory 12C, and a communication unit 13C. The processor 11C is configured, for example, by a central processing unit (CPU). The processor 11C includes an acquisition unit 111C and a display control unit 112C. The acquisition unit 111C and the display control unit 112C may be realized by the processor 11C executing an information processing program, or may be configured by a dedicated hardware circuit such as an ASIC. The information processing program may be recorded on a non-transitory computer-readable recording medium.

[0237] The communication unit 13C is a communication interface that connects the server 10C to the network NT. The communication unit 13C receives first photographing information transmitted by the second information terminal 40C. The communication unit 13C receives text information transmitted by the second information terminal 40C. As a result, the communication unit 13C acquires content information recorded by the second information terminal 40C, i.e., text information input by the second information terminal 40C while the photographing device 30 was photographing multiple images. The communication unit 13C receives instruction signals from the first information terminal 20 that indicate various instructions received from the user. The communication unit 13C transmits display data for displaying various display screens to the first information terminal 20.

[0238] The memory 12C is configured as a non-volatile rewritable storage device such as a hard disk drive or a solid state drive, and includes a design drawing information storage unit 121, a first image information storage unit 122, and a text information storage unit 126.

[0239] The text information storage unit 126 stores text information input by the second information terminal 40C. The text information is associated with meta information including the shooting ID and the input date and time of the text information. The text information is stored each time the above text information is received.

[0240] The acquisition unit 111C acquires multiple shooting dates and times of multiple first images corresponding to the shooting ID from the first image information storage unit 122, and also acquires the input date and time of text information corresponding to the shooting ID from the text information storage unit 126.

[0241] The display control unit 112C displays information indicating that the content information is associated with a first icon, of the multiple first icons, whose shooting time is close to the recording time of the content information, on the blueprint. In the third embodiment, the shooting time is the shooting date and time of the first image, the content information is text information input by the second information terminal 40C while the multiple first images were being captured by the imaging device 30, and the recording time is the input date and time when the text information was input. Furthermore, the information indicating that the text information is associated with a first icon is a second icon different from the multiple first icons. That is, the display control unit 112C displays a second icon indicating that the text information is associated with a first icon, of the multiple first icons, whose shooting date and time is close to the input date and time of the text information, on the blueprint.

[0242] First, the processing of the second information terminal 40C in the fourth embodiment will be described.

[0243] FIG. 22 is a first flowchart showing an example of the processing of the second information terminal 40C in this embodiment 4, and FIG. 23 is a second flowchart showing an example of the processing of the second information terminal 40C in this embodiment 4.

[0244] The processes in steps S131 to S133 are the same as those in steps S11 to S13 shown in FIG. 2, and therefore will not be described further.

[0245] Next, in step S134, the second information terminal 40C determines whether text information has been input. After the start point registration button is pressed, the second information terminal 40C displays a software keyboard and an input completion button for accepting input of text information. The software keyboard and input completion button are displayed on the display screen of the second information terminal 40C. If the photographer wants to record text information while moving and taking pictures using the imaging device 30, the photographer inputs the text information using the software keyboard. For example, when the photographer has finished inputting the text information, the photographer presses the input completion button. This allows the photographer to leave a text comment at the shooting location of the first image.

[0246] If it is determined that text information has been input (YES in step S134), then in step S135, the second information terminal 40C stores the text information. The text information is stored in a memory provided in the second information terminal 40C. The text information includes the input date and time. The input date and time of the text information is the date and time when input of the text information started, the date and time when input of the text information finished, the date and time when the input completion button was pressed, or the date and time when the text information was stored in the memory.

[0247] On the other hand, if it is determined that text information has not been input (NO in step S134), then in step S136, the second information terminal 40C determines whether or not a plurality of first images have been received from the photographing device 30. Here, if it is determined that a plurality of first images have not been received from the photographing device 30 (NO in step S136), the process returns to step S134.

[0248] On the other hand, if it is determined that a plurality of first images have been received from the photographing device 30 (YES in step S136), the process proceeds to step S137.

[0249] The processing in steps S137 to S140 is the same as the processing in steps S18 to S21 in FIGS. 2 and 3, and therefore a description thereof will be omitted.

[0250] Next, in step S141, the second information terminal 40C transmits at least one piece of text information to the server 10C.

[0251] In the fourth embodiment, the second information terminal 40C may accept a selection of text information to be transmitted to the server 10C. The photographer may record multiple pieces of text information while traveling. When multiple pieces of text information are recorded, the photographer may select text information to be transmitted to the server 10C from the multiple pieces of text information recorded while traveling. The second information terminal 40C may transmit at least one selected piece of text information to the server 10C. The selection of text information and transmission of the text information may be performed some time after the processing of step S140 is performed. If no text information is recorded while the photographing device 30 is capturing multiple first images, the processing of step S141 may not be performed.

[0252] Next, the processing of the server 10C in the fourth embodiment will be described.

[0253] FIG. 24 is a first flowchart showing an example of the processing of the server 10C in the fourth embodiment, and FIG. 25 is a second flowchart showing an example of the processing of the server 10C in the fourth embodiment.

[0254] The processing in steps S151 to S153 is the same as the processing in steps S31 to S33 in FIG. 5, and therefore a description thereof will be omitted.

[0255] Next, in step S154, the acquisition unit 111C determines whether the communication unit 13C has received text information.

[0256] If it is determined that the communication unit 13C has not received the text information (NO in step S154), the process proceeds to step S156.

[0257] On the other hand, if it is determined that the communication unit 13C has received the text information (YES in step S154), in step S155, the acquisition unit 111C stores the text information received by the communication unit 13C in the text information storage unit 126. Meta information is associated with the text information. The meta information includes the shooting ID and the input date and time of the text information.

[0258] The processes in steps S156 and S157 are the same as those in steps S36 and S37 in FIG. 5, and therefore will not be described further.

[0259] Next, in step S158, the display control unit 112C determines whether the input date and time of the text information is within the shooting period. If it is determined that the input date and time of the text information is not within the shooting period (NO in step S158), in step S159, the display control unit 112C transmits display data of the design drawing to the first information terminal 20 via the communication unit 13C, thereby displaying the design drawing on the display 23 of the first information terminal 20.

[0260] The process of step S160 is the same as the process of step S40 in FIG. 5, and therefore a description thereof will be omitted.

[0261] On the other hand, if it is determined that the input date and time of the text information falls within the shooting period (YES in step S158), in step S161, the display control unit 112C selects, from among the multiple shooting locations, a shooting location whose shooting date and time is closest to the input date and time of the text information. The display control unit 112C selects a shooting location whose shooting date and time is the same as the input date and time of the text information. If there is no shooting location whose shooting date and time is the same as the input date and time of the text information, the display control unit 112C selects a shooting location whose shooting date and time is closest to the input date and time of the text information.

[0262] The processing in steps S162 to S166 is the same as the processing in steps S42 to S46 in FIG. 6, and therefore a description thereof will be omitted.

[0263] Next, in step S167, the display control unit 112C sends a display instruction to the first information terminal 20 via the communication unit 13C to display a second icon indicating that the text information has been associated with a first icon among the multiple first icons whose shooting date and time is close to the input date and time of the text information in a blank area within the blueprint, thereby displaying the second icon in a blank area within the blueprint.

[0264] FIG. 26 is a diagram showing an example of a design drawing screen 200C displayed on the first information terminal 20 in the fourth embodiment.

[0265] The blueprint screen 200C displays a blueprint 201 of the workspace and a text display field 206. A plurality of first icons 202 are superimposed on the blueprint screen 200C. Furthermore, on the blueprint screen 200C, a second icon 203C indicating that text information has been associated with a first icon 202 whose photographing date and time is close to the input date and time of the text information is displayed in a blank area of ​​the blueprint 201. The second icon 203C is, for example, a rectangular image resembling text. The second icon 203C is, for example, represented by the letter "T."

[0266] The plurality of first icons 202 and the frame 204 in the fourth embodiment are the same as the plurality of first icons 202 and the frame 204 in the first embodiment.

[0267] The display control unit 112C displays the second icon 203C in a location that does not overlap with any characters or figures in the blueprint 201. Note that the display control unit 112C may display the second icon 203C in a blank area within the building in the blueprint 201, a blank area outside the building in the blueprint 201, or a portion outside the area in the blueprint 201 that includes the plurality of first icons 202.

[0268] The second icon 203C is associated with text information input by the second information terminal 40C. The operation unit 24 may accept a user selection of the second icon 203C. When the second icon 203C is selected, the display control unit 112C may read out text information corresponding to the selected second icon 203C from the text information storage unit 126 and display the read text information in the text display field 206. The text display field 206 displays the text information associated with the second icon 203C.

[0269] In addition, when a plurality of pieces of text information are associated with the photography ID, the display control unit 112C may display a plurality of second icons 203C on the design drawing 201, each of which indicates an input point of the plurality of pieces of text information.

[0270] Furthermore, in the fourth embodiment, the display control unit 112C displays the second icon 203C, which is different from the multiple first icons 202. However, the present disclosure is not particularly limited to this. The display control unit 112C may change the display mode of a first icon whose shooting date and time is close to the input date and time of the text information from that of the other first icons. In this case, the display control unit 112C may change the color of a first icon whose shooting date and time is close to the input date and time of the text information from that of the other first icons, or may change the shape of a first icon whose shooting date and time is close to the input date and time of the text information from that of the other first icons.

[0271] In this way, multiple first icons 202 indicating the shooting locations of the multiple first images captured by the imaging device 30 and second icons 203C indicating the input locations of text information input by the second information terminal 40C while capturing the multiple first images are displayed on the design drawing 201. Therefore, the user can know that there are text comments at the shooting locations of the first images. Furthermore, the user can reduce the time it takes to organize the text information input while capturing the multiple first images, thereby improving user convenience.

[0272] (Embodiment 5) In Embodiment 2, audio information recorded by second information terminal 40A is acquired while multiple first images are being captured by imaging device 30. In contrast, in Embodiment 5, it is determined whether the acquired audio information includes speech from multiple people, the audio information is converted into text information, and identifiable text information for each of the multiple people is further displayed.

[0273] The configuration of the information processing system in the fifth embodiment is the same as the configuration of the information processing system 1A in the second embodiment shown in Fig. 8. Therefore, the fifth embodiment will be described using the information processing system 1A in the second embodiment shown in Fig. 8.

[0274] The display control unit 112A analyzes the audio information and determines whether the audio information includes speech from multiple people. The display control unit 112A also converts the audio information into text information using existing voice recognition technology. The display control unit 112A further displays text information that can be identified for each of the multiple people.

[0275] Next, the processing of the server 10A in the fifth embodiment will be described.

[0276] FIG. 27 is a first flowchart showing an example of the processing of the server 10A in the fifth embodiment, and FIG. 28 is a second flowchart showing an example of the processing of the server 10A in the fifth embodiment.

[0277] The processing in steps S171 to S186 is the same as the processing in steps S71 to S86 in FIGS. 13 and 14, and therefore a description thereof will be omitted.

[0278] Next, in step S187, the display control unit 112A analyzes the speech information. The display control unit 112A performs speaker identification on the speech information using speech recognition technology. The display control unit 112A inputs the speech information to a trained machine learning model and acquires speech information classified by speaker from the machine learning model. When the speech information is input, the machine learning model outputs the speech information classified by speaker.

[0279] Next, in step S188, the display control unit 112A determines whether the audio information includes speech from multiple people. If it is determined that the audio information includes speech from multiple people (YES in step S188), in step S189, the display control unit 112A indicates that the audio information is associated with a first icon, among the multiple first icons, whose shooting date and time is close to the recording start date and time of the audio information, and transmits a display instruction to the first information terminal 20 via the communication unit 13A to display second icons representing speech from multiple people in a blank area of ​​the blueprint.

[0280] Next, in step S190, the display control unit 112A converts the voice information for each of the plurality of people into text information.

[0281] Next, in step S191, the display control unit 112A displays text information identifiable for each of the multiple people on the display 23 by sending a display instruction to the first information terminal 20 via the communication unit 13A to display text information identifiable for each of the multiple people.

[0282] On the other hand, if it is determined that the audio information does not include speech from multiple people (NO in step S188), in step S192, the display control unit 112A indicates that the audio information is associated with a first icon among the multiple first icons whose shooting date and time is closest to the recording start date and time of the audio information, and sends a display instruction to the first information terminal 20 via the communication unit 13A to display a second icon indicating speech from one person in a blank area within the blueprint, thereby displaying the second icon indicating speech from one person in a blank area within the blueprint.

[0283] Next, in step S193, the display control unit 112A converts the voice information of one person into text information.

[0284] Next, in step S194, the display control unit 112A displays the text information of one person on the display 23 by sending a display instruction to display the text information of one person to the first information terminal 20 via the communication unit 13A.

[0285] FIG. 29 is a diagram showing an example of a design drawing screen 200D displayed on the first information terminal 20 in the fifth embodiment.

[0286] The blueprint screen 200D displays a blueprint 201 of the workspace and a text display field 209. A plurality of first icons 202 are superimposed on the blueprint screen 200D. Furthermore, on the blueprint screen 200D, second icons 2031D and 2032D indicating that audio information is associated with first icons 202 whose shooting date and time are close to the recording start date and time of the audio information are displayed in a blank area of ​​the blueprint 201. The second icon 2031D is, for example, a rectangular image imitating a plurality of people. The second icon 2032D is, for example, a rectangular image imitating a single person.

[0287] The plurality of first icons 202 and the frame 204 in the fifth embodiment are the same as the plurality of first icons 202 and the frame 204 in the first embodiment.

[0288] The display control unit 112A displays the second icons 2031D and 2032D in locations that do not overlap with characters or figures in the blueprint 201. Note that the display control unit 112A may display the second icons 2031D and 2032D in a blank area within the building in the blueprint 201, a blank area outside the building in the blueprint 201, or a portion outside the area in the blueprint 201 that includes the plurality of first icons 202.

[0289] The second icon 2031D is associated with audio information including speech by multiple people recorded by the second information terminal 40A. The second icon 2032D is associated with audio information including speech by one person recorded by the second information terminal 40A. The operation unit 24 may accept a user selection of the second icon 2031D or 2032D. When the second icon 2031D or 2032D is selected, the display control unit 112A may read out audio information corresponding to the selected second icon 2031D or 2032D from the audio information storage unit 124 and convert the read audio information into text information.

[0290] The display control unit 112A displays the converted text information in the text display field 209. The text display field 209 displays the text information converted from the audio information associated with the second icon 2031D or 2032D.

[0291] 29 , when second icon 2031D is selected, display control unit 112A displays text information identifiable for each of a plurality of people in text display field 209. In Fig. 29 , display area 2091 for text information of the first speaker, display area 2092 for text information of the second speaker, and display area 2093 for text information of the third speaker are displayed in different colors.

[0292] In addition, when a plurality of pieces of audio information are associated with the shooting ID, the display control unit 112A may display a plurality of second icons 2031D, 2032D on the design drawing 201, each of which indicates the recording start point of the plurality of pieces of audio information.

[0293] Furthermore, in the fifth embodiment, the display control unit 112A displays second icons 2031D and 2032D that are different from the multiple first icons 202, but the present disclosure is not particularly limited to this. The display control unit 112A may change the display mode of a first icon whose shooting date and time is close to the recording start date and time of audio information from that of other first icons. In this case, the display control unit 112A may change the color of a first icon whose shooting date and time is close to the recording start date and time of audio information from that of other first icons, or may change the shape of a first icon whose shooting date and time is close to the recording start date and time of audio information from that of other first icons.

[0294] In addition, in the present embodiment 5, as in the embodiment 2, audio information recorded by the second information terminal 40A is acquired while the imaging device 30 is capturing a plurality of first images, but the present disclosure is not particularly limited to this. In the present embodiment 5, as in the embodiment 3, audio information recorded by the imaging device 30B may be acquired while the imaging device 30B is capturing a plurality of first images.

[0295] In addition, in the fifth embodiment, when the second icons 2031D and 2032D are selected, the display control unit 112A converts the audio information corresponding to the second icons 2031D and 2032D into text information and displays the converted text information, but the present disclosure is not particularly limited to this. When the second icons 2031D and 2032D are selected, the display control unit 112A may output the audio information from the speaker without converting the audio information into text information.

[0296] Furthermore, two or more of the first to fifth embodiments may be combined.

[0297] The technology disclosed herein is useful as a technology for displaying images because it can notify the user that there is recorded content information in addition to the image taken at the shooting location, and can reduce the processing steps required to access the content information.

Claims

1. An information processing method executed by a computer, comprising: displaying a bird's-eye view on a display of an information terminal; displaying on the bird's-eye view a plurality of first icons indicating a plurality of shooting locations according to shooting times; acquiring recorded content information; and displaying on the bird's-eye view information indicating that the content information is associated with a first icon among the plurality of first icons whose shooting time is close to the recording time of the content information.

2. The information processing method according to claim 1, wherein the information is a second icon different from the plurality of first icons.

3. The information processing method according to claim 1, wherein the information is the first icon having a display mode different from other first icons.

4. An information processing method according to any one of claims 1 to 3, wherein the plurality of first icons are respectively associated with a plurality of first images taken at the plurality of photographing locations by a first photographing device, and the content information includes a second image taken by a second photographing device while the plurality of first images are being taken by the first photographing device.

5. An information processing method according to any one of claims 1 to 3, wherein the multiple first icons are respectively associated with multiple images taken at the multiple shooting locations by an imaging device, and the content information includes audio information recorded by a recording device while the multiple images were being taken by the imaging device.

6. An information processing method according to any one of claims 1 to 3, wherein the multiple first icons are respectively associated with multiple images taken at the multiple shooting locations by an imaging device, and the content information includes text information input by an input device while the multiple images are being taken by the imaging device.

7. An information processing method according to any one of claims 1 to 3, wherein the multiple first icons are respectively associated with multiple images taken at the multiple shooting locations by an imaging device, and the content information includes text information converted from audio information recorded by a recording device while the multiple images are being taken by the imaging device.

8. The information processing method according to claim 2, wherein said displaying of information includes associating said second icon with said first icon by a frame of a predetermined shape.

9. An information processing method as described in claim 8, wherein the multiple first icons are each associated with a shooting direction of a camera that captured images at the multiple shooting locations, and a point on the outer edge of the first icon identified by the shooting direction associated with the first icon is connected to the frame line.

10. The information processing method according to claim 2, wherein the display of the information includes displaying the second icon in a location that does not overlap with characters or figures in the bird's-eye view.

11. The information processing method of claim 10, wherein the bird's-eye view includes a blueprint of a building, and displaying the information includes displaying the second icon in a margin within the building on the blueprint, in a margin outside the building on the blueprint, or in a portion of the blueprint outside an area including the plurality of first icons.

12. The information processing method of claim 5, further comprising: determining whether the audio information includes speech from multiple people; and converting the audio information into text information; and displaying the information further comprises displaying the text information identifiable for each of the multiple people.

13. An information processing device having a processor, wherein the processor displays a bird's-eye view on a display of an information terminal, displays a plurality of first icons indicating a plurality of shooting locations on the bird's-eye view according to shooting times, acquires recorded content information, and displays information on the bird's-eye view indicating that the content information is associated with a first icon among the plurality of first icons whose shooting time is close to the recording time of the content information.

14. An information processing program that causes a computer to function in such a manner that a bird's-eye view is displayed on an information terminal display, a plurality of first icons indicating a plurality of shooting locations are displayed on the bird's-eye view according to the shooting time, recorded content information is acquired, and information indicating that the content information is associated with a first icon among the plurality of first icons whose shooting time is closest to the recording time of the content information is displayed on the bird's-eye view.