Information processing method, information processing device, and information processing program

Abstract

The information processing method is an information processing method in a computer, and includes: displaying a bird's-eye view on a display of a terminal device; displaying a plurality of imaging point icons indicating a series of a plurality of imaging points in the bird's-eye view; and displaying a first moving image associated with a first imaging site indicated by a first imaging site icon when the selection of the first imaging site icon among the plurality of imaging site icons is detected. The first moving image is a moving image between shooting times of each of two second shooting points represented by two second shooting icons displayed before and after the first shooting point icon.

Description

Technical Field

[0001] This disclosure relates to techniques for displaying images captured at multiple shooting locations. Background Technology

[0002] Patent document 1 discloses the following technology: capturing and displaying motion images of a set time before and after a utility pole passing through an image recorded by a dashcam mounted on a vehicle, based on the time the pole passes by.

[0003] In the technology of Patent Document 1, in order to extract moving images from the footage recorded by the dashcam, the user needs to set the time before and after the passing time relative to the reference location. Therefore, even if the user wants to extract moving images between given locations before and after the reference location, it is difficult to correctly set the travel time from the reference location to each given location, resulting in a problem that the moving images cannot be extracted properly.

[0004] Prior art literature

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2023-144527 Summary of the Invention

[0007] This disclosure was made to solve such a problem, and its purpose is to provide a technique that can properly grasp the situation from the selected shooting location to the two shooting locations before and after it.

[0008] -Methods for solving problems-

[0009] One aspect of the information processing method disclosed herein is a computer-based information processing method comprising: displaying a bird's-eye view on a display of a terminal device; displaying, in the bird's-eye view, a plurality of shooting location icons representing a series of multiple shooting locations; and, when a selection of a first shooting location icon among the plurality of shooting location icons is detected, displaying a first motion image associated with the first shooting location represented by the first shooting location icon, the first motion image being a motion image between the respective shooting times of two second shooting locations represented by two second shooting icons displayed before and after the first shooting location icon. Attached Figure Description

[0010] Figure 1 This is a diagram of the overall structure of the information processing system.

[0011] Figure 2 This is an example of a design diagram displayed on the monitor of an information terminal.

[0012] Figure 3This is an example of the screen displayed when the first shooting location icon is selected.

[0013] Figure 4 This is a diagram showing an example of the first motion image.

[0014] Figure 5 This is a flowchart illustrating the first example of the information processing system's processing during the filming process.

[0015] Figure 6 This is a flowchart illustrating an example of peripheral motion image display processing.

[0016] Figure 7 This is a flowchart illustrating the second example of the information processing system's operation during the filming process.

[0017] Figure 8 This is an illustrative diagram illustrating a variation of a method for creating partial moving images corresponding to each shooting location. Detailed Implementation

[0018] (The process of realizing this disclosure)

[0019] The following user interface is being developed: on a two-dimensional design drawing of a construction site, multiple camera location icons are displayed, representing a series of actual photographs taken at various locations within the construction site. When a camera location icon is selected, an image taken at that location is displayed. This allows construction site managers to monitor the construction site's status without physically visiting the site.

[0020] Furthermore, the following function was also studied: in such a user interface, not only are images captured at the selected shooting location displayed, but also motion images up to the two shooting locations before and after that shooting location. Thus, the supervisor can not only properly monitor the selected shooting location, but also properly monitor the construction progress from that shooting location to the two shooting locations before and after it. In the technology of Patent Document 1, it is difficult to achieve the above-mentioned function because it is difficult for the user to accurately set the movement time from the selected shooting location to the two shooting locations before and after it.

[0021] Therefore, the inventors have repeatedly and thoroughly studied the technology that enables proper control of the situation from the selected shooting location to the two shooting locations before and after it, and thus came up with the present disclosure as shown below.

[0022] (1) One aspect of the information processing method of the present disclosure is an information processing method in a computer, comprising: displaying a bird's-eye view on a display of a terminal device; displaying a plurality of shooting location icons representing a series of multiple shooting locations in the bird's-eye view; and when a selection of a first shooting location icon among the plurality of shooting location icons is detected, displaying a first motion image associated with the first shooting location represented by the first shooting location icon, the first motion image being a motion image between the respective shooting times of two second shooting locations represented by two second shooting icons displayed before and after the first shooting location icon.

[0023] According to this structure, motion images between the shooting times of the two second shooting locations before and after the selected first shooting location are displayed. Therefore, it is possible to properly grasp the situation from the selected first shooting location to the two second shooting locations before and after it.

[0024] (2) In the information processing method described in (1) above, the information processing method may also include: when the selection of the first shooting location icon is detected, the display mode of the two second shooting location icons displayed in the bird's-eye view is different from the display mode of other shooting location icons.

[0025] In this structure, when the selection of the first shooting location icon is detected, the display of the two second shooting location icons in the bird's-eye view changes to the display of the other shooting location icons. Therefore, by observing the bird's-eye view, users can easily grasp which two shooting locations are shown and at which time of shooting.

[0026] (3) In the information processing method described in (1) or (2) above, the display of the first motion image may also include: displaying corresponding time information for the shooting time of the currently displayed image contained in the first motion image, the shooting time of the first shooting location, and the shooting times of the two second shooting locations.

[0027] In this case, since the time information is displayed, the user is able to grasp the shooting location of the currently displayed image contained in the first motion image, and its relative positional relationship with the first shooting location and the two second shooting locations.

[0028] (4) In the information processing method described in (3) above, the display of the time information may also include: displaying the shooting time of the first shooting location with an icon that has the same display method as the first shooting location icon displayed in the bird's-eye view; and displaying the shooting time of each of the two second shooting locations with two icons that have the same display method as the two second shooting location icons displayed in the bird's-eye view.

[0029] In this case, the shooting time of the first shooting location, as shown in the bird's-eye view, is displayed using an icon with the same display style as the icon for the first shooting location. Additionally, two icons, each with the same display style as the icons for the two second shooting locations shown in the bird's-eye view, are used to display the shooting time of each of the two second shooting locations. Therefore, the user can intuitively grasp the shooting location of the currently displayed image contained in the first moving image, and its relative positional relationship to the first shooting location and the two second shooting locations.

[0030] (5) In any of the information processing methods described in (1) to (4) above, the information processing method may further include: extracting motion images from the motion images between the start and end points of the shooting at the multiple shooting locations, i.e., the overall motion images, and extracting motion images between the shooting times of the shooting locations before and after each shooting location, i.e., the first part of the motion images; and establishing a correspondence between the first part of the motion images and each shooting location and storing them, and displaying the first motion images includes: displaying the first part of the motion images corresponding to the first shooting location as the first motion images.

[0031] In this structure, when the selection of the first shooting location icon is detected, the first portion of the motion image, which is corresponding to and stored with the first shooting location, is displayed as the first motion image. Therefore, compared to the case where, whenever the selection of the first shooting location icon is detected, the motion images between the shooting times of the two second shooting locations represented by the two second shooting icons displayed before and after the first shooting location icon are extracted from the overall motion image and displayed, the display of the first image when the first shooting location icon is selected can be performed quickly.

[0032] (6) In any of the information processing methods described in (1) to (4) above, the information processing method may further include: extracting a second part of motion image from the motion image between the start and end points of the shooting at the multiple shooting locations, i.e., the overall motion image, wherein the second part of motion image is the motion image from the average of the shooting time of the shooting locations before each shooting location and the shooting time of each shooting location, i.e., the first average time, to the average of the shooting time of the shooting locations after each shooting location, i.e., the second average time; and establishing a correspondence between the second part of motion image and each shooting location and storing it, and displaying the first motion image includes: displaying a motion image obtained by combining the motion image after the shooting time of the third shooting location in the second part of motion image corresponding to the second shooting location before the first shooting location, i.e., the third shooting location, the second part of motion image corresponding to the first shooting location, and the motion image before the shooting time of the fourth shooting location in the second part of motion image corresponding to the second shooting location after the first shooting location, as the first motion image.

[0033] In this case, the motion images from the first averaging time to the second averaging time, i.e., the second part of the motion images, are mapped to and stored for each shooting location. Furthermore, the motion image synthesized using the second part of the motion images mapped to the third, first, and fourth shooting locations is displayed as the first motion image. Therefore, it is possible to store the second part of the motion images corresponding to each shooting location that are necessary to construct the first motion image without duplicating the second part of the motion images corresponding to other shooting locations.

[0034] (7) In any of the information processing methods described in (1) to (4) above, the information processing method may also include: extracting motion images from the start and end points of the shooting at each of the multiple shooting locations, i.e., the overall motion image, and extracting motion images from each shooting location and the shooting point one after each shooting location, i.e., the third part motion image; and, if a shooting image containing a given target is detected in the third part motion image, extracting from the overall motion image a motion image consisting only of images including the shooting image containing the given target, and having a period that repeats the third part motion image, i.e., a target motion image, and non-repeating the period... The motion image obtained by synthesizing the third motion image and the target motion image, namely the fourth motion image, is associated with each shooting location and stored. If a shooting image that does not contain the given target is detected in the third motion image, the third motion image is associated with each shooting location and stored. The first motion image is displayed as the first motion image, which is a motion image obtained by synthesizing the third motion image or the fourth motion image corresponding to the second shooting location one time before the first shooting location, and the third motion image or the fourth motion image corresponding to the first shooting location.

[0035] In this structure, the first moving image is a composite image obtained by non-repeatingly combining a third or fourth moving image corresponding to a second moving image one time prior to the first shooting location, and a third or fourth moving image corresponding to the first shooting location. The fourth moving image is a composite image obtained by non-repeatingly combining the third moving image and the target moving image.

[0036] Therefore, users can not only observe the motion images between the two second shooting locations one before and one after the first shooting location, but also the motion images containing the given target during the period immediately before and / or immediately after the first shooting location. Thus, users can focus on the given target.

[0037] (8) In the information processing method described in (7) above, the detection of whether the captured image contains the given target may include: when the overall motion image is input into the model and one or more shooting times are included in the period of the third part of the motion image, the captured image containing the given target in the third part of the motion image is detected. The model performs machine learning on the relationship between the shooting times of the motion image captured in the space containing the multiple shooting locations, the motion image containing one or more captured images of the annotation target existing in the space (i.e., the second motion image), and the captured images of one or more captured images of the annotation target contained in the second motion image. The annotation target is the target that is specified by the user as the object to be annotated.

[0038] In this scenario, the user can observe not only the motion images of the captured images containing the annotated target during the respective shooting times of the two second shooting locations before and after the first shooting location, but also the motion images of the captured images containing the annotated target during the periods immediately before and / or immediately after the first shooting location. Thus, the user can focus on the target that can be designated as an annotation object.

[0039] (9) In any of the information processing methods described in (1) to (6) above, the shooting location before one of the multiple shooting locations is a shooting location n times earlier than the shooting location, and the shooting location after the shooting location is a shooting location m times later than the shooting location, where n and m are natural numbers greater than or equal to 1.

[0040] In this case, since motion images of the two second shooting locations n before and m after the selected first shooting location are displayed, it is possible to properly grasp the situation from the selected first shooting location to the two second shooting locations n before and m after.

[0041] Furthermore, this disclosure can be implemented not only as an information processing method that performs the characteristic processes described above, but also as an information processing apparatus or the like that having a characteristic structure corresponding to the characteristic processes performed by the information processing method. It can also be implemented as a computer program that causes a computer to execute the characteristic processes included in such an information processing method. Therefore, the following other methods can also achieve the same effect as the information processing method described above.

[0042] (10) The information processing apparatus in another aspect of this disclosure includes a processor that performs the following processing: displaying a bird's-eye view on a display of a terminal device; displaying multiple shooting location icons representing a series of multiple shooting locations in the bird's-eye view; and when a selection of a first shooting location icon among the multiple shooting location icons is detected, displaying a first motion image associated with the first shooting location represented by the first shooting location icon, the first motion image being a motion image between the respective shooting times of two second shooting locations represented by two second shooting icons displayed before and after the first shooting location icon.

[0043] (11) In another aspect of this disclosure, the information processing program causes a computer to perform the following processing: displaying a bird's-eye view on a display of a terminal device; displaying multiple shooting location icons representing a series of multiple shooting locations in the bird's-eye view; and when the selection of a first shooting location icon among the multiple shooting location icons is detected, displaying a first motion image associated with the first shooting location represented by the first shooting location icon, the first motion image being a motion image between the respective shooting times of two second shooting locations represented by two second shooting icons displayed before and after the first shooting location icon.

[0044] This disclosure can also be implemented as an information processing system that operates through such an information processing procedure. Furthermore, it is of course possible to distribute such a computer program via computer-readable, non-transitory recording media such as CD-ROMs or communication networks such as the Internet.

[0045] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Furthermore, the embodiments described below represent specific examples of the present disclosure. The numerical values, shapes, constituent elements, steps, and order of steps shown in the following embodiments are examples and are not intended to limit the present disclosure. Additionally, any constituent elements in the following embodiments that are not described in the independent claim representing the highest-level concept will be described as arbitrary constituent elements. Furthermore, various elements can be combined in all embodiments.

[0046] (Implementation Method)

[0047] Figure 1 This is an overall structural diagram of the information processing system 1. The information processing system 1 is a system that extracts and displays motion images from the initial and final shooting times of a selected shooting location, based on motion images from multiple shooting locations within a series of shooting locations, and displays the motion images from the initial and final shooting times of the selected shooting location.

[0048] like Figure 1As shown, the information processing system 1 includes a server 10, an information terminal 20 (terminal device), a camera 30, and a communication device 40. The server 10 (information processing device and computer), the information terminal 20, and the communication device 40 are communicatively connected to each other via a network NT. An example of a network NT is the Internet.

[0049] Server 10 is, for example, a cloud server consisting of one or more computers. However, this is just one example; server 10 can also be composed of edge servers or installed on information terminal 20. The way server 10 is installed on information terminal 20 is an example of how information terminal 20 is composed of information processing devices.

[0050] Information terminal 20 is held by a user. The user is, for example, the manager of a given space captured by camera device 30. The given space is, for example, a construction site. However, this is just one example; the given space could also be a construction site, factory, shop, office, etc. Information terminal 20 can be, for example, a portable computer such as a smartphone or tablet, or a fixed computer. Information terminal 20 displays various images and screens on a monitor according to display instructions from server 10. Figure 1 The example diagram shows one information terminal 20, but multiple information terminals can be connected to the server 10 via a network NT. The information terminal 20 includes: a communication unit 21, a processor 22, a display 23, and an operation unit 24.

[0051] The communication unit 21 is the communication interface that connects the information terminal 20 to the network NT. The communication unit 21 sends various instructions received from the user by the operation unit 24 to the server 10. The communication unit 21 receives various images and display instructions from the server 10.

[0052] The processor 22, for example, is a central processing unit, which displays various images and screens on the display 23 according to the display instructions of various images and screens received by the communication unit 21.

[0053] The display 23 is composed of various display devices such as liquid crystal displays and organic EL (Electro-Luminescence) displays, and displays various images and screens under the control of the processor 22.

[0054] The operation unit 24 consists of a keyboard, a touch panel, and a mouse, and accepts various instructions input by the user.

[0055] The communication device 40, for example, is a portable information terminal such as a smartphone or tablet computer, which connects the imaging device 30 to the network NT. The communication device 40 and the imaging device 30 are connected via a short-range wireless communication path such as Bluetooth (registered trademark).

[0056] The shooting device 30 is, for example, a wide-angle camera that captures moving images at a given frame rate. A wide-angle camera, also known as a 360-degree camera, is a camera capable of acquiring images from all 360 degrees. The shooting device 30 is, for example, a portable shooting device held by a person. The person being shot is, for example, a construction worker or site supervisor. Alternatively, the shooting device 30 may also be a conventional camera.

[0057] The photographer uses the shooting device 30 to film the construction site and moves within the site. At the starting point of the filming, the photographer aligns the shooting direction of the shooting device 30 with a reference direction (hereinafter referred to as the reference direction) and presses the filming button on the shooting device 30. In the following description, the reference direction is set to north. However, the reference direction is not limited to north and can be other directions. When the photographer reaches the ending point of the filming, they press the filming button again. This ends the filming process using the shooting device 30. Upon completion of the filming, the shooting device 30 transmits the captured moving images to the communication device 40.

[0058] The communication device 40 receives moving images transmitted by the imaging device 30. In this case, the communication device 40 displays a design drawing of the construction site and accepts the photographer's designation of the start and end points of the shooting on the design drawing. The start and end points of the shooting are determined by the photographer entering instructions specifying their locations on the design drawing of the site displayed on the monitor of the communication device 40.

[0059] The design drawing screen includes a two-dimensional coordinate axis. For example, the east-west direction in the design drawing is defined as the horizontal axis or X-axis, and the north-south direction as the vertical axis or Y-axis. Therefore, the start and end points of the shooting are defined using two-dimensional coordinate values. The communication device 40 transmits the shooting information, including the captured motion images, to the server 10 via the network NT.

[0060] Shooting information is generated each time a shooting action is performed. A single shooting action refers to a series of actions taken by the operator holding the shooting device 30 at the construction site, from the start of shooting to the end of shooting. The shooting information includes: motion images captured at the start and end points of a single shooting action, and metadata about these motion images. The metadata includes: shooting ID, shooting start time, shooting end time, design drawing ID, shooting start point, and shooting end point.

[0061] The shooting ID is an identifier used to identify the shooting action. The shooting start time is the shooting time at the starting location. The shooting end time is the shooting time at the ending location. The shooting time can be obtained, for example, by a clock provided with the shooting device 30. Furthermore, the shooting time may also include the year, month, and day on which the shooting took place. Similarly, the various times described later may also include the year, month, and day. The design ID is an identifier used to identify the design drawing of a given space where the moving image was captured. The shooting start and shooting end locations are used to determine the shooting location and shooting direction of each image constituting the moving image.

[0062] Server 10 includes a processor 11, a memory 12, and a communication unit 13. The processor 11 may be, for example, a central processing unit (CPU). The processor 11 includes an instruction receiving unit 111 and a display control unit 112. The instruction receiving unit 111 and the display control unit 112 can be implemented by the processor 11 executing information processing programs, or they may be constructed using dedicated hardware circuits such as ASICs. The information processing programs may also be recorded on a non-transitory computer-readable recording medium.

[0063] Communication unit 13 is the communication interface connecting server 10 to network NT. Communication unit 13 receives shooting information sent by communication device 40. Communication unit 13 receives various instructions from the user from information terminal 20. Communication unit 13 sends various images and display instructions to information terminal 20.

[0064] The memory 12 is composed of a non-volatile, rewritable storage device such as a hard disk drive or a solid-state drive. The memory 12 includes: a design information storage unit 121, an image information storage unit 122, and a moving image information storage unit 123.

[0065] Design drawing information storage unit 121 stores design drawing information. Design drawing information establishes a correspondence between a design drawing ID and the design drawing of a given space identified by that design drawing ID. A design drawing is a drawing representing the design of a construction site; it can be a top view, blueprint, map, or perspective view of the construction site. A design drawing is an example of a bird's-eye view. A bird's-eye view can also be called an aerial view; it can be a view viewed from above or from a high vantage point.

[0066] Image information storage unit 122 stores image information. Image information is information that establishes a correspondence between multiple images contained in the moving images captured by the imaging device 30 and the metadata of each image. The metadata includes: shooting ID, shooting time, design drawing ID, shooting location, and shooting direction. Specifically, the processor 11 creates image information based on the shooting information received by the communication unit 13 and stores the created image information in the image information storage unit 122.

[0067] In detail, the processor 11 obtains the start and end locations of the shooting from the metadata contained in the shooting information received from the communication unit 13. The processor 11 uses Visual SLAM (Simultaneous Localization and Mapping) technology to determine the shooting location of each image based on the start and end locations of the shooting, and multiple images (frames) constituting the moving images contained in the shooting information. The shooting location of each image is represented by two-dimensional coordinate values ​​on the design drawing. Furthermore, the processor 11 also uses Visual SLAM technology to determine the shooting direction of each image. The shooting direction of each image is represented, for example, by a three-dimensional polar coordinate vector with north as the reference direction.

[0068] The imaging device 30 captures moving images at a given frame rate, therefore the number of images constituting the moving images is specified in frames. To store image information, including all the images constituting the captured moving images and the metadata of each image, a large storage capacity is required in the memory 12. Therefore, the processor 11 determines multiple shooting locations for the objects displayed on the design drawing using a given method.

[0069] For example, processor 11 uses Visual SLAM technology to extract images of pre-defined keyframes from multiple images constituting a motion picture and determines the shooting location corresponding to that image. However, the method for determining the shooting location is not limited to this. For example, processor 11 may also divide a design drawing of a given space from which motion pictures have been captured into multiple regions and determine multiple shooting locations such that each of a given number of regions contains one shooting location. Alternatively, processor 11 may also extract images from multiple images constituting a motion picture at time intervals sufficiently long than the frame rate (e.g., 1 second, 5 seconds, 10 seconds, 1 minute, etc.) and determine the shooting location corresponding to that image.

[0070] The processor 11 extracts images (later captured images) taken at each determined shooting location from the motion image contained in the shooting information. The processor 11 obtains metadata from the shooting information, and appends each shooting location and the shooting direction of the captured image at each shooting location to the metadata. The processor 11 creates image information that corresponds to the metadata and the extracted captured images at each shooting location, and stores the created image information in the image information storage unit 122.

[0071] The motion image information storage unit 123 stores the motion image (later, the overall motion image) between the shooting start and shooting end times contained in the shooting information received by the communication unit 13. Specifically, the processor 11 stores the overall motion image contained in the shooting information received by the communication unit 13 in the motion image information storage unit 123.

[0072] Furthermore, the motion image information storage unit 123 establishes corresponding storage of partial motion images with each shooting location of the object displayed on the design drawing. A partial motion image is a motion image that includes multiple shooting images of each shooting location and its surroundings. Specifically, the processor 11 extracts motion images (first partial motion images) from the overall motion image, specifically from the shooting times of two shooting locations one before and one after each shooting location, and stores these partial motion images in the motion image information storage unit 123, establishing a correspondence between each shooting location.

[0073] The instruction receiving unit 111 acquires instructions input by the user at the information terminal 20. Specifically, when the communication unit 13 receives an instruction from the user at the information terminal 20, the instruction receiving unit 111 detects the input of that instruction. Instructions may include, for example, instructions for selecting a design drawing, specifying a shooting time, selecting a shooting location icon, displaying a selected image, displaying surrounding moving images, and providing annotations for the target.

[0074] The design selection indicator tells you to choose the design corresponding to the user-inputted design ID as the processing object. The shooting time specification indicator tells you to specify the shooting time entered by the user as the shooting time of the processing object. The shooting location icon refers to the icon that overlaps with the shooting location contained in the design drawing. The shooting location icon selection indicator tells you to select the shooting location icon from the multiple shooting location icons that overlap with the design drawing as the shooting location icon for processing.

[0075] The "selected image display indicator" refers to the indicator that displays the captured image of the shooting location indicated by the icon representing the selected shooting location as the processing object. The "surrounding motion image display indicator" refers to the indicator that displays the motion image between the shooting times of the two shooting locations before and after the selected shooting location indicated by the icon representing the processing object. The "target annotation indicator" refers to the indicator that displays annotations for targets contained in the captured images specified by the user.

[0076] When the instruction receiving unit 111 detects the input of a design drawing selection instruction, the display control unit 112 reads the design drawing selected by the selection instruction from the design drawing information storage unit 121. The display control unit 112 then displays the read design drawing on the display 23 of the information terminal 20.

[0077] When the instruction receiving unit 111 detects the input of a specified instruction for the shooting time, the display control unit 112 displays multiple shooting location icons, representing a series of multiple shooting locations including the shooting location where the shooting was taken at the shooting time specified by the specified instruction, overlapping on the design drawing.

[0078] Figure 2 This is a diagram showing an example of a design drawing 200 displayed on the display 23 of the information terminal 20. Figure 2 An example is shown in design drawing 200 where a series of N shooting location icons 210 are superimposed. In this example, the shooting location icons 210 consist of white circular images. Shooting location icon 210 (1) shows the initial (first) shooting location in a series of N shooting locations. Shooting location icon 210 (N) shows the final (Nth) shooting location in a series of N shooting locations.

[0079] Here, it is assumed that the user selects a shooting location icon 210 representing one shooting location from among multiple shooting location icons 210 displayed overlapping with the design drawing 200, thereby instructing the receiving unit 111 to detect the input of the selection instruction for the shooting location icon 210. Subsequently, the shooting location icon 210 selected by the user is recorded as the first shooting location icon 210(x). The shooting location represented by the first shooting location icon 210(x) is recorded as the first shooting location x.

[0080] Figure 3 This diagram shows an example of the screen displayed when the first shooting location icon 210(x) is selected. In this case, the display control unit 112 changes the display mode of the first shooting location icon 210(x). Figure 3 An example is shown where the color of the first shooting location icon 210(x) is changed from white to black. However, the method of changing the display of the first shooting location icon 210(x) is not limited to this. For example, the shape of the first shooting location icon 210(x) can be changed, or the filled-in pattern can be changed to a grid or dots, etc.

[0081] Similarly, the display control unit 112 makes the display method of the shooting location icon 210(x-1), which represents the shooting location x one before the first shooting location x, different from the other shooting location icons 210. The display control unit 112 makes the display method of the shooting location icon 210(x+1), which represents the shooting location x one after the first shooting location x, different from the other shooting location icons 210.

[0082] From now on, the shooting location icon 210(x-1) representing the shooting location x-1 one before the first shooting location x will be recorded as the second shooting location icon 210(x-1), and the shooting location icon 210(x+1) representing the shooting location x+1 one after the first shooting location x will be recorded as the second shooting location icon 210(x+1). In addition, the shooting location x-1 represented by the second shooting location icon 210(x-1) will be recorded as the second shooting location x-1, and the shooting location x+1 represented by the second shooting location icon 210(x+1) will be recorded as the second shooting location x+1.

[0083] Figure 3 An example is shown where the filled-in pattern of the second shooting location icon 210 (x-1) and the second shooting location icon 210 (x+1) is changed to dots. However, the method for changing the display of the second shooting location icon 210 (x-1) and the second shooting location icon 210 (x+1) is not limited to this. For example, the shape of the second shooting location icon 210 (x-1) and the second shooting location icon 210 (x+1) can also be changed, as can the filled-in color.

[0084] Furthermore, the display control unit 112 displays the image display screen 300 adjacent to the design drawing 200. However, it is not limited to this; the display control unit 112 may also display the image display screen 300 at a position on the display 23 of the information terminal 20 that is away from the design drawing 200. Alternatively, the display control unit 112 may also display the image display screen 300 overlapping with a portion of the design drawing 200.

[0085] The display control unit 112 retrieves the image captured at the first shooting location x from the image information storage unit 122 and displays a scaled-down image 310 of the image and a motion image display button 320 on the image display screen 300.

[0086] Here, when the user clicks on the thumbnail image 310, the instruction receiving unit 111 detects input of a display instruction for selecting an image. In this case, the display control unit 112 retrieves the image captured at the first shooting location x from the image information storage unit 122 and displays the image on the display 23 of the information terminal 20.

[0087] When the user clicks the motion image display button 320, the receiving unit 111 receives an input indicating that it is detecting the display of surrounding motion images. In this case, the display control unit 112 generates a first motion image associated with the first shooting location x and displays the first motion image on the display 23 of the information terminal 20. The so-called first motion image associated with the first shooting location x refers to a motion image included in the overall motion image as a motion image representing the surrounding conditions of the first shooting location x.

[0088] Specifically, when an input indicating a display of surrounding motion images is detected, the display control unit 112 retrieves a partial motion image corresponding to the first shooting location x from the motion image information storage unit 123. This partial motion image is the motion image between the shooting times of two second shooting locations x-1 and x+1, one before and one after the first shooting location x. Thus, the display control unit 112 generates the retrieved partial motion image as the first motion image associated with the first shooting location x. The display control unit 112 displays the first motion image on the display 23 of the information terminal 20.

[0089] Figure 4 This is a diagram showing an example of a first moving image. Specifically, the display control unit 112 displays a display screen 400 on the display 23 of the information terminal 20. The display screen 400 includes: a display bar 401, a replay button 410, a pause button 420, a stop button 430, a replay time adjustment unit 440, and a volume adjustment unit 450.

[0090] The display control unit 112 displays the first motion picture in the display bar 401. Since the play button 410, pause button 420, stop button 430 and volume adjustment unit 450 have the same structure as those in a typical motion picture playback player, detailed descriptions are omitted.

[0091] The playback time adjustment unit 440 includes a progress bar 441 and a display bar 442 for displaying and changing the playback position of the first moving image. Since the progress bar 441 has the same structure as the progress bar (search bar) of a typical moving image playback player, detailed description is omitted. The display control unit 112 overlays time information on the progress bar 441, establishing corresponding time information for the currently displayed image included in the first moving image, the shooting time of the first shooting location x, and the shooting times of the two second shooting locations x-1 and x+1.

[0092] Specifically, at the left end of the progress bar 441, which indicates the start position of the regeneration of the first motion image, the display control unit 112 overlays an icon 443 (x-1) that has the same display method as the second shooting location icon 210 (x-1) shown in the design drawing 200, as the shooting time of the second shooting location x-1.

[0093] At the right end of the progress bar 441, which indicates the end position of the regeneration of the first motion image, the display control unit 112 overlays an icon 443 (x+1) that has the same display method as the second shooting location icon 210 (x+1) shown in the design drawing 200, as the shooting time of the second shooting location x+1.

[0094] The display control unit 112 displays the image reproduction position captured at the first shooting location x in the progress bar 441, and overlays an icon 443(x) that represents the shooting time at the first shooting location x, displaying the same display method as the first shooting location icon 210(x) shown in the design drawing 200. The reproduction position of the image captured at the first shooting location x in the progress bar 441 refers to the position located to the right of the left end of the progress bar 441, which is the product of the ratio of the elapsed time from the shooting time at the second shooting location x-1 to the shooting time at the first shooting location x, and the length of the progress bar 441 in the long direction. The total reproduction time of the first moving image is the elapsed time from the shooting time at the second shooting location x-1 to the shooting time at the second shooting location x+1.

[0095] The display control unit 112 displays the current regeneration position of the first motion image in the progress bar 441, and displays an icon 444 indicating the shooting time of the currently displayed image included in the first motion image.

[0096] The display control unit 112 displays the current regeneration time of the first motion image in the display bar 442. Figure 4 In the example, "00:02") and the total regeneration time of the first moving image (in Figure 4 In the example, it is "00:06".

[0097] Therefore, by observing the regeneration timing adjustment unit 440, the user can intuitively grasp the shooting location of the currently displayed image contained in the first motion image, and its relative positional relationship with the first shooting location x and the two second shooting locations x-1 and x+1.

[0098] Alternatively, the display control unit 112 may replace the overlapping display of icons 443(x-1), 443(x), and 443(x+1) in the progress bar 441, and display the shooting time of the second shooting location x-1, the shooting time of the first shooting location x, and the shooting time of the second shooting location x+1 in the lower or upper part of the progress bar 441.

[0099] (Processing during filming of action scenes)

[0100] Next, the processing of information processing system 1 based on the photographer's shooting action will be explained. Figure 5 This is a flowchart illustrating the first example of the processing performed by the information processing system 1 during the shooting action.

[0101] In step S1, the operation unit of the communication device 40 accepts input of the shooting start location (hereinafter, the starting point). For example, the photographer inputs the specified starting point on the design drawing screen displayed on the monitor of the communication device 40. The photographer can specify the starting point by tapping on the design drawing screen or by specifying the coordinates of the starting point.

[0102] Next, the photographer, on-site, orients the shooting device 30 northward and inputs a shooting start instruction to the shooting device 30 to perform a shooting action in the given space. Thus, in step S2, the shooting device 30 acquires motion images, i.e., overall motion images, of the shooting times at the start and end points of the shooting action.

[0103] Next, in step S3, the operation unit of the communication device 40 accepts input of the shooting end location (later, the endpoint). For example, the photographer inputs the specified endpoint on the design drawing screen displayed on the monitor of the communication device 40. The photographer can specify the endpoint by tapping on the design drawing screen, or by inputting the coordinates of the endpoint.

[0104] Next, in step S4, the communication device 40 obtains shooting information containing the overall motion image captured by the shooting device 30 from the shooting device 30, and uploads (sends) the obtained shooting information to the server 10.

[0105] Next, in step S5, the display control unit 112 uses its own position estimation processing to calculate the shooting locations of each of the multiple images constituting the overall motion image included in the shooting information obtained in step S4. As the own position estimation processing, VSLAM (Visual Simultaneous Localization and Mapping) can be used.

[0106] Next, in step S6, the display control unit 112 determines multiple shooting locations of the object displayed on the design drawing using a given method. For example, the display control unit 112 uses Visual SLAM technology to extract images of pre-set keyframes from multiple images constituting the overall motion image and determines the shooting location corresponding to those images.

[0107] Next, in step S7, the display control unit 112 creates a list of shooting times for each shooting location determined in step S6.

[0108] Next, in step S8, the display control unit 112 sets the multiple shooting locations determined in step S6 as processing objects and begins processing the shooting locations of the processing objects. Here, the case with N (N is an integer of 2 or more) shooting locations will be described. The display control unit 112 sets the shooting locations of the processing objects in the order of shooting time. j is the index of the shooting location of the specified processing object, and j = 1 to N.

[0109] Next, in step S9, the display control unit 112 generates a partial motion image corresponding to the shooting location j, establishes a correspondence between the partial motion image and the shooting location j, and stores it in the motion image information storage unit 123. Specifically, the display control unit 112 refers to the list of shooting times created in step S7 and performs the following processing.

[0110] The display control unit 112 extracts motion images from the overall motion image contained in the shooting information obtained in step S4, specifically from the shooting locations j-1 one before shooting location j and j+1 one after shooting location j, at the respective shooting times, as partial motion images corresponding to shooting location j. The display control unit 112 establishes a correspondence between the extracted partial motion images and shooting location j and stores them in the motion image information storage unit 123.

[0111] Next, in step S10, the display control unit 112 determines whether the processing of all shooting locations j determined in step S6 has ended. If the processing of all shooting locations j has not ended, the processing returns to step S8, and the next shooting location is set as the shooting location j to be processed. On the other hand, if the processing of all shooting locations j has ended, that is, if the processing of shooting locations j (= N) has ended, the processing ends.

[0112] (Peripheral motion image display and processing)

[0113] Next, the processing of the surrounding motion image of the shooting location, represented by the icon of the shooting location selected by the user (hereinafter referred to as the surrounding motion image display processing) will be explained. Figure 6This is a flowchart illustrating an example of peripheral motion image display processing. The processing begins when the receiving unit 111 detects input of a selection instruction from the design drawing.

[0114] In step S21, the display control unit 112 displays the design drawing 200 selected by the selection instruction received by the instruction receiving unit 111 on the display 23 of the information terminal 20.

[0115] Next, in step S22, the instruction receiving unit 111 detects whether a specified instruction for the shooting time has been input. If the instruction receiving unit 111 does not detect the input of a specified instruction for the shooting time (in step S22, no), the process of step S22 is repeated. If the instruction receiving unit 111 detects the input of a specified instruction for the shooting time (in step S22, yes), the process proceeds to step S23.

[0116] In step S23, the display control unit 112 displays multiple shooting location icons 210, which represent a series of multiple shooting locations including the shooting locations that were shot at the shooting time specified by the specified instruction detected by the instruction receiving unit 111, superimposed on the design drawing 200 shown in step S21.

[0117] Next, in step S24, the receiving unit 111 is instructed to detect whether a selection instruction for the shooting location icon 210 has been input. If the receiving unit 111 does not detect the input of the selection instruction for the shooting location icon 210 (No in step S24), the process of step S24 is repeated. If the receiving unit 111 detects the input of the selection instruction for the shooting location icon 210 (Yes in step S24), the process proceeds to step S25.

[0118] Next, in step S25, the display control unit 112 displays the image display screen 300, which includes the thumbnail image 310 and the motion image display button 320, on the display screen 23 of the information terminal 20.

[0119] At this time, the display control unit 112 further changes the display mode of the first shooting location icon 210(x), the second shooting location icon 210(x-1) representing the second shooting location x one step before the first shooting location x represented by the first shooting location icon 210(x), and the second shooting location icon 210(x+1) representing the second shooting location x one step after the first shooting location x to a display mode different from that of other shooting location icons 210.

[0120] Next, in step S26, the instruction receiving unit 111 detects whether a display instruction for a peripheral motion image has been input. If the instruction receiving unit 111 does not detect the input of a display instruction for a peripheral motion image (No in step S26), the process proceeds to step S27. If the instruction receiving unit 111 detects the input of a display instruction for a peripheral motion image (Yes in step S26), the process proceeds to step S28.

[0121] Furthermore, the process of changing the display mode of the first shooting location icon 210(x) and the two second shooting location icons 210(x-1) and 210(x+1) in step S25 can also be performed when the input of the display instruction of the instruction receiving unit 111 detects the surrounding motion image (Yes in step S26).

[0122] In step S27, the instruction receiving unit 111 detects whether a selection instruction for the shooting location icon 210 has been input. If the instruction receiving unit 111 does not detect the input of the selection instruction for the shooting location icon 210 (No in step S27), the process returns to step S26. If the instruction receiving unit 111 detects the input of the selection instruction for the shooting location icon 210 (Yes in step S27), the process returns to step S25.

[0123] In step S28, the display control unit 112 generates a first motion image associated with the first shooting location x represented by the first shooting location icon 210(x) detected by the selection indication in step S24 or step S27.

[0124] Next, in step S29, the display control unit 112 displays the display screen 400 on the display 23 of the information terminal 20, and displays the first motion image generated in step S28 in the display bar 401 of the display screen 400. After step S29, the process returns to step S24.

[0125] Thus, in the above embodiment, the motion image between the shooting times of the two second shooting locations x-1 and x+1, one before and one after the first shooting location x selected by the user, i.e., the first motion image, is displayed on the display 23 of the information terminal 20. Therefore, the user can properly grasp the situation from the selected first shooting location to the two second shooting locations before and after it.

[0126] Furthermore, when the selection of the first shooting location icon 210(x) is detected, the display method of the two second shooting location icons 210(x-1) and 210(x+1) shown in the bird's-eye view becomes different from the display method of the other shooting location icons. Therefore, by observing the design drawing 200, the user can easily understand which two shooting locations' motion images are displayed on the display screen 400 at their respective shooting times.

[0127] The following variations are possible with respect to this disclosure.

[0128] (1) In the above-described embodiments, the following example is illustrated: in step S9 ( Figure 5 In the overall motion image, the display control unit 112 extracts the motion images between the shooting times of the shooting locations j-1 one before the shooting location j and j+1 one after the shooting location j, respectively, as partial motion images corresponding to the shooting location j, and stores them in the motion image information storage unit 123.

[0129] However, alternatively, the display control unit 112 can extract a second portion of the motion image from the overall motion image, from the average of the shooting times of each shooting location j-1 before each shooting location j and each shooting location j, i.e., the first average time, to the average of the shooting times of each shooting location j and each shooting location j+1 after each shooting location j, i.e., the second average time. Furthermore, the display control unit 112 can also establish a correspondence between this second portion of the motion image and each shooting location j and store it in the motion image information storage unit 123.

[0130] Correspondingly, in step S28 ( Figure 6 In this configuration, the display control unit 112 can acquire not only the partial motion image corresponding to the first shooting location x from the motion image information storage unit 123, but also the partial motion images corresponding to two second shooting locations x-1 and x+1, one before and one after the first shooting location x. Furthermore, the display control unit 112 can use the acquired three partial motion images to generate a first motion image associated with the first shooting location x.

[0131] Specifically, the display control unit 112 extracts motion images from the portion of motion images corresponding to the second shooting location x-1 (the third shooting location) one hour before the first shooting location x, and uses these motion images after the shooting time of the second shooting location x-1 as a first segmented motion image. Additionally, the display control unit 112 extracts motion images from the portion of motion images corresponding to the second shooting location x+1 (the fourth shooting location) one hour after the first shooting location x, and uses these motion images before the shooting time of the second shooting location x+1 as a second segmented motion image. Furthermore, the display control unit 112 generates a first motion image by combining the first segmented motion image, the portion of motion images corresponding to the first shooting location x, and the second segmented motion image.

[0132] In this case, the motion images from the first averaging time to the second averaging time are stored in the motion image information storage unit 123 as partial motion images corresponding to each shooting location j, which are necessary to construct the first motion image. Therefore, it is possible to avoid the repetition of a part or the whole of the partial motion images corresponding to each shooting location j between partial motion images corresponding to other shooting locations. As a result, it is possible to prevent the storage capacity of the memory 12 from becoming too large.

[0133] (2) Alternatively, unlike the above-described embodiment and its variation (1), as shown below, the display control unit 112 extracts partial motion images corresponding to each shooting location from the overall motion image and detects whether the partial motion image contains a captured image of a given target. Furthermore, if the display control unit 112 detects that the partial motion image contains a captured image of a given target, it may extend the partial motion image extracted from the overall motion image corresponding to each shooting location, so that the captured image of the given target is displayed continuously. This structure can be implemented, for example, as follows.

[0134] Figure 7 This is a flowchart illustrating a second example of the processing performed by information processing system 1 during the filming action. Specifically, Figure 5 The processing shown is as follows Figure 7 The deformation is as shown. In detail, after step S7, the process proceeds to step S11. In step S11, the display control unit 112 uses a learned model constructed by machine learning to detect the capture time (later, detection time) of one or more captured images, including captured images of annotated targets contained in the overall motion image, and creates a list of the detected one or more detection times.

[0135] In detail, the memory 12 pre-stores the following model: a second moving image, which is a moving image captured in a space containing multiple shooting locations and including one or more images containing annotation targets existing in that space, is subjected to machine learning to determine the relationship between the shooting times of each of the more than one images containing annotation targets in the second moving image. An annotation target refers to a target contained in the captured image stored in the image information storage unit 122, which is an object designated by the user as an annotation object by the annotation indication obtained by the receiving unit 111.

[0136] The display control unit 112 acquires (detects) one or more shooting moments obtained by inputting the overall motion image contained in the shooting information acquired in step S4 into the model stored in the memory 12, and uses these moments as detection moments for each of the shooting images containing the annotation target. The display control unit 112 creates a list of the detection moments for each of these one or more shooting images.

[0137] Subsequently, in step S9a, which is a variation of step S9, the display control unit 112 takes into account the list of detection times created in step S11, generates partial motion images corresponding to each shooting location j, establishes a correspondence between the partial motion images and each shooting location j, and stores them in the motion image information storage unit 123. Specifically, similar to step S9, the display control unit 112 refers to the list of shooting times created in step S7 (hereinafter, the first list), and further refers to the list of detection times created in step S11 (hereinafter, the second list), and performs the following processing.

[0138] Figure 8 This is an illustrative diagram illustrating a variation of a method for creating partial moving images corresponding to each shooting location. Figure 8 An example is shown below: When the first list contains the shooting times t1 to t7 for each of the 7 shooting locations, the display control unit 112 takes into account the multiple detection times that are consecutive with a given frame period from time ta to time tb and the multiple detection times that are consecutive with a given frame period from time tc to time td contained in the second list, and creates partial motion images corresponding to the 1st to 6th shooting locations respectively.

[0139] In step S9a, the display control unit 112 extracts motion images (third part motion images) from the overall motion image for each shooting location j and a subsequent shooting location j+1 at the shooting time of each shooting location j, and uses them as candidate motion images (later, partial candidate motion images) to become partial motion images corresponding to each shooting location j.

[0140] exist Figure 8 In the example, the display control unit 112 extracts motion images from the overall motion image for period a, from the shooting time t1 at the first shooting location to the shooting time t2 at the second shooting location, as partial candidate motion images corresponding to the first shooting location. Similarly, the display control unit 112 extracts motion images for periods b, c, d, e, and f from the overall motion image as partial candidate motion images corresponding to the second, third, fourth, fifth, and sixth shooting locations.

[0141] Next, if the display control unit 112 detects images containing annotation targets within a portion of the candidate motion images corresponding to each shooting location j, where one or more detection times included in the second list are included within the period of the candidate motion images, then the display control unit 112 extracts motion images consisting only of images containing annotation targets from the overall motion image, and having a period that repeats the period of the candidate motion images, as target motion images corresponding to each shooting location j.

[0142] exist Figure 8 In the example, within period 'a' of the partial candidate motion image corresponding to the first shooting location, there are multiple detection times consecutively from time 'ta' to time 'tb' with a given frame period, as included in the second list. Therefore, the display control unit 112 detects captured images containing annotation targets within this partial candidate motion image. In this case, the motion image consisting only of images containing captured images of annotation targets taken at multiple detection times consecutively from time 'ta' to time 'tb' with a given frame period repeats with the partial candidate motion image corresponding to the first shooting location during the period from time 'ta' to time 'tb'. Therefore, the display control unit 112 extracts the motion image between time 'ta' and time 'tb' from the overall motion image as the target motion image corresponding to the first shooting location.

[0143] Since during period b of the candidate motion image corresponding to the second shooting location, a portion of the detection times included in the second list, from time tc to time td, are consecutive with a given frame period, the display control unit 112 detects captured images containing annotation targets within this portion of the candidate motion image. In this case, the motion image consisting only of images including captured images of annotation targets taken at multiple detection times consecutive with a given frame period from time tc to time td repeats with the candidate motion image corresponding to the second shooting location during the period from time tc to time t3. Therefore, the display control unit 112 extracts the motion image between time tc and time td from the overall motion image as the target motion image corresponding to the second shooting location.

[0144] Since the partial candidate motion image corresponding to the third shooting location includes a portion of multiple detection times consecutively from time tc to time td with a given frame period, which are included in the second list, during period c, the display control unit 112 detects captured images containing annotation targets within this partial candidate motion image. In this case, the motion image consisting only of images including captured images of annotation targets taken at multiple detection times consecutively from time tc to time td with a given frame period repeats with the partial candidate motion image corresponding to the third shooting location during the period from time t3 to time t4. Therefore, the display control unit 112 extracts the motion image between time tc and time td from the overall motion image as the target motion image corresponding to the third shooting location.

[0145] Since the partial candidate motion image corresponding to the fourth shooting location includes a portion of multiple detection times consecutively from time tc to time td within a given frame period, as included in the second list, the display control unit 112 detects captured images containing annotation targets within this partial candidate motion image. In this case, the motion image consisting only of images including captured images of annotation targets captured at multiple detection times consecutively from time tc to time td within a given frame period repeats with the partial candidate motion image corresponding to the fourth shooting location during the period from time t4 to time td. Therefore, the display control unit 112 extracts the motion image between time tc and time td from the overall motion image as the target motion image corresponding to the fourth shooting location.

[0146] Since the detection time included in the second list is not included in the period e and period f of the partial candidate motion images corresponding to the 5th and 6th shooting locations, the display control unit 112 detects the shooting images that do not contain the annotation target in the partial candidate motion images.

[0147] Furthermore, when the display control unit 112 detects a captured image containing an annotation target in a subset of candidate motion images corresponding to each shooting location j, it creates a motion image (the fourth part of the motion image) obtained by synthesizing the candidate motion image and the target motion image without repetition during the creation process, and uses this motion image as a subset of the motion image corresponding to each shooting location j. The display control unit 112 establishes a correspondence between this subset of the motion image and each shooting location j and stores it in the motion image information storage unit 123. When the display control unit 112 detects a captured image that does not contain an annotation target in a subset of candidate motion images corresponding to each shooting location j, it establishes a correspondence between the candidate motion image and each shooting location j and stores it.

[0148] exist Figure 8 In the example described above, the display control unit 112 detects images containing the annotated target in the candidate motion images corresponding to the first shooting location. Therefore, the display control unit 112 non-repeatedly synthesizes the candidate motion images corresponding to the first shooting location (i.e., the motion image during the period a from shooting time t1 to shooting time t2) and the target motion image corresponding to the first shooting location (i.e., the motion image during the period ta to shooting time tb). The display control unit 112 creates the motion image during the period a from shooting time t1 to shooting time t2 obtained through this synthesis, as the partial motion image corresponding to the first shooting location, establishes a correspondence between this partial motion image and the first shooting location, and stores it in the motion image information storage unit 123.

[0149] exist Figure 8In the example described above, the display control unit 112 detects images containing the annotated target in the candidate motion images corresponding to the second shooting location. Therefore, the display control unit 112 non-repeatedly synthesizes the candidate motion images corresponding to the second shooting location (i.e., the motion image during the period b from shooting time t2 to shooting time t3) and the target motion image corresponding to the second shooting location (i.e., the motion image during the period from shooting time tc to shooting time td). The display control unit 112 creates a motion image of period b1 from shooting time t2 to shooting time td obtained through this synthesis, and stores this motion image, corresponding to the second shooting location, in the motion image information storage unit 123.

[0150] exist Figure 8 In the example described above, the display control unit 112 detects images containing the annotated target in the candidate motion images corresponding to the third shooting location. Therefore, the display control unit 112 non-repeatedly synthesizes the candidate motion images corresponding to the third shooting location (i.e., the motion image during the period c from shooting time t3 to shooting time t4) and the target motion image corresponding to the third shooting location (i.e., the motion image during the period c from shooting time tc to shooting time td). The display control unit 112 creates a motion image c1 obtained through this synthesis, during the period c1 from shooting time tc to shooting time td, as the candidate motion image corresponding to the third shooting location, and stores this motion image in the motion image information storage unit 123, establishing a correspondence between this motion image and the third shooting location.

[0151] exist Figure 8 In the example described above, the display control unit 112 detects images containing the annotated target in the candidate motion images corresponding to the fourth shooting location. Therefore, the display control unit 112 non-repeatedly synthesizes the candidate motion images corresponding to the fourth shooting location (i.e., the motion image during the period d from shooting time t4 to shooting time t5) and the target motion image corresponding to the fourth shooting location (i.e., the motion image during the period from shooting time tc to shooting time td). The display control unit 112 creates a motion image d1 during the period d1 from shooting time tc to shooting time t5 obtained through this synthesis, and stores this motion image as a partial motion image corresponding to the fourth shooting location in the motion image information storage unit 123, establishing a correspondence between this partial motion image and the fourth shooting location.

[0152] exist Figure 8In the example described above, the display control unit 112 detects images in the candidate motion images corresponding to the fifth shooting location that do not contain the annotation target. Therefore, the display control unit 112 creates a partial candidate motion image corresponding to the fifth shooting location, that is, a motion image of the period e from shooting time t5 to shooting time t6, as a partial motion image corresponding to the fifth shooting location, establishes a correspondence between this partial motion image and the fifth shooting location, and stores it in the motion image information storage unit 123.

[0153] exist Figure 8 In the example described above, the display control unit 112 detects images in the partial candidate motion images corresponding to the sixth shooting location that do not contain the annotation target. Therefore, the display control unit 112 creates a partial candidate motion image corresponding to the sixth shooting location, that is, a motion image of the period f from shooting time t6 to shooting time t7, as a partial motion image corresponding to the sixth shooting location, establishes a correspondence between this partial motion image and the sixth shooting location, and stores it in the motion image information storage unit 123.

[0154] Furthermore, in this variant example, in Figure 6 In step S28 shown, the display control unit 112 generates a first motion image associated with the first shooting location x as follows.

[0155] Specifically, the display control unit 112 generates a first motion image by synthesizing a portion of the motion image corresponding to the second shooting location x-1 one before the first shooting location x, and a portion of the motion image corresponding to the first shooting location x, without repeating them during the period.

[0156] exist Figure 8 In the example, when the first shooting location x is the first shooting location, the display control unit 112 generates the first motion image as a portion of the motion image corresponding to the first shooting location x, that is, the motion image of the period a from shooting time t1 to shooting time t2.

[0157] exist Figure 8 In the example, when the first shooting location x is the second shooting location, the display control unit 112 non-repeatedly combines the motion image corresponding to the first shooting location one time before the first shooting location x (i.e., the motion image during the period a from shooting time t1 to shooting time t2) and the motion image corresponding to the first shooting location x (i.e., the motion image during the period b1 from shooting time t2 to shooting time td). The display control unit 112 generates the motion image during the period a+b1 from shooting time t1 to shooting time td obtained by this synthesis as the first motion image associated with the second shooting location.

[0158] exist Figure 8 In the example, when the first shooting location x is the third shooting location, the display control unit 112 non-repeatedly combines the motion image corresponding to the second shooting location one step before the first shooting location x (i.e., the motion image during the period b1 from shooting time t2 to shooting time td) and the motion image corresponding to the first shooting location x (i.e., the motion image during the period c1 from shooting time tc to shooting time td). The display control unit 112 generates the motion image during the period b1 from shooting time t2 to shooting time td obtained by this synthesis as the first motion image associated with the third shooting location.

[0159] exist Figure 8 In the example, when the first shooting location x is the fourth shooting location, the display control unit 112 non-repeatedly combines the motion image corresponding to the third shooting location one step before the first shooting location x (i.e., the motion image of period c1 from shooting time tc to shooting time td) and the motion image corresponding to the first shooting location x (i.e., the motion image of period d1 from shooting time tc to shooting time t5). The display control unit 112 generates the motion image of period d1 from shooting time tc to shooting time t5 obtained by this synthesis as the first motion image associated with the fourth shooting location.

[0160] exist Figure 8 In the example, when the first shooting location x is the fifth shooting location, the display control unit 112 non-repeatedly combines the motion image corresponding to the fourth shooting location one step before the first shooting location x (i.e., the motion image of period d1 from shooting time tc to shooting time t5) and the motion image corresponding to the first shooting location x (i.e., the motion image of period e from shooting time t5 to shooting time t6). The display control unit 112 generates the motion image of period d1+e from shooting time tc to shooting time t6 obtained by this synthesis as the first motion image associated with the fifth shooting location.

[0161] exist Figure 8 In the example, when the first shooting location x is the sixth shooting location, the display control unit 112 non-repeatedly combines the motion image corresponding to the fifth shooting location one step before the first shooting location x (i.e., the motion image during the period e from shooting time t5 to shooting time t6) and the motion image corresponding to the first shooting location x (i.e., the motion image during the period f from shooting time t6 to shooting time t7). The display control unit 112 generates the motion image during the period e+f from shooting time t5 to shooting time t7 obtained by this synthesis as the first motion image associated with the sixth shooting location.

[0162] According to this variation, not only are motion images, including images of targets that can be designated as annotations, displayed on the display screen 400 during the respective shooting times of the second shooting locations x-1 and x+1 one before and one after the first shooting location x, but motion images, including images of targets that can be designated as annotations, are also displayed on the display screen 400 during the period immediately before and / or immediately after those shooting times. Thus, the user can focus on the target that can be designated as an annotation.

[0163] (3) In the above-described embodiments and in the modified example (1) above, a shooting location n times earlier than a shooting location can be used instead of a shooting location one time earlier than a shooting location. n is a natural number of 2 or more. Similarly, a shooting location m times later than a shooting location can be used instead of a shooting location one time later than a shooting location. m is a natural number of 2 or more.

[0164] Specifically, in the above implementation method, in step S9 ( Figure 5 In the above, the display control unit 112 can also be used as a partial motion image corresponding to the shooting location j, and the motion images between the shooting times of the shooting locations jn n before the shooting location j and the shooting locations j+m m after the shooting location j are extracted from the overall motion image.

[0165] In the above variation (1), in step S9 ( Figure 5 In this process, the display control unit 112 can also calculate the average of the shooting times of each shooting location j n times before each shooting location j and each shooting location j as a first average time. Additionally, the display control unit 112 can also calculate the average of the shooting times of each shooting location j and each shooting location j+m times after each shooting location j as a second average time. Furthermore, the display control unit 112 can also extract the motion image from the first average time to the second average time from the overall motion image as a partial motion image.

[0166] Correspondingly, in step S28 ( Figure 6 In this configuration, the display control unit 112 can acquire not only partial motion images corresponding to the first shooting location x from the motion image information storage unit 123, but also partial motion images corresponding to two second shooting locations xn and x+m n times before and m times after the first shooting location x. Furthermore, the display control unit 112 can use these partial motion images to generate a first motion image associated with the first shooting location x.

[0167] In detail, the display control unit 112 can also extract motion images after the shooting time of the second shooting location xn from the partial motion images corresponding to the second shooting location xn n times before the first shooting location x, as the first segmented motion image. Additionally, the display control unit 112 can also extract motion images before the shooting time of the second shooting location x+m from the partial motion images corresponding to the second shooting location x+m m times after the first shooting location x, as the second segmented motion image. Furthermore, the display control unit 112 can also generate the first motion image by combining the first segmented motion image, the partial motion image corresponding to the first shooting location x, and the second segmented motion image.

[0168] (4) In step S11 of the above-mentioned variation (2) Figure 7 In the display control unit 112, the shooting time of one or more captured images, including the captured image of a given target contained in the overall motion image, can be detected by known image recognition processing.

[0169] Industrial availability

[0170] This disclosure is useful in the technical field of managing the progress of construction at a construction site remotely.

Claims

1. An information processing method, which is an information processing method in a computer, comprising: Display a bird's-eye view on the terminal device's screen; The bird's-eye view displays multiple shooting location icons representing a series of multiple shooting locations; and When the selection of a first shooting location icon among the plurality of shooting location icons is detected, a first moving image associated with the first shooting location represented by the first shooting location icon is displayed. The first motion image is the motion image between the shooting times of the two second shooting locations represented by the two second shooting icons displayed before and after the first shooting location icon.

2. The information processing method according to claim 1, wherein, The information processing method further includes: when the selection of the first shooting location icon is detected, making the display mode of the two second shooting location icons displayed in the bird's-eye view different from the display mode of other shooting location icons.

3. The information processing method according to claim 1 or 2, wherein, The first motion image shown includes: The display establishes corresponding time information for the shooting time of the currently displayed image included in the first motion image, the shooting time of the first shooting location, and the shooting times of each of the two second shooting locations.

4. The information processing method according to claim 3, wherein, The time information displayed includes: The shooting time of the first shooting location is displayed using an icon with the same display method as the icon of the first shooting location shown in the bird's-eye view; and The shooting time of each of the two second shooting locations is displayed using two icons that display in the same way as the icons of the two second shooting locations shown in the bird's-eye view.

5. The information processing method according to claim 1, wherein, The information processing method further includes: From the motion images of the start and end points of each shooting location (i.e., the overall motion image), the motion images of the shooting locations before and after each shooting location (i.e., the first part of the motion image) are extracted; and The first part of the moving images is mapped to each shooting location and stored. Displaying the first moving image includes: displaying the first portion of the moving image corresponding to the first shooting location, as the first moving image.

6. The information processing method according to claim 1, wherein, The information processing method further includes: From the motion images (i.e., the overall motion image) between the start and end points of the shooting at the multiple shooting locations, a second part of the motion image is extracted. This second part of the motion image is the motion image from the average of the shooting times at each shooting location before and after each shooting location (i.e., the first average time) to the average of the shooting times at each shooting location after each shooting location (i.e., the second average time). The second part of the moving images is mapped to each shooting location and stored. Displaying the first moving image includes displaying a moving image obtained by combining a moving image obtained after the shooting time of the third shooting location in the second part of the moving images corresponding to the second shooting location (i.e., the third shooting location before the first shooting location), the second part of the moving images corresponding to the first shooting location, and a moving image obtained before the shooting time of the fourth shooting location in the second part of the moving images corresponding to the second shooting location (i.e., the fourth shooting location after the first shooting location), as the first moving image.

7. The information processing method according to claim 1, wherein, The information processing method includes: extracting motion images from the start and end points of the shooting at each of the multiple shooting locations, i.e., the overall motion image; and extracting motion images from each shooting location and the subsequent shooting locations at their respective shooting times, i.e., the third part of the motion image; and If an image containing a given target is detected in the third part of the moving image, a target moving image is extracted from the overall moving image, consisting only of images including the image containing the given target and having a period that repeats the third part of the moving image. The fourth part of the moving image, obtained by synthesizing the third part of the moving image and the target moving image without repeating the period, is then mapped to and stored with each shooting location. If an image not containing the given target is detected in the third part of the moving image, the third part of the moving image is mapped to and stored with each shooting location. The first motion image shown includes: The first moving image is a composite image obtained by combining the third or fourth moving image corresponding to the second shooting location one time before the first shooting location, and the third or fourth moving image corresponding to the first shooting location, without repetition.

8. The information processing method according to claim 7, wherein, The captured image used to detect whether it contains the given target includes: When the overall motion image is input into the model to obtain one or more shooting times, which are included in the period of the third part of the motion image, the model detects shooting images containing the given target in the third part of the motion image. The model performs machine learning on the relationship between the shooting times of the motion image (i.e., the second motion image) which is a motion image taken in a space containing the multiple shooting locations and which contains one or more shooting images of the annotation target present in the space, and the shooting times of the one or more shooting images of the annotation target contained in the second motion image. The annotation target is the target of the object that the user specifies as the annotation target.

9. The information processing method according to any one of claims 1, 5, and 6, wherein, The shooting locations preceding one of the multiple shooting locations are the shooting locations n times earlier than the given shooting location. The shooting locations following the first shooting location are shooting locations m times later than the first shooting location. The n and m are natural numbers greater than or equal to 1.

10. An information processing apparatus, comprising a processor, The processor performs the following processing: Display a bird's-eye view on the terminal device's screen; The bird's-eye view displays multiple shooting location icons representing a series of multiple shooting locations; and When the selection of a first shooting location icon among the plurality of shooting location icons is detected, a first moving image associated with the first shooting location represented by the first shooting location icon is displayed. The first motion image is the motion image between the shooting times of the two second shooting locations represented by the two second shooting icons displayed before and after the first shooting location icon.

11. An information processing program that causes a computer to perform the following processing: Display a bird's-eye view on the terminal device's screen; The bird's-eye view displays multiple shooting location icons representing a series of multiple shooting locations; and When the selection of a first shooting location icon among the plurality of shooting location icons is detected, a first moving image associated with the first shooting location represented by the first shooting location icon is displayed. The first motion image is the motion image between the shooting times of the two second shooting locations represented by the two second shooting icons displayed before and after the first shooting location icon.

Images