Annotation support device, annotation support method, and annotation support program

Abstract

To efficiently annotate a pixel region where a specific device such as a valve to be operated is photographed.SOLUTION: An annotation support device 10 includes: a storage device 20 that stores a street view image, a moving image, street view image / moving image correspondence information for associating the street view image with the moving image, and correspondence information between each frame image constituting the moving image and pixel region information in which an operation target device is photographed; a subject designation unit 11 that receives designation of annotation information for annotating a pixel region in the street view image; a subject tracking unit 12 that tracks a subject in the pixel region in the moving image; an annotation unit 13 that annotates the annotation information to the pixel region tracked on the plurality of street view images by the subject tracking unit 12; and an output unit 14 that displays the street view image annotated by the annotation unit 13.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an annotation support device, an annotation support method, and an annotation support program that add annotation information of an annotation to a street view image. [Background technology]

[0002] In plants, managers formulate work plans by understanding on-site conditions from data captured by cameras and laser scanners in order to complete construction and inspection work at low cost and in a short period of time.

[0003] For example, when inspecting a pump, isolation work is required to operate a valve to prevent water from flowing into the pump. When formulating an isolation work plan, the work planner refers to the piping and instrumentation diagram to confirm the valves and valve IDs to be operated, and enters them as the equipment to be operated in the isolation work plan.

[0004] To prevent erroneous operation of the valve, the work planner uses a system that links and manages Street View images with the shooting location and displays them on the worker's terminal. In doing so, the planner annotates a marker on the pixel area of ​​the Street View image that includes the valve to be operated. Annotation is the act of adding information such as a marker that indicates the location of the equipment to be operated. However, there are many valves inside a plant, and in order to annotate the valves to be operated, the work planner must manually specify each valve, which requires a lot of work.

[0005] Prior art for automatic annotation of images and videos includes a technology for extracting pixel regions in which a specific device was captured by collating feature amounts calculated from each pixel region in an image or video. For example, Patent Document 1 describes an invention for automatic annotation of pixel regions including human body parts from multiple images in order to improve the efficiency of creating training data for a learning model for medical diagnosis. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2021 / 166094 Summary of the Invention [Problem to be solved by the invention]

[0007] According to the invention described in Patent Document 1, it is possible to efficiently annotate multiple images to any part of the human body by manually annotating only one image showing that part of the human body, automatically calculating the feature amount of the pixel area, and extracting pixel areas with similar feature amounts from other images. However, in annotating the valve to be operated in an isolation work, it is necessary to extract pixel areas including the valve from all street view images, and then manually designate the valve to be operated as the annotation target by visual confirmation, which requires man-hours.

[0008] Therefore, an object of the present invention is to efficiently annotate pixel regions in which specific devices such as valves to be operated are photographed. [Means for solving the problem]

[0009] In order to solve the above-mentioned problems, the annotation support device of the present invention is characterized in that it comprises a subject designation unit that accepts a designation to annotate annotation information in a pixel area in which a subject is photographed in a Street View image, a subject tracking unit that tracks the subject in the pixel area in a video associated with the Street View image, an annotation unit that annotates the annotation information in the pixel areas tracked by the subject tracking unit on a plurality of the Street View images, and an output unit that displays the Street View images annotated by the annotation unit.

[0010] The annotation support method of the present invention is characterized by comprising the steps of: a subject designation unit receiving designation of annotation information for annotating a pixel region for a Street View image stored in a memory unit; a subject tracking unit tracking a subject in the pixel region in a video corresponding to the Street View image; an annotation unit annotating the annotation information for the pixel region tracked by the subject tracking unit on a plurality of the Street View images; and an output unit displaying the Street View image annotated by the annotation unit.

[0011] The annotation assistance program of the present invention causes a computer to execute the steps of: accepting designation of annotation information for annotating pixel regions of a Street View image stored in a memory unit; tracking a subject of the pixel region in a video corresponding to the Street View image; annotating the annotation information for the tracked pixel regions on a plurality of the Street View images; and displaying the Street View images annotated with the annotation information. Other means will be described in the description of the embodiment of the invention. Effect of the Invention

[0012] According to the present invention, it is possible to efficiently annotate pixel regions in which specific equipment, such as a valve to be operated, is photographed. [Brief description of the drawings]

[0013] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of an annotation support device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating a hardware configuration of the annotation support device. [Diagram 3] FIG. 11 is a diagram showing a processing flow of a subject designation unit. [Figure 4] FIG. 13 is a diagram showing an example of designating a pixel region in which an image of a target device is captured; [Diagram 5] FIG. 11 is a diagram illustrating an example of pixel region information. [Figure 6] FIG. 11 is a diagram illustrating an example of pixel region information. [Figure 7] FIG. 2 is a diagram illustrating an example of a street view image database. [Figure 8] FIG. 11 is a diagram showing a processing flow of a subject tracking unit. [Figure 9] FIG. 1 is a diagram showing an example of a street view image / video correspondence database. [Figure 10] FIG. 13 is a diagram showing another example of the street view image / video correspondence database. [Figure 11] FIG. 11 is a diagram illustrating an example of acquiring video information. [Figure 12] FIG. 4 is a diagram illustrating an example of a frame image database. [Figure 13] FIG. 13 is a diagram showing a processing flow of the annotation section. [Figure 14] FIG. 11 is a diagram showing an example of an output screen that displays, via an output device, street view images and annotation information stored in a street view image database. [Figure 15] FIG. 11 is a diagram showing another example of an output screen that displays, via an output device, street view images and annotation information stored in the street view image database. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0015] FIG. 1 is a diagram showing an example of the overall configuration of an annotation support device 10 according to an embodiment of the present invention. 1, annotation support device 10 processes information in a storage device 20 and information input via an input device 30, and outputs the processing results to an output device 40, thereby presenting information that supports planning and implementation of isolation work in a plant. The annotation support device 10 includes a subject designation unit 11, a subject tracking unit 12, an annotation unit 13, and an output unit 14.

[0016] Storage device 20 is, for example, a server equipped with a large-capacity storage device, and includes a street view image database 21, a video database 22, a street view image / video correspondence database 23, and a frame image database 24. Storage device 20 is a storage unit that stores street view images, videos, street view image / video correspondence information that associates these street view images with the videos, and correspondence information between each frame image that constitutes the video and pixel area information in which the target device was photographed.

[0017] The street view image database 21 stores street view images captured by a camera or a laser scanner and annotations. The street view images are either images displayed in association with the capture location or panoramic images.

[0018] Video database 22 stores videos captured by a camera. Street view image / video correspondence database 23 stores information on the correspondence between street view images and videos, such as frame number information and shooting location information. Street view image / video correspondence database 23 records information on the correspondence between street view images and videos based on the shooting time, frame number, or shooting location. The frame number is calculated using at least one of the playback time of a frame image corresponding to a video or a street view image created from this video, and the shooting time of the video.

[0019] The frame image database 24 temporarily stores each frame image obtained by dividing a moving image and pixel area information in which a target device is photographed. In other words, the frame image database 24 stores correspondence information between each frame image constituting a moving image and pixel area information in which a target device is photographed.

[0020] Furthermore, the subject designation unit 11 accepts designation of a pixel area in which a target device is photographed on a certain Street View image, and accepts designation to annotate this pixel area with annotation information. Here, annotation information is at least one of a marker image indicating the position of the target device, the device name, the operating status of the device, text data indicating past inspection records, images, videos, or link information to these pieces of information. The subject designation unit 11 accepts designation of a pixel area in which a target device is photographed on a Street View image, or accepts input of information identifying the target device.

[0021] The subject tracking unit 12 tracks the subject in the pixel area in the video associated with the street view image. The subject tracking unit 12 identifies the video associated with the street view image based on the correspondence between the shooting time, frame number, or shooting position of the street view image and the video. The subject tracking unit 12 calculates the frame number using at least one of the playback time of the video or the frame image corresponding to the street view image created from the video and the shooting time of the video. The subject tracking unit 12 estimates the shooting position in a two-dimensional coordinate system or a three-dimensional coordinate system using self-location estimation and environmental map creation by Lidar, or self-location estimation and environmental map creation by video. Then, the subject tracking unit 12 extracts the feature amount of the pixel area specified by the user, reads the video corresponding to the street view image used to specify the pixel area, and extracts a pixel area having a feature amount that matches the feature amount in each frame image of the video. In the feature amount comparison, the subject tracking unit 12 uses a histogram intersection method or a difference between feature amounts as an evaluation index for the match rate of the feature amount.

[0022] The annotation unit 13 annotates the pixel area tracked by the subject tracking unit 12 with annotation information. The annotation unit 13 links a marker image indicating the position of the device to be operated and text data indicating the device name (annotation information) to the pixel area information, and assigns the combination of the text data and the pixel area information to the street view image as metadata.

[0023] The output unit 14 outputs the street view image annotated by the annotation unit 13 and displays the annotation information added to the street view image. The output unit 14 displays the annotation information on a pixel region associated with the annotation information, or displays the annotation information outside the street view image using a line, an arrow, or a speech bubble from the pixel region associated with the annotation information.

[0024] FIG. 2 is a diagram showing the hardware configuration of the annotation support device 10. As shown in FIG. The annotation support device 10 supports a procedure for adding annotations to a target device that is being operated and recorded in a video. The annotation support device 10 is a computer including a CPU (Central Processing Unit) 71, a ROM (Read Only Memory) 72, and a RAM (Random Access Memory) 73.

[0025] The CPU 71 is a central processing unit that executes a program and accesses data stored in the ROM 72 or the RAM 73 to realize each functional unit. The ROM 72 is a non-volatile read-only memory, and stores various data, a BIOS (Basic I / O System), etc. The RAM 73 is a volatile readable / writable memory, and temporarily stores various data and programs.

[0026] The digitization progress calculation device 1 further includes an interface unit 74 , a communication unit 75 , a display unit 76 , and a storage unit 77 . The interface unit 74 is, for example, a USB (Universal Serial Bus) host controller, which inputs information via the input device 30 and outputs information to the output device 40 .

[0027] The communication unit 75 is, for example, a network interface card, and is a part for communicating with other devices, such as the storage device 20, via a network. The display unit 76 is, for example, a liquid crystal display, and displays characters, figures, images, and the like.

[0028] The storage unit 77 is, for example, a hard disk drive or a solid state drive (SSD) that stores a relatively large amount of data and programs. The storage unit 77 stores an annotation support program 171 and the like. The CPU 71 executes the annotation support program 171 to realize each of the functional units shown in FIG.

[0029] Below, the functions constituting the annotation support device 10 and the information handled by each unit will be described with specific examples.

[0030] 《Subject specification section 11》 First, the subject designation unit 11 displays a certain street view image acquired from the street view image database 21 via the output device 40. The user designates a pixel area in which the device to be annotated is photographed via the input device 30. In this embodiment, the input device 30 is a mouse, but may also be a keyboard or a gaming controller. Also, in this embodiment, the output device 40 is a PC monitor, but may also be a tablet or a VR (Virtual Reality) device.

[0031] Annotation refers to attaching a note to a pixel area, for example, adding a marker indicating the location of a device to be operated or text data describing an inspection record. A Street View system is a system that manages images by linking them to the shooting location and displays them on a terminal according to a user's request. A Street View image is an image that includes shooting location information. Any metadata can be added to a Street View image.

[0032] FIG. 3 is a diagram showing a processing flow of the subject designation unit 11. As shown in FIG. First, the subject designation unit 11 displays one street view image recorded in the street view image database 21 (step 111). Here, the street view image to be displayed may be selected at random, or, for example, the device ID of the device to be operated input via the input device 30 such as a keyboard may be referenced, and a street view image annotated with this device ID may be selected.

[0033] The user determines whether the device to be annotated is included in the street view image displayed in step 111 (step 112). In this embodiment, the determination in step 112 is performed by the user through visual confirmation. In step S112, if the device to be annotated is captured in the street view image (Yes), the process proceeds to step S113. If the device to be annotated is not captured in the street view image (No), the process returns to step S111.

[0034] The user specifies a pixel area in which the target device was photographed in the street view image displayed in step 111, and subject designation unit 11 accepts this designation (step 113). The user specifies the pixel area in which the target device was photographed, for example, by clicking on the street view image displayed on output device 40 via input device 30 such as a mouse. When subject designation unit 11 associates information about the specified pixel area with the street view image and stores it in street view image database 21 (step S114), the processing in FIG. 3 ends.

[0035] FIG. 4 is a diagram showing an example of a pixel region designation in which an image of a target device is captured. A street view image 61 is displayed in window 50. This street view image 61 shows an annotation target region 611 that has been selected by dragging a mouse cursor 51. The mouse cursor 51 is dragged via input device 30 of Figure 1. The user selects a pixel region 612 of the device to be operated as the annotation target region 611. The user specifies the pixel area 612 in which the target device is photographed by dragging the mouse as the input device 30, but the pixel area may also be specified via a keyboard, for example. In addition, in this embodiment, the pixel area 612 in which the target device is photographed is specified using a rectangle, but an ellipse or the outline of the device to be operated may also be used.

[0036] FIG. 5 is a diagram showing an example of pixel region information specified in step 113 of FIG. The pixel region information is the position information of all pixels within a specified region in the street view image 61.

[0037] FIG. 6 is a diagram showing a method for specifying pixel region information. For example, pixel positions are defined using a two-dimensional coordinate system in which the lower left corner of the image is the origin and each coordinate axis is normalized by the maximum value of the angle of view along the coordinate axis. In the embodiment, the position information of all pixels in the region specified in step 113 is defined as pixel region information, but for example, if the specified pixel region is a polygon, the position information of each vertex may be defined as pixel region information.

[0038] FIG. 7 is a diagram showing an example of the street view image database 21. As shown in FIG. The street view image database 21 includes a street view image field, a shooting date and time field, a shooting position field, an annotation information field, and a pixel area information field. The shooting position field further includes an (X, Y) field and a (θ, φ) field. The pixel area information field further includes an x ​​field and a y field.

[0039] In the Street View image database 21, pixel area information indicating the area of ​​the device to be operated that appears in the Street View image, as well as annotation information for this pixel area, such as shooting date and time information, shooting location information, and device information to be operated, are recorded in association with each image file listed in each Street View image column.

[0040] In this embodiment, as an example of annotation information, a marker image indicating that it is a device to be operated is described, but text data, images, and videos indicating the device name, the operating status of the device, and past inspection records may also be used. For the shooting position, one of a one-dimensional coordinate system corresponding to the shooting time, a two-dimensional coordinate system parallel to the ground, and a three-dimensional coordinate system obtained by adding a vertical direction to the two-dimensional coordinate system, and a direction within the image are used. In this embodiment, the (X, Y) column indicates the coordinates of the two-dimensional coordinate system parallel to the ground. The (θ, φ) column indicates the direction within the image.

[0041] "Subject Tracking Unit 12" Next, the subject tracking unit 12 in FIG. 1 will be described. FIG. 8 shows a processing flow of the subject tracking unit 12. First, the subject tracking unit 12 acquires from the street view image database 21 the street view image information and pixel area information used to specify the pixel area in which the target device was photographed in step 113 (step 1201). The street view image information is, for example, the frame number of a video when a street view image is created from a video, or the shooting position of the street view image. The target device is a device that the user intends to operate, and is a device to which annotations, etc., are added to prevent erroneous operation.

[0042] The subject tracking unit 12 calculates the feature amount of the pixel region from the pixels inside the pixel region (step 1202). As the feature amount of the pixel region, for example, a Histograms of Oriented Gradients (HoG) feature amount that expresses the gradient of change in color or brightness of an image, or a Scale-Invariant Feature Transform (SIFT) feature amount that expresses the spatial distribution of the edges of an object is used.

[0043] The subject tracking unit 12 acquires video information including the frame numbers and shooting positions of the street view images from the street view image information acquired in step 1201 via the street view image / video correspondence database 23 (step 1203). Here, the video information refers to a frame number range and a shooting range, for example, the frame numbers and shooting positions at the start and end of shooting. In addition, without being limited to this, the user may set the frame number range and the shooting range, and the subject tracking unit 12 may acquire video information including the frame numbers and shooting positions of the street view images.

[0044] FIG. 9 is a diagram showing an example of the street view image / video correspondence database 23. As shown in FIG. In the Street View image / video correspondence database 23, the image files listed in the Street View image column and the video files listed in the video column are recorded in correspondence with each other using a common frame number listed in the frame number column or a common shooting position listed in the shooting position column.

[0045] FIG. 10 is a diagram showing another example of the street view image / video correspondence database 23. As shown in FIG. In the street view image / video correspondence database 23, the image files listed in the street view image column and the video files listed in the video column are associated and recorded using the common shooting position listed in the shooting position column.

[0046] Correspondence between Street View images and videos based on the shooting position is based on the shooting position in two-dimensional or three-dimensional spatial coordinates estimated using self-location estimation and environmental map creation (Simultaneous Localization and Mapping) using LiDAR (Light Detection And Ranging) or self-location estimation and environmental map creation using video. LiDAR is a technology that uses laser light to measure the distance to an object and the shape of the object based on the information of the reflected light.

[0047] In addition, when there are multiple videos corresponding to the street view image because the same area was shot at different times, the latest video information may be obtained based on, for example, the time when the videos were shot.

[0048] First, in step 113, subject designation unit 11 inputs, as street view image information, the frame number of the street view image used to designate the pixel region in which the target device was photographed. Then, subject tracking unit 12 references street view image / video correspondence database 23, and outputs, as video information, the video file corresponding to the input frame number, together with the start frame number and end frame number of the video (step 1203).

[0049] Next, the subject tracking unit 12 acquires a moving image from the moving image database 22 based on the moving image information acquired in step 1203 (step 1204).

[0050] FIG. 11 is a diagram showing an example of acquiring video information. The top row shows street view image information. The middle row shows the street view image / video correspondence database 23, and explains the operation of identifying a video file corresponding to a frame number input by the subject tracking unit 12. The bottom row shows video information, and information on the video file identified by the street view image / video correspondence database 23 is shown.

[0051] Returning to Fig. 8, the description will be continued. The subject tracking unit 12 divides the moving image acquired from the moving image database 22 into frame images (step 1205). Here, a frame image refers to a still image corresponding to each frame constituting the moving image. The subject tracking unit 12 stores the frame images acquired in this process in the frame image database 24.

[0052] The subject tracking unit 12 determines the street view image used to specify the pixel region where the target device was photographed as a reference image, and selects the frame image having the closest frame number or photographing position from the frame image database 24 (step 1206). The photographing position of the frame image is obtained using Lidar Simultaneous Localization and Mapping or Visual Simultaneous Localization and Mapping.

[0053] The subject tracking unit 12 extracts a pixel area having the same feature amount as the pixel area acquired in step 1201 from the frame image selected in step 1206 by comparing the feature amount (step 1207). Specifically, the subject tracking unit 12 shifts the pixel area to be extracted from the bottom left of the frame image by one pixel at a time in the right and upward directions, and calculates the feature amount of all patterns of pixel areas until it reaches the top right of the frame image. Then, the subject tracking unit 12 compares the feature amount with the feature amount of the pixel area in the reference image in which the target device is photographed, and determines the pixel area having a matching rate of the feature amount higher than a predetermined value as the tracked pixel area. When comparing the feature amount, if the feature amount is expressed by a graph, it is preferable to use the histogram intersection method as an evaluation index of the matching rate of the feature amount. Also, if the feature amount is expressed by a numerical value, it is preferable to use the difference of the feature amount as an evaluation index of the matching rate of the feature amount. In this way, the subject tracking unit 12 tracks the subject (the target device) photographed in the pixel area accepted by the subject designation unit 11 in the frame image during operation.

[0054] The subject tracking unit 12 associates the pixel area information in which the tracked target device is photographed with the frame image selected in step 1206 through the feature matching in step 1207, and stores the correspondence in the frame image database 24 (step 1208).

[0055] FIG. 12 is a diagram showing an example of the frame image database 24. As shown in FIG. The frame image database 24 includes a frame image column, a frame number column, and a tracked pixel area information column. The frame image column is a column for storing information about a frame image. The frame number column is a column for storing a number indicating this frame image. The tracked pixel area information column is a column for storing information about a pixel area on another frame image that has the same feature amount as a pixel area designated in advance by the user on a frame image presented to the user. The subject tracking unit 12 records the frame number and the tracked pixel area information acquired in step 1207 for each frame image file described in the frame image sequence in association with each other.

[0056] Returning to Fig. 8, the explanation will be continued. In step 1209, the subject tracking unit 12 determines whether or not the feature amounts have been compared with all frame images recorded in the frame image database 24. If the feature amounts have been compared with all frame images recorded in the frame image database 24 (Yes), the processing in Fig. 8 ends. If the feature amounts have not been compared with any of the frame images recorded in the frame image database 24 (No), the processing proceeds to step 1210.

[0057] In step 1210, in order to track the target device in the video, the subject tracking unit 12 changes the reference image to the frame image selected in step 1206. Then, the process returns to step 1206, and a frame image having a frame number or shooting position closest to the reference image is selected from the frame image database 24.

[0058] "Annotation section 13" The annotation section 13 in FIG. 1 will be described. FIG. 13 shows a processing flow by the annotation unit 13. First, the annotation unit 13 selects one frame image from the frame image database 24 (step 131).

[0059] Next, in step 132, the annotation unit 13 determines whether or not a frame number corresponding to the frame image selected in step 131 exists in the street view image database 21. In this embodiment, a frame number that exists in common in the street view image database 21 and the frame image database 24 is searched for, but the shooting time or shooting position may also be used.

[0060] If the frame number corresponding to the frame image exists in the Street View image database 21 (Yes), the process proceeds to step 133. If the frame number corresponding to the frame image does not exist in the Street View image database 21 (No), the process returns to step 131 to reselect a frame image.

[0061] Next, the annotation unit 13 extracts, from the frame image database 24, frame numbers and post-tracking pixel area information that exist in common between the street view image database 21 and the frame image database 24 (step 133). The post-tracking pixel area information on the frame image includes an image of the target device.

[0062] The annotation unit 13 retrieves a street view image corresponding to the frame number from the street view image database 21, and annotates the pixel area in which the tracked operation target device is photographed, based on the pixel area information (step 134).

[0063] The annotation unit 13 stores the annotated street view image in the street view image database 21 in association with the annotation (step 135).

[0064] In step 136, the annotation unit 13 determines whether or not the process has been performed on all street view images stored in the frame image database 24. If the process has been performed on all street view images (Yes), the process of Fig. 13 ends. If the process has not been performed on any street view images (No), the process returns to step 131.

[0065] Output section 14 The output unit 14 in FIG. 1 will now be described. FIG. 14 is a diagram showing an example of an output screen on which the street view images stored in the street view image database 21 and annotation information are displayed via the output device 40. As shown in FIG.

[0066] A street view image 62 is displayed in window 53. A target device 621 is photographed in this street view image 62, and a marker 622 and text associated with the marker 622 are shown. The annotation information is displayed at the pixel where the target device was photographed. That is, annotation unit 13 adds a marker image indicating the position of the device to be operated and text data indicating the device name to the street view image as an annotation. Based on this marker 622, the user can easily identify the device to be operated.

[0067] FIG. 15 is a diagram showing an example of an output screen on which the street view images stored in the street view image database 21 and annotation information are displayed via the output device 40. As shown in FIG. A street view image 62 is displayed in the window 53. A target device 621 is photographed in this street view image 62, and annotation information 623 is displayed outside the street view image 62 using a thick line.

[0068] The display location of the annotation information 623 may be outside the street view image 62, and the pixel area may be connected to the annotation information 623 using a line, an arrow, or a speech bubble, and is not limited to this. In this embodiment, markers, solid lines, and text are shown as examples of annotation information, but the information may also be text data indicating the name of the equipment, the operating status of the equipment, or past inspection records, images, videos, or link information to these.

[0069] According to the embodiment of the present invention described above, it is possible to reduce the number of times that pixel regions for annotating annotation information from Street View images captured by all target devices are manually specified, thereby reducing the number of steps required for annotation work. This is achieved by utilizing the Street View image / video database 23 in addition to the Street View image database 21 and video database 22, and it is possible to reduce the number of steps required for annotation work by approximately 80%. Note that annotations in the street view image are not limited to the pixel area in which the target device is photographed, and may be any subject, and are not limited thereto.

[0070] The configuration and effects of the present invention will be described below.

[0071] [1] A subject designation unit (11) that accepts designation of annotating a pixel area in which a subject is captured in a Street View image with annotation information; an object tracking unit (12) that tracks the object in the pixel region in a video associated with the street view image; an annotation unit (13) that annotates the annotation information for pixel regions tracked on the plurality of street view images by the object tracking unit (12); an output unit (14) that displays the street view image annotated by the annotation unit (13); An annotation support device comprising:

[0072] [2] The annotation information is at least one of a marker image indicating the position of the target device, a device name, an operating status of the device, text data indicating past inspection records, an image, and a video. 2. The annotation support device according to claim 1,

[0073] [3] The annotation unit (13) assigns a marker image indicating the position of the device to be operated and text data indicating the device name to the pixel area. 2. The annotation support device according to claim 1,

[0074] [4] The subject tracking unit (12) identifies the video associated with the Street View image based on a correspondence relationship between the shooting time, frame number, or shooting position of the Street View image and the video. 2. The annotation support device according to claim 1,

[0075] [5] The subject tracking unit (12) calculates the frame number using at least one of a playback time of a frame image corresponding to the video or a street view image created from the video and a shooting time of the video. 5. The annotation support device according to claim 4,

[0076] [6] The street view image is either an image displayed in association with the shooting location or a panoramic image. 2. The annotation support device according to claim 1,

[0077] [7] The subject tracking unit (12) uses, as the photographing position, one of a one-dimensional coordinate system corresponding to the photographing time, a two-dimensional coordinate system parallel to the ground, and a three-dimensional coordinate system obtained by adding a vertical direction to the two-dimensional coordinate system, and a direction within the image. 7. The annotation support device according to claim 6,

[0078] [8] The subject tracking unit (12) estimates the shooting position in a two-dimensional coordinate system or a three-dimensional coordinate system by using self-location estimation and environmental map creation by Lidar, or by using self-location estimation and environmental map creation by video. 8. The annotation support device according to claim 7,

[0079] [9] The subject designation unit (11) accepts designation of a pixel area in which the target device is photographed on the street view image, or accepts input of information identifying the target device. 2. The annotation support device according to claim 1,

[0080]

[10] The subject tracking unit (12) extracts features of a pixel area designated by a user, reads a video corresponding to the street view image used to designate the pixel area, and extracts pixel areas having features that have a high degree of coincidence with the features in each frame image of the video. 2. The annotation support device according to claim 1,

[0081]

[11] The subject tracking unit (12) uses a histogram intersection method or a difference between feature amounts as an evaluation index for the matching rate of the feature amounts in the feature comparison. 11. The annotation support device according to claim 10,

[0082]

[12] The output unit (14) displays the street view image and annotation information associated with the street view image. 2. The annotation support device according to claim 1,

[0083]

[13] The output unit (14) displays the annotation information on or from the pixel area where the target device is photographed outside the street view image using a line, an arrow, or a speech bubble. 13. The annotation support device according to claim 12,

[0084]

[14] A step in which a subject designation unit (11) receives designation of annotation information for annotating pixel regions of a street view image stored in a storage unit; A subject tracking unit (12) tracks a subject in the pixel region in a video corresponding to the street view image; Annotating the annotation information by an annotation unit (13) for pixel regions tracked by the object tracking unit (12) on the plurality of street view images; an output unit (14) displaying the street view image annotated by the annotation unit (13); An annotation support method comprising:

[0085]

[15] To the computer, A step of receiving designation of annotation information for annotating pixel regions for a street view image stored in a storage unit (20); tracking a subject in the pixel region in a video corresponding to the street view image; annotating the tracked pixel regions on the plurality of street view images with the annotation information; displaying the street view image annotated with the annotation information; An annotation support program for executing the above.

[0086] <<Variation>> The present invention is not limited to the above-described embodiment, and includes various modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those having all of the configurations described. It is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0087] The above configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware such as an integrated circuit. The above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a recording device such as a memory, a hard disk, or an SSD (Solid State Drive), or a recording medium such as a flash memory card or a DVD (Digital Versatile Disk).

[0088] In each embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it may be considered that almost all components are connected to each other. [Explanation of symbols]

[0089] 10: Annotation support device 11: Subject specification section 12: Subject tracking section 13: Annotation section 14: Output section 20: Storage device 30: Input device 40: Output device 21: Street View Image Database 22:Video Database 23: Street View image / video compatible database 24: Frame image database

Claims

1. A subject designation unit that accepts designation of annotation information to be added to a pixel area in which a subject is captured in a street view image; an object tracking unit that tracks the object in the pixel region in a video associated with the street view image; an annotation unit that annotates the annotation information with respect to pixel regions tracked by the object tracking unit on the plurality of street view images; an output unit that displays the street view image annotated by the annotation unit; An annotation support device comprising:

2. The annotation information is at least one of a marker image indicating the position of the device to be operated, a device name, an operating status of the device, text data indicating past inspection records, an image, and a video. The annotation support device according to claim 1 .

3. the annotation unit assigns a marker image indicating a position of the operation target device and text data indicating a device name to the pixel area in association with the pixel area; The annotation support device according to claim 1 .

4. the subject tracking unit identifies the video associated with the street view image based on a correspondence relationship between the street view image and the video in terms of shooting time, frame number, or shooting position; The annotation support device according to claim 1 .

5. the subject tracking unit calculates the frame number using at least one of a playback time of a frame image corresponding to the video or a street view image created from the video and a shooting time of the video; 5. The annotation support device according to claim 4, wherein the annotation support device comprises:

6. The street view image is either an image displayed in association with a shooting location or a panoramic image. The annotation support device according to claim 1 .

7. the subject tracking unit uses, as the photographing position, any one of a one-dimensional coordinate system corresponding to the photographing time, a two-dimensional coordinate system parallel to the ground, and a three-dimensional coordinate system obtained by adding a vertical direction to the two-dimensional coordinate system, and a direction within the image; The annotation support device according to claim 6 .

8. The subject tracking unit estimates a shooting position in a two-dimensional coordinate system or a three-dimensional coordinate system by using self-location estimation and environmental map creation by Lidar, or by using self-location estimation and environmental map creation by video, 8. The annotation support device according to claim 7,

9. the subject designation unit accepts designation of a pixel area in which the operation target device is photographed on the street view image, or accepts input of information identifying the operation target device; The annotation support device according to claim 1 .

10. the subject tracking unit extracts features of a pixel area designated by a user, reads a video corresponding to the street view image used to designate the pixel area, and extracts pixel areas having features that have a high degree of coincidence with the features in each frame image of the video; The annotation support device according to claim 1 .

11. the subject tracking unit uses a histogram intersection method or a difference between feature amounts as an evaluation index for a match rate of the feature amounts in the feature amount comparison; The annotation support device according to claim 10 .

12. the output unit displays the street view image and annotation information associated with the street view image. The annotation support device according to claim 1 .

13. the output unit displays the annotation information on or from the pixel area where the target device is photographed outside the street view image using a line, an arrow, or a speech bubble. The annotation support device according to claim 12 .

14. A step in which a subject designation unit receives designation of annotation information for annotating a pixel region for a street view image stored in a storage unit; an object tracking unit tracking an object in the pixel region in a video corresponding to the street view image; Annotating the annotation information by an annotation unit for pixel regions tracked by the object tracking unit on the plurality of street view images; an output unit displaying the street view image annotated by the annotation unit; An annotation support method comprising:

15. On the computer, receiving designation of annotation information for annotating pixel regions for the street view image stored in the storage unit; tracking a subject in the pixel region in a video corresponding to the street view image; annotating the tracked pixel regions on the plurality of street view images with the annotation information; displaying the street view image annotated with the annotation information; An annotation support program for executing the above.

Images