Search support device, search support system, search support method, and program

Abstract

A search assistance device according to one aspect of the present invention comprises: a captured video acquisition unit that acquires a captured video; an object information acquisition unit that acquires object information indicating objects included in the captured video; a setting unit that sets a search target image containing a search target object; a determination unit that uses the search target image or the object information to determine whether or not a portion of a search target candidate is hidden by a peripheral object; and a video creation unit that, when it is determined that a portion of a search target candidate is hidden by a peripheral object, uses an image indicating the peripheral object out of the captured video to create a comparison video to be compared with in order to search for the search target object.

Description

[Technical field]

[0001] The present disclosure relates to a search support device, a search support system, a search support method, and a program. [Background technology]

[0002] Conventionally, there are known techniques for searching for objects or people in remote locations. For example, Patent Document 1 describes a method of photographing a subject using a multi-lens camera and removing unnecessary objects from an image in which characters are partially obstructed by applying a vector median filter. Patent Document 2 describes a method of extracting a target person from an image and using images from multiple points to identify the person even if part of the person's image is missing due to an obstacle. Non-Patent Document 1 describes a method of integrating images from a visible light camera and an invisible light camera to increase the visibility of the target object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-24258 A [Patent Document 2] JP 2018-93423 A [Non-patent literature]

[0004] [Non-Patent Document 1] NEC Corporation, Tokyo Institute of Technology, "NEC and Tokyo Institute of Technology jointly develop "multi-modal image fusion technology" that uses AI to dramatically improve visibility under poor conditions ~ Automatic synthesis of visible and invisible light images ~", [online], June 5, 2017, [Retrieved June 20, 2022], Internet<URL:https: / / www.titech.ac.jp / news / pdf / webtokyotechpr20170605_okutomi.pdf> Summary of the Invention [Problem to be solved by the invention]

[0005] In the above-mentioned technology, images captured by a multi-lens camera, images taken at multiple locations, or multiple types of images are used to search for objects in remote locations, but there are cases where only images captured from a specific direction can be acquired due to restrictions on the number of cameras, etc. In cases where the search target object is hidden by surrounding objects in an image captured from a specific direction, there is a possibility that the search target object cannot be detected.

[0006] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a search support device, a search support system, a search support method, and a program that are capable of detecting a search target object with a high probability even if the search target object is hidden by surrounding objects. [Means for solving the problem]

[0007] A search support device according to a first aspect includes an image capture unit that captures captured video, a setting unit that sets a search target image including a search target object, an object information acquisition unit that acquires object information indicating an object included in the captured video, a determination unit that determines whether or not a part of a search target candidate is hidden by a peripheral object using the search target image or the object information, and an image creation unit that creates a comparison image to be compared in order to search for the search target object using an image of the captured video that indicates the peripheral object when the determination unit determines that a part of the search target candidate is hidden by the peripheral object. a conversion unit that converts the search target image set by the setting unit into a converted image seen from an arbitrary viewpoint, the setting unit acquiring the search target image from another device, the conversion unit converting the search target image into the converted image, and the determination unit detecting an object included in the captured video that matches one of the converted images as the search target candidate by comparing the converted image converted by the conversion unit with an object included in the captured video, determining whether or not a part of the detected search target candidate is hidden by a peripheral object, and searching for the search target object by comparing the comparison video created by the video creation unit with the converted image converted by the conversion unit. . A search support device according to another aspect includes an image capture unit that captures an image, a setting unit that sets a search target image including a search target object, an object information acquisition unit that acquires object information indicating an object included in the image capture unit, a determination unit that determines whether or not a portion of a search target candidate is hidden by a peripheral object using the search target image or the object information, and when the determination unit determines that a portion of the search target candidate is hidden by the peripheral object, an image creation unit that creates a comparison image to be compared to search for the search target object using an image of the image capture unit that indicates the peripheral object, and a conversion unit that converts the search target image set by the setting unit into a converted image viewed from an arbitrary viewpoint, wherein the image creation unit includes a synthesis unit that creates a composite image as the comparison image by synthesizing an image that indicates the peripheral object that is hiding a portion of the search target candidate with the converted image, and a search unit that searches for the search target object by comparing an object included in the image capture unit with the composite image created by the synthesis unit. A search support device according to another aspect includes an image capture unit that captures captured video, a setting unit that sets a search target image including a search target object, an object information acquisition unit that acquires object information indicating an object included in the captured video, a determination unit that determines whether or not a part of a search target candidate is hidden by a peripheral object using the search target image or the object information, and when the determination unit determines that a part of the search target candidate is hidden by a peripheral object, an image creation unit that creates a comparison image to be compared in order to search for the search target object using an image indicating the peripheral object in the captured video, and a conversion unit that converts the search target image set by the setting unit into a converted image viewed from an arbitrary viewpoint, and the setting unit selects an object image included in the captured video based on an operation by a searcher. The system includes a target image acquisition unit that acquires a specified object image as the search target image, and the conversion unit converts the object image acquired by the target image acquisition unit into a plurality of converted images. The determination unit detects an object included in the captured image that matches one of the converted images as the search target candidate by comparing the converted image converted by the conversion unit with an object included in the captured image, and determines whether or not a portion of the detected search target candidate is hidden by a surrounding object. If a portion of the search target candidate is hidden by a surrounding object, the image creation unit transmits the comparison image to a searcher's terminal device, and if a portion of the search target candidate is not hidden by a surrounding object, the image creation unit transmits the converted image that matches the captured image to the searcher's terminal device.

[0008] A search support method according to a second aspect includes: The search support device, A step of setting a search target image including a search target object; The search support device, acquiring a captured image and object information indicating an object included in the captured image; The search support device,determining whether a part of the search target candidate is hidden by a surrounding object by using the search target image or the object information; The search support device, creating a comparison image to be compared in order to search for the search target object using an image showing the peripheral object from among the captured images when it is determined that a part of the search target candidate is hidden by a peripheral object; and a step of converting the set search target image into a converted image viewed from an arbitrary viewpoint by the search support device, in which the search support device acquires the search target image from another device, converts the search target image into the converted image, and detects an object included in the captured video that matches one of the converted images as the search target candidate by comparing the converted converted image with an object included in the captured video, determines whether or not a part of the detected search target candidate is hidden by a surrounding object, and searches for the search target object by comparing the created comparison video with the converted converted image. . A search support method according to another aspect includes the steps of: a search support device setting a search target image including a search target object; a step of the search support device acquiring an image captured by the captured video and object information indicating an object included in the captured video; a step of the search support device determining whether or not a part of a search target candidate is hidden by a peripheral object using the search target image or the object information; a step of the search support device creating a comparison image to be compared in order to search for the search target object using an image of the captured video indicating the peripheral object when it is determined that a part of the search target candidate is hidden by the peripheral object; a step of the search support device converting the set search target image into a converted image viewed from an arbitrary viewpoint; a step of the search support device creating a composite image by combining an image indicating the peripheral object that is hiding a part of the search target candidate with the converted image as the comparison image; and a step of the search support device searching for the search target object by comparing an object included in the captured video with the created composite image. A search support method according to another aspect includes a step of a search support device setting a search target image including a search target object, a step of the search support device acquiring a captured video and object information indicating an object included in the captured video, a step of the search support device determining whether or not a part of a search target candidate is hidden by a peripheral object using the search target image or the object information, a step of the search support device creating a comparison image to be compared in order to search for the search target object using an image of the captured video indicating the peripheral object when it is determined that a part of the search target candidate is hidden by the peripheral object, and a step of the search support device converting the set search target image into a converted image viewed from an arbitrary viewpoint. the search support device acquires an object image designated based on an operation of a searcher from among object images included in the captured video as the search target image, converts the acquired object image into a plurality of transformed images, and detects an object included in the captured video that matches one of the transformed images by comparing the transformed transformed image with an object included in the captured video as the search target candidate, determines whether or not a portion of the detected search target candidate is hidden by a surrounding object, and if a portion of the search target candidate is hidden by a surrounding object, transmits the comparison image to a searcher's terminal device, and if a portion of the search target candidate is not hidden by a surrounding object, transmits the transformed image that matches the captured video to the searcher's terminal device.

[0009] A program according to a third aspect includes the steps of: setting a search target image including a search target object in a computer mounted on a search support device; acquiring a captured video and object information indicating an object included in the captured video; determining whether or not a part of a search target candidate is hidden by a peripheral object using the search target image or the object information; and, when it is determined that a part of the search target candidate is hidden by a peripheral object, creating a comparison video to be compared in order to search for the search target object using an image of the captured video indicating the peripheral object. and converting the set search target image into a converted image seen from an arbitrary viewpoint, acquiring the search target image from another device, converting the search target image into the converted image, detecting an object included in the captured video that matches one of the converted images as the search target candidate by comparing the converted converted image with an object included in the captured video, determining whether or not a part of the detected search target candidate is hidden by a surrounding object, and searching for the search target object by comparing the created comparison video with the converted converted image. . A program according to another aspect causes a computer mounted on a search support device to execute the steps of: setting a search target image including a search target object; acquiring a captured video and object information indicating an object included in the captured video; determining whether or not a part of a search target candidate is hidden by a peripheral object using the search target image or the object information; when it is determined that a part of the search target candidate is hidden by a peripheral object, creating a comparison image to be compared in order to search for the search target object using an image indicating the peripheral object in the captured video; converting the set search target image into a converted image viewed from an arbitrary viewpoint; creating a composite image by combining an image indicating the peripheral object hiding a part of the search target candidate with the converted image as the comparison image; and searching for the search target object by comparing an object included in the captured video with the created composite image. . A program according to another aspect causes a computer mounted on a search support device to execute the steps of: setting a search target image including a search target object; acquiring a captured video and object information indicating an object included in the captured video; determining whether or not a part of a search target candidate is hidden by a peripheral object using the search target image or the object information; when it is determined that a part of the search target candidate is hidden by a peripheral object, creating a comparison image to be compared in order to search for the search target object using an image of the captured video indicating the peripheral object; and converting the set search target image into a converted image viewed from an arbitrary viewpoint, an object image designated based on the operation of the explorer from among the object images included in the captured image is acquired as the search target image, the acquired object image is converted into a plurality of converted images, and an object included in the captured image that matches one of the converted images is detected as the search target candidate by comparing the converted converted image with an object included in the captured image, and it is determined whether or not a portion of the detected search target candidate is hidden by a surrounding object, and if a portion of the search target candidate is hidden by a surrounding object, the comparison image is transmitted to a terminal device for the explorer, and if a portion of the search target candidate is not hidden by a surrounding object, the converted image that matches the captured image is transmitted to the terminal device for the explorer.

[0010] A search support system according to a fourth aspect includes an imaging device that generates an imaged video, an object information generating device that generates object information indicating an object included in the imaged video, a setting unit that sets a search target image including a search target object, a determination unit that uses the search target image or the object information to determine whether or not a part of a search target candidate is hidden by a peripheral object, and an image creation unit that creates a comparison image to be compared in order to search for the search target object using an image of the imaged video that indicates the peripheral object when the determination unit determines that a part of the search target candidate is hidden by the peripheral object. a conversion unit that converts the search target image set by the setting unit into a converted image seen from an arbitrary viewpoint, the setting unit acquiring the search target image from another device, the conversion unit converting the search target image into the converted image, and the determination unit detecting an object included in the captured video that matches one of the converted images as the search target candidate by comparing the converted image converted by the conversion unit with an object included in the captured video, determining whether or not a part of the detected search target candidate is hidden by a peripheral object, and searching for the search target object by comparing the comparison video created by the video creation unit with the converted image converted by the conversion unit. and a search support device. A search support system according to another aspect includes an imaging device that generates an imaged video; an object information generating device that generates object information indicating an object included in the imaged video; a setting unit that sets a search target image including a search target object; a determination unit that uses the search target image or the object information to determine whether or not a portion of a search target candidate is hidden by a peripheral object; an image creation unit that creates a comparison image to be compared to search for the search target object using an image of the imaged video indicating the peripheral object when the determination unit determines that a portion of the search target candidate is hidden by the peripheral object; and a conversion unit that converts the search target image set by the setting unit into a converted image viewed from an arbitrary viewpoint, wherein the image creation unit includes a synthesis unit that creates a composite image as the comparison image by synthesizing an image indicating the peripheral object that is hiding a portion of the search target candidate with the converted image, and the search support device includes a search unit that searches for the search target object by comparing an object included in the imaged video with the composite image created by the synthesis unit. A search support system according to another aspect includes an imaging device that generates an imaged video, an object information generating device that generates object information indicating an object included in the imaged video, a setting unit that sets a search target image including a search target object, a determination unit that uses the search target image or the object information to determine whether or not a part of a search target candidate is hidden by a peripheral object, an image creation unit that creates a comparison image to be compared in order to search for the search target object using an image of the imaged video indicating the peripheral object when the determination unit determines that a part of the search target candidate is hidden by the peripheral object, and a conversion unit that converts the search target image set by the setting unit into a converted image viewed from an arbitrary viewpoint, and the setting unit converts the object information generated by the setting unit into an object image specified based on an operation by a searcher from among object images included in the imaged video. a target image acquisition unit that acquires an image as the search target image, the conversion unit converts the object image acquired by the target image acquisition unit into a plurality of converted images, the determination unit detects an object included in the captured image that matches one of the converted images as the search target candidate by comparing the converted image converted by the conversion unit with an object included in the captured image, and determines whether or not a portion of the detected search target candidate is hidden by a surrounding object, and if a portion of the search target candidate is hidden by a surrounding object, the image creation unit transmits the comparison image to a searcher's terminal device, and if a portion of the search target candidate is not hidden by a surrounding object, the image creation unit transmits the converted image that matches the captured image to the searcher's terminal device. Effect of the Invention

[0011] According to the present disclosure, a search target object can be detected with a high probability even if the search target object is hidden by surrounding objects. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing an example of a search support system according to a first embodiment. [Diagram 2] 1 is a block diagram showing an example of a functional configuration of a search support device according to a first embodiment. [Diagram 3] FIG. 2 is a diagram for explaining a process of a search support device in the first embodiment. [Figure 4] 4 is a flowchart showing an example of a processing procedure in a search support device according to the first embodiment. [Diagram 5] FIG. 13 is a block diagram showing an example of a functional configuration of a search support device according to a second embodiment. [Figure 6] FIG. 11 is a diagram illustrating a process performed by a search support device in a second embodiment. [Figure 7] 13 is a flowchart illustrating an example of a processing procedure in a search support device according to the second embodiment. [Figure 8] FIG. 13 is a block diagram showing an example of a functional configuration of a search support device according to a third embodiment. [Figure 9] FIG. 13 is a diagram illustrating a process performed by a search support device in a third embodiment. [Figure 10] 13 is a flowchart illustrating an example of a processing procedure in a search support device according to the third embodiment. [Figure 11] FIG. 13 is a diagram for explaining a modified example of the search support system. [Figure 12] FIG. 13 is a block diagram showing an example of a functional configuration of a search support device according to a fourth embodiment. [Figure 13]FIG. 13 is a diagram illustrating a process performed by a search support device in a fourth embodiment. [Figure 14] 20 is a flowchart showing an example of a processing procedure in a search support device according to the fourth embodiment. [Figure 15] FIG. 13 is a diagram illustrating a modification of a search support device in the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments of the present disclosure will be described with reference to the drawings. Note that the scope of the present disclosure is not limited to the following embodiments, and can be changed as desired within the scope of the technical ideas of the present disclosure. For example, the images in the embodiments may be replaced with videos, and the videos in the embodiments may be replaced with images. In addition, in the following drawings, the scale and number of each structure may be different from the scale and number of the actual structure in order to make each configuration easier to understand.

[0014] FIG. 1 is a block diagram showing an example of a search support system according to a first embodiment. The search support system includes, for example, a search support device 100, a camera system 200, and a searcher's terminal device 300. The search support device 100 and the searcher's terminal device 300 are installed in, for example, a management room that manages search targets such as lost items in a search area. The camera system 200 is placed in a search area in a remote location away from the search support device 100 and the searcher's terminal device 300. The search support device 100 and the camera system 200 are connected via a network NW such as the Internet. In this embodiment, the search support device 100, the camera system 200, and the searcher's terminal device 300 are shown as separate devices, but are not limited to this. One information processing device may have the functions of the search support device 100 and the functions of the searcher's terminal device 300, and the camera system 200 may have the functions of the search support device 100.

[0015] The search support device 100 performs processing to support the search of a search target object. The search support device 100 may be configured to be connected to, for example, a plurality of camera systems 200 and a plurality of searcher terminal devices 300, and to provide a cloud service that supports the search of the search target object. The search target object may be, for example, a lost item in a store, but is not limited thereto, and may be a person such as a lost child.

[0016] The searcher's terminal device 300 is an information processing device such as a personal computer or a smartphone operated by a searcher who searches for a search target object. The searcher's terminal device 300 includes an operation interface that accepts operations by the searcher, a display interface that presents various information to the searcher, a storage device, and a control unit such as a CPU (Central Processing Unit) that performs various processes.

[0017] The camera system 200 is provided in a search area. The search area may be, for example, a shopping mall, a restaurant, or other store, but is not limited thereto. The camera system 200 includes a plurality of camera devices 210, an object information generating device 216, and a control unit 220. The object information generating device 216 may be, for example, an infrared camera device 212 and a radar device 214. The camera device 210 generates an image captured in the search area. The camera device 210 generates an image captured by irradiating visible light. For example, one or more camera devices 210 are installed in one search area. Each of the camera devices 210 generates an image captured in the search area from its respective installation location. In this embodiment, the camera device 210 generates a time-sequential image, but is not limited thereto, and may generate images that are not time-sequential. In this embodiment, the object information generating device 216 includes, for example, an infrared camera device 212 and a radar device 214, but is not limited thereto, and may have another configuration as long as it can generate object information as described later.

[0018] The object information generating device 216 acquires object information indicating an object included in the captured image. The infrared camera device 212 generates an infrared image as object information by emitting infrared light. The radar device 214 irradiates a search area with a pulse of laser light and measures the reflected light to measure distance information to the object as object information.

[0019] The object information is information indicating a two-dimensional or three-dimensional shape of the search target object. Specifically, the object information is information indicating a two-dimensional or three-dimensional shape indicating the search target object or a peripheral object group consisting of one peripheral object or multiple peripheral objects that hide a part of the search target object. For example, when the object information generating device 216 is an infrared camera device 212, the object information is information indicating a two-dimensional shape. When the object information generating device 216 is a radar device 214, the object information is information indicating a three-dimensional shape acquired by, for example, swinging the transmission direction of a radar transmission signal vertically and horizontally. For example, when the object information generating device 216 is an infrared camera device 212, the object information acquiring unit 108 of the search support device 100 described later acquires object information of the search target by separating the search target from the perspective image and the peripheral objects that hide the search target. When the object information generation device 216 is a radar device 214, the object information acquisition unit 108 of the search support device 100 described later acquires object information of the search object by distinguishing between the search object and surrounding objects that hide the search object based on differences in distance information.

[0020] The control unit 220 controls the camera device 210, the infrared camera device 212, and the radar device 214. The control unit 220 operates the camera device 210, the infrared camera device 212, and the radar device 214 in time synchronization. In this way, the control unit 220 acquires captured images and object information synchronized in time. The control unit 220 transmits the captured images and object information synchronized in time to the search support device 100. The control unit 220 may perform camera control such as changing the imaging direction and zooming the imaging range for the camera device 210, the infrared camera device 212, or the radar device 214 based on the operation of the searcher, for example.

[0021] Fig. 2 is a block diagram showing an example of a functional configuration of searching support device 100 in the first embodiment. Fig. 3 is a diagram for explaining the process of searching support device 100 in the first embodiment.

[0022] The search support device 100 includes, for example, a search area side communication unit 102, a terminal side communication unit 104, a captured image acquisition unit 106, an object information acquisition unit 108, a setting unit 110, a conversion unit 112, a determination unit 114, an image creation unit 116, and a search unit 118. The search area side communication unit 102 is a communication interface such as a NIC (Network Interface Card) or a wireless communication module for connecting to a network NW. The terminal side communication unit 104 is a communication interface for communicating via, for example, a LAN line. Each unit such as the captured image acquisition unit 106, the object information acquisition unit 108, the setting unit 110, the conversion unit 112, the determination unit 114, the image creation unit 116, and the search unit 118 is realized by, for example, a computer such as a CPU mounted on the search support device 100 executing a program stored in a program memory.

[0023] The captured video acquisition unit 106 acquires the captured video P10. The captured video P10 is supplied via the search area side communication unit 102. The captured video acquisition unit 106 stores, for example, search area information indicating the search area, camera information that uniquely identifies the camera device 210, and the captured video in a storage device (not shown) in association with each other.

[0024] The object information acquisition unit 108 acquires object information indicating objects included in the captured video P10. The object information is supplied via the search area side communication unit 102. The object information includes object information of a search target and object information of surrounding objects. The object information acquisition unit 108 stores, for example, search area information indicating the search area, device information that uniquely identifies the infrared camera device 212 or the radar device 214, and the object information in a storage device (not shown) in association with each other. The object information acquisition unit 108 may store the acquired object information in association with the search area information or the device information.

[0025] The setting unit 110 sets a search target image P12 including a search target object input via the terminal side communication unit 104. The search target image P12 is a still image viewed from a specific viewpoint provided by a person (search requester) requesting a search for a search target object such as an image of a lost item or a face image of a lost child. The setting unit 110 may acquire, as the search target image P12, an image uploaded to the search support device 100 from another device such as the smartphone of the search requester.

[0026] The conversion unit 112 converts the search target image P12 set by the setting unit 110 into a viewpoint conversion image P14 viewed from an arbitrary viewpoint. The conversion unit 112 may generate one viewpoint conversion image P14 from the search target image P12, or may generate multiple viewpoint conversion images P14. The conversion unit 112 may, for example, create a viewpoint conversion image P14 in which the viewpoint of the search target is converted by cutting out an area including the search target included in the search target image P12 and performing image processing such as angle correction, rotation, and size change on the area including the search target. The conversion unit 112 may, for example, create a viewpoint conversion image P14 in which the viewpoint of the search target is converted by predicting a portion of the search target image P12 in which the search target does not appear and combining the predicted portion with an area including the search target included in the search target image P12. The conversion unit 112 may, for example, predict the shape, etc. of the search target by searching a search site or the like for an object similar to the search target based on the product name information of the search target.

[0027] The determination unit 114 determines whether or not a part of the search target candidate is hidden by a peripheral object by using the search target image P12 or the object information. The search target candidate may be an object included in the captured video P10. The determination unit 114 may compare the viewpoint converted image P14 obtained by converting the search target image P12 with the object included in the captured video P10 to detect an object included in the captured video P10 that matches one of the viewpoint converted images P14 as a search target candidate, and determine whether or not a part of the detected search target candidate is hidden by a peripheral object. The determination unit 114 may determine whether or not a part of the object included in the captured video P10 is hidden by a peripheral object by determining whether or not a part of the object information is missing. The search target candidate is, for example, an object included in the captured video P10 that matches at least one of the search target image P12 and one or more viewpoint converted images P14.

[0028] When the determination unit 114 determines that a part of the search target candidate is hidden by a peripheral object, the image creation unit 116 creates a comparison image to be compared to search for the search target object using an image showing the peripheral object in the captured image P10. The image creation unit 116 creates, for example, an overall image P16 representing the entire search target candidate as a comparison image. The image creation unit 116 interpolates the part hidden by the peripheral object by using, for example, an infrared image acquired by the infrared camera device 212 to see through the peripheral object. The image creation unit 116 identifies the location of the peripheral object using a distance image acquired by the radar device 214, and interpolates the location of the peripheral object. In this way, the image creation unit 116 creates the overall image P16.

[0029] The search unit 118 searches for the search target object by comparing the entire video P16 created by the video creation unit 116 with the viewpoint conversion image P14 converted by the conversion unit 112. The search unit 118 transmits the search result to the searcher's terminal device 300 via the terminal side communication unit 104. For example, when a search target candidate is found, the search unit 118 may transmit the captured video P10 including the search target candidate and position information of the search target candidate as the search result. This allows the searcher to visually recognize the search target by displaying the captured video P10 and to search for the search target by checking the video of the camera device 210 capturing the position of the search target candidate.

[0030] FIG. 4 is a flowchart showing an example of a processing procedure in the search support device 100 in the first embodiment. First, the terminal side communication unit 104 acquires a search target image P12 from the searcher's terminal device 300, and the setting unit 110 sets the search target image P12 (step S100). Next, the conversion unit 112 creates a viewpoint converted image P14 from the search target image P12 (step S102). Next, the captured image acquisition unit 106 acquires a captured image P10 of the search area from the camera system 200, and the object information acquisition unit 108 acquires object information (step S104). Next, the determination unit 114 determines whether or not an object (search target candidate) that matches the viewpoint converted image P14 among objects included in the captured image P10 has been detected (step S106). Note that the determination unit 114 may determine that a search target candidate that matches the viewpoint converted image P14 has been detected when the search target candidate matches the viewpoint converted image P14 with a degree of matching equal to or higher than a predetermined reference value. That is, the determination unit 114 is not limited to determining whether the entire images match, and may determine whether a part of the images match. The reference value is preferably a value that can determine that an object included in the captured video P10 and the viewpoint converted image P14 are the same object, but may be a value appropriately changed by the searcher. If the determination unit 114 does not detect an object that matches the viewpoint converted image P14 (step S106: NO), the process returns to step S104 and the search continues.

[0031] When the determination unit 114 detects a search target candidate that matches the viewpoint conversion image P14 (step S106: YES), it determines whether or not a part of the search target candidate is hidden by a peripheral object (step S108). When the determination unit 114 determines that a part of the search target candidate is not hidden by a peripheral object (step S108: NO), it notifies the search result (step S116). When the determination unit 114 determines that a part of the search target candidate is hidden by a peripheral object (step S108: YES), the video creation unit 116 cuts out and extracts an image including an image showing a peripheral object (peripheral object image) (step S110), and creates an entire video P16 in which the entire search target candidate can be confirmed using the peripheral object image (step S112). The video creation unit 116 recognizes a part of the image showing the search target candidate that is hidden by the peripheral object based on the peripheral object image, and uses object information to interpolate the part of the image showing the search target candidate that is hidden by the peripheral object.

[0032] Next, the search unit 118 judges whether the viewpoint converted image P14 matches the overall video P16 (step S114). If the search unit 118 does not detect the viewpoint converted image P14 that matches the overall video P16 (step S114: NO), the process returns to step S104. If the search unit 118 detects the viewpoint converted image P14 that matches the overall video P16 (step S114: YES), the search unit 118 notifies the search result (step S116). In step S116, the search unit 118 transmits, for example, an image showing the search target candidate that is the source of the overall video P16, position information of the search target candidate, and the like, to the searcher's terminal device 300 as the search result.

[0033] As described above, according to the search support device 100 of the first embodiment, the captured image P10 and object information are acquired, and it is determined whether or not a part of the search target candidate is hidden by a peripheral object. When it is determined that a part of the search target candidate is hidden by a peripheral object, the whole image P16 representing the whole of the search target candidate can be created using the object information. When the search target candidate is hidden by a peripheral object, there is a high possibility that the true search target cannot be searched for. In addition, when the number of camera devices 210 is limited, there is a high possibility that the true search target cannot be searched for. In contrast, according to the search support device 100 of the first embodiment, when the search target candidate detected based on the comparison between the captured image P10 and the viewpoint conversion image P14 is hidden by a peripheral object, the whole image P16 can be created using the object information. As a result, the search support device 100 can, for example, transmit the whole image P16 to the searcher's terminal device 300 and show the whole image P16 to the searcher to allow the searcher to search whether the search target candidate is the true search target. As a result, according to search support device 100 of the first embodiment, the search target object can be detected with a high probability even if the search target object is hidden by surrounding objects.

[0034] In the first embodiment, the search support system includes an infrared camera device 212 or a radar device 214 to acquire object information, and can create an overall video P16 using the object information acquired by the infrared camera device 212 or the radar device 214. As a result, the search support system and search support device 100 can create an overall video P16 by processing such as interpolating parts of search target candidates that are hidden by surrounding objects.

[0035] According to the first embodiment of the search support device 100, the search support device 100 may convert the search target image P12 into a viewpoint converted image P14 seen from an arbitrary viewpoint, and compare the viewpoint converted image P14 with an object included in the captured video P10 to detect an object included in the captured video P10 that matches one of the viewpoint converted images P14 as a search target candidate. According to the first embodiment of the search support device 100, since the search unit 118 that searches for the search target object by comparing the entire video P16 with the viewpoint converted image P14 is provided, the search target object can be searched for even if it is hidden by surrounding objects.

[0036] The second embodiment will be described below. FIG. 5 is a block diagram showing an example of a functional configuration of a search support device 100B in the second embodiment. FIG. 6 is a diagram for explaining the processing of the search support device 100B in the second embodiment. The search support device 100B in the second embodiment differs from the first embodiment in that it creates a composite image P20 by combining an image of a surrounding object that hides a part of a search target candidate with a viewpoint conversion image P14, and searches for a search target object by comparing the created composite image P20 with an object included in a captured video P10. The following mainly describes the differences.

[0037] In the search support device 100B, as shown in Fig. 5, the image creation unit 116 includes a synthesis unit 120 that creates a synthetic image as a comparison image by synthesizing an image showing a surrounding object that hides a part of the search target candidate with a converted image. The synthetic image in the second embodiment corresponds to the comparison image that is compared to search for the search target object in the first embodiment. Furthermore, the search support device 100B of the second embodiment includes a search unit 122 that searches for the target candidate by comparing an object included in the captured image with the synthetic image created by the synthesis unit 120.

[0038] As shown in FIG. 6, the synthesis unit 120 creates a plurality of viewpoint conversion images P14 from the search target image P12, synthesizes a peripheral object image with each of the created viewpoint conversion images P14, and creates a plurality of synthetic images P20 as comparison images. The synthetic image P20 may be an image or a video, for example, a still image, or a video including a plurality of images that are continuous in time. For example, the synthesis unit 120 cuts out and extracts an image including a peripheral object image from the captured video P10 using object information of the peripheral object, and creates the synthetic image P20 by superimposing the extracted peripheral object image on the viewpoint conversion image P14. The search unit 118 compares the synthetic image P20 with the captured video P10. As a result, the search unit 122 can detect a search target candidate when at least one of the synthetic images P20 matches a search target candidate included in the captured video P10.

[0039] FIG. 7 is a flowchart showing an example of a processing procedure in the search support device 100B in the second embodiment. First, the terminal side communication unit 104 and the setting unit 110 acquire the search target image P12 from the searcher's terminal device 300 (step S200). Next, the conversion unit 112 creates a viewpoint converted image P14 from the search target image P12 (step S202). Next, the search area side communication unit 102 and the object information acquisition unit 108 acquire the captured image P10 and object information of the search area from the camera system 200 (step S204). Next, the determination unit 114 determines whether or not an object matching the viewpoint converted image P14 has been detected among the objects included in the captured image P10 (step S206). If the determination unit 114 does not detect an object matching the viewpoint converted image P14 (step S206: NO), the process returns to step S204 and the search continues.

[0040] When the determination unit 114 detects an object (search target candidate) that matches the viewpoint converted image P14 (step S206: YES), the determination unit 114 determines whether the search target candidate is hidden by a peripheral object (step S208). When the search target candidate is not hidden by a peripheral object (step S208: NO), the determination unit 114 notifies the search result (step S216). When the search target candidate is hidden by a peripheral object (step S208: YES), the synthesis unit 120 cuts out and extracts an image including an image showing a peripheral object (peripheral object image) (step S210), and creates a synthetic video P20 by synthesizing the peripheral object image with the viewpoint converted image P14 (step S212).

[0041] Next, the search unit 122 judges whether the composite image P20 matches the captured video P10 (step S214). If the composite image P20 does not match the captured video P10 (step S214: NO), the search unit 122 returns the process to step S204. If the composite image P20 matches the captured video P10 (step S214: YES), the search unit 122 notifies the search result (step S216). The search unit 122 transmits, for example, an image showing the search target candidate that matches the composite image P20, position information of the search target candidate, and the like, as the search result to the searcher's terminal device 300.

[0042] The second embodiment of the search support device 100B includes a synthesis unit 120 that creates a synthetic image P20 by synthesizing an image of a peripheral object that hides a part of a search target candidate with a viewpoint converted image P14, and a search unit 122 that searches for the search target object by comparing an object included in a captured video P10 with the synthetic image P20 created by the synthesis unit 120. Thus, the search support device 100B can detect the search target object with a high probability even if the search target object is hidden by a peripheral object.

[0043] The third embodiment will be described below. FIG. 8 is a block diagram showing an example of a functional configuration of a search support device 100C in the third embodiment. FIG. 9 is a diagram for explaining the processing of the search support device 100C in the third embodiment. The search support device 100C in the third embodiment performs both the processing of comparing the whole video P16 with the viewpoint conversion image P14 as in the first embodiment, and the processing of comparing the composite image P20 with the captured video P10 as in the second embodiment. This allows the search support device 100C in the third embodiment to detect the search target object with an even higher probability.

[0044] The image creation unit 130 creates an entire image P16 in which the entire search target candidate can be confirmed from the captured image P10 and the object information, and further creates a composite image P20a by synthesizing an image of a surrounding object included in the captured image P10 with the viewpoint converted image P14. The search unit 132 compares the entire image P16 with the viewpoint converted image P14 as in the first embodiment, and also compares the composite image P20a with the captured image P10 as in the second embodiment. The search unit 132 searches for the search target object based on the comparison result between the entire image P20a and the viewpoint converted image P14 and the comparison result between the composite image P20a and the captured image P10. The search unit 132 may determine that the search target object has been found, for example, when both the similarity between the entire image P20 and the viewpoint converted image P14 and the similarity between the composite image P20a and the captured image P10 are equal to or higher than a reference. The search unit 132 may notify that the search target object may be found when, for example, one of the similarity between the entire video P20 and the viewpoint conversion image P14 and the similarity between the composite image P20a and the captured video P10 is equal to or greater than a reference level. This allows the exploration support device 100C to search for the search target object with higher accuracy.

[0045] FIG. 10 is a flowchart showing an example of a processing procedure in the search support device 100C in the third embodiment. The search support device 100C performs steps S200 to S206 as described above. When the determination unit 114 detects an object (search target candidate) that matches the viewpoint conversion image P14 (step S206: YES), the determination unit 114 determines whether the search target candidate is hidden by a peripheral object (step S208). When the search target candidate is not hidden by a peripheral object (step S208: NO), the determination unit 114 notifies the search result (step S216). When the search target candidate is hidden by a peripheral object (step S208: YES), the video creation unit 130 cuts out and extracts an image including an image showing a peripheral object (peripheral object image) (step S210), creates a composite video P20 by combining the peripheral object image with the viewpoint conversion image P14, and creates an entire video P16 using the peripheral object image (step S300).

[0046] Next, the search unit 132 judges whether the composite image P20 matches the captured image P10, and judges whether the viewpoint converted image P14 matches the entire video P16 (step S302). If the composite image P20 does not match the captured image P10 and the viewpoint converted image P14 does not match the entire video P16 (step S302: NO), the search unit 132 returns the process to step S204. If the composite image P20 matches the captured image P10 or the viewpoint converted image P14 matches the entire video P16 (step S302: YES), the search unit 122 notifies the search result (step S304). If the composite image P20 matches the captured image P10, the search unit 132 transmits an image showing the search target candidate and position information of the search target candidate as the search result to the searcher's terminal device 300. When the viewpoint converted image P14 matches the overall video P16, the search unit 132 transmits an image showing the search target candidate that is the basis of the overall video P16, position information of the search target candidate, and the like to the searcher's terminal device 300 as the search result.

[0047] Below, modified examples of the first embodiment, the second embodiment, and the third embodiment will be described. FIG. 11 is a diagram for explaining a modified example of the search support system. The camera system 200 of the search support system in the modified example may include a mobile robot 230 having a camera device. The mobile robot 230 moves based on, for example, control information from the search support device 100 or an operation of the searcher's terminal device 300. The mobile robot 230 moves in a search area, acquires a captured image, and transmits it to the search support device 100. The search support device 100 acquires the captured image received from the mobile robot 230 as object information. The search support device 100 creates an entire image P16 of a search target candidate included in the captured image received from the camera device 210 using the object information received from the mobile robot 230. As a result, the search support device 100 can create the entire image P16 using, for example, a captured image captured from an imaging direction different from the imaging direction of the camera device 210.

[0048] The mobile robot 230 may move based on the operation of a searcher, and transmit the captured image P10 acquired while moving to the search support device 100. In this way, the mobile robot 230 can provide the captured image P10 to the search support device 100 in place of the camera device 210. As a result, the mobile robot 230 can assist in searching for a search target even in a place where a camera device such as the camera device 210 or the infrared camera device 212 is not installed.

[0049] The exploration support device 100 may control the movement of the mobile robot 230 to obtain the captured image P10 in order to create, for example, a 360-degree free viewpoint video. When the exploration support device 100 detects a search target candidate using a fixed camera device 210 and wants to check the captured image P10 of the search target candidate in more detail, the exploration support device 100 transmits to the mobile robot 230 control information for moving the mobile robot 230 to the location where the search target candidate was found. When the mobile robot 230 moves to the vicinity of the search target candidate, the exploration support device 100 can obtain an image for creating a free viewpoint video by controlling the zoom and imaging direction of the mobile robot 230.

[0050] The fourth embodiment will be described below. A search support device 100D in the fourth embodiment differs from the above-described embodiments in that a search target is searched for based on an object specified by a searcher in a captured video P10.

[0051] Fig. 12 is a block diagram showing an example of a search support device 100D according to a fourth embodiment. Fig. 13 is a diagram for explaining the process of the search support device 100D according to the fourth embodiment. The setting unit 110# of the search support device 100D according to the fourth embodiment includes a target image acquisition unit 110a that acquires an object image P30 designated based on an operation of a searcher among object images included in a captured video P10 as a search target image. The setting unit 110# sets the object image P30 acquired by the target image acquisition unit 110a as a search target image including a search target object.

[0052] FIG. 14 is a flowchart showing an example of a processing procedure in the search support device 100D in the fourth embodiment. The search support device 100D transmits the captured image P10 received from the camera system 200 by the search area side communication unit 102 to the searcher's terminal device 300 via the terminal side communication unit 104# (step S400). As a result, the searcher's terminal device 300 presents the captured image P10 to the searcher. The searcher's terminal device 300 transmits designation information indicating that an object included in the captured image P10 is designated by the searcher's operation to the search support device 100D. The terminal side communication unit 104# of the search support device 100D receives the designation information (step S402). The target image acquisition unit 110a acquires the object image P30 designated by the searcher based on the designation information as a search target image (step S404).

[0053] The conversion unit 112# creates a viewpoint-converted image P32 by converting the viewpoint of the object image P30 acquired by the target image acquisition unit 110a as a search target image (step S406). Next, the captured image acquisition unit 106 acquires the captured image P10, and the object information acquisition unit 108 acquires the object information (step S408). Next, the determination unit 114# detects an object included in the captured image P10 that matches one of the viewpoint-converted images P32 as a search target candidate by comparing the viewpoint-converted image P32 with the captured image P10 (step S412: YES), the determination unit 114# determines whether the search target candidate is hidden by a surrounding object (step S414). If the determination unit 114# does not detect an object that matches the viewpoint-converted image P32 (step S412: NO), the process returns to step S408.

[0054] If a part of the search target candidate is hidden by a peripheral object (step S414: YES), the whole image creation unit 116# cuts out and extracts an image including a peripheral object image (step S418), and creates a whole image P34 in which the whole of the search target candidate can be confirmed from the viewpoint conversion image P32 using the peripheral object image (step S420). The terminal side communication unit 104# transmits the whole image P34 to the searcher's terminal device 300 (step S422). If a part of the search target candidate is not hidden by a peripheral object (step S414: NO), the search support device 100D transmits the viewpoint conversion image P32 that matches the object image P30 to the searcher's terminal device 300 via the terminal side communication unit 104# (step S316). This allows the search support device 100D to search for the search target based on the object specified by the searcher in the captured image P10.

[0055] A modified example of the fourth embodiment will be described below. FIG. 15 is a diagram for explaining a modified example of the fourth embodiment. The camera system 200 of the search support system in the modified example includes a mobile robot 240 having a camera device. The mobile robot 240 moves, for example, based on control information from the search support device 100D or the operation of the searcher's terminal device 300. The mobile robot 240 moves in a search area, for example, based on the operation of the searcher, acquires a captured image P10, and transmits it to the search support device 100D. The search support device 100D acquires the captured image P10 received from the mobile robot 240. This allows the searcher to search for a search target by moving the mobile robot 240. As a result, the mobile robot 240 can assist in searching for a search target even in a place where a camera device such as the camera device 210 or the infrared camera device 212 is not installed.

[0056] The search support device 100D may acquire the captured image P10 from the camera device 210, and may acquire the captured image acquired from the mobile robot 240 as object information. The search support device 100D creates an overall image of the search target candidate included in the captured image P10 received from the camera device 210 using the captured image received from the mobile robot 240. In this way, the search support device 100D can create an overall image using captured images captured from an imaging direction different from that of the camera device 210, for example.

[0057] The exploration support device 100D may control the movement of the mobile robot 240 to acquire captured images in order to create, for example, a 360-degree free viewpoint video. The exploration support device 100D detects a search target candidate using the fixed camera device 210, and when it is desired to check the captured image of the search target candidate in more detail, transmits to the mobile robot 240 control information for moving the mobile robot 240 to the location where the search target candidate was found. When the mobile robot 240 moves to the vicinity of the search target candidate, the exploration support device 100D can acquire images for creating a free viewpoint video by controlling the zoom and image capture direction of the mobile robot 240.

[0058] Hereinafter, other modified examples of the fourth embodiment will be described. The explorer's terminal device 300 has a goggle-type display device (not shown) that is worn on the explorer's head, and may display the captured image P10 on the goggle-type display device. The captured image P10 displayed on the goggle-type display device is preferably a 360-degree free viewpoint image so that the explorer can freely change the viewpoint, but is not limited to this. The captured image P10 may be captured by controlling the imaging direction and imaging range of the camera system 200 according to the explorer's viewpoint, may be captured by controlling the movement of the mobile robot 240 according to the explorer's viewpoint, or may be a virtual reality image that virtually creates an image of the search range.

[0059] The searcher's terminal device 300 includes a detection device for detecting the searcher's gesture, detects finger movements as the searcher's gesture, generates designation information, and transmits it to the search support device 100D. The terminal side communication unit 104# of the search support device 100D receives the designation information, and the target image acquisition unit 110a acquires the object image P30 designated by the searcher based on the designation information as a search target image. The conversion unit 112# creates a viewpoint conversion image P32 by performing viewpoint conversion on the object image P30 designated by the searcher. The determination unit 114# detects a search target candidate by comparing the viewpoint conversion image P32 with the captured image P10, and determines whether the search target candidate is hidden by a peripheral object. When the search target candidate is hidden by a peripheral object, the overall image creation unit 116# creates an overall image P34 from the viewpoint conversion image P32 using object information of the search target object and object information of the peripheral object. The overall image creation unit 116# transmits the overall image P34 to the searcher's terminal device 300 via the terminal side communication unit 104#. When the search target candidate is not hidden by a surrounding object, the search support device 100D transmits the viewpoint converted image P32 that matches the object image P30 to the searcher's terminal device 300 via the terminal side communication unit 104#. This allows the search support device 100D to search for the search target using the overall image P34 or the viewpoint converted image P32. In addition, the search support device 100D allows the searcher to wear a goggle-type display device, so that the searcher can feel as if he or she is searching at the site.

[0060] Although the embodiments of the present disclosure have been described above in detail with reference to the drawings, the specific configurations are not limited to the above-mentioned embodiments, and designs within the scope of the present disclosure are also included. The configurations described in the above-mentioned embodiments can be combined in any combination. [Explanation of symbols]

[0061] 100, 100B, 100C, 100D...search support device, 102...search area side communication unit, 104...terminal side communication unit, 106...captured image acquisition unit, 108...object information acquisition unit, 110...setting unit, 110a...target image acquisition unit, 112...conversion unit, 114...determination unit, 116...image creation unit, 116#...overall image creation unit, 118...search unit, 120...synthesis unit, 122...search unit, 130...image creation unit, 132...search unit, 200...camera system, 210...camera device, 212...infrared camera device, 214...radar device, 216...object information generation device, 220...control unit, 230, 240...mobile robot, 300...searcher terminal device

Claims

1. a captured image acquisition unit that acquires a captured image; A setting unit that sets a search target image including a search target object; an object information acquisition unit that acquires object information indicating an object included in the captured video; a determination unit that determines whether or not a part of a search target candidate is hidden by a surrounding object by using the search target image or the object information; an image creation unit that creates a comparison image to be compared in order to search for the search target object, using an image showing the peripheral object from among the captured images, when the determination unit determines that a part of the search target candidate is hidden by a peripheral object; and a conversion unit that converts the search target image set by the setting unit into a converted image viewed from an arbitrary viewpoint, The setting unit acquires the search target image from another device, The conversion unit converts the search target image into the converted image, the determination unit detects, as the search target candidate, an object included in the captured video that matches one of the converted images by comparing the converted image converted by the conversion unit with an object included in the captured video, and determines whether or not a part of the detected search target candidate is hidden by a surrounding object; a search unit that searches for the search target object by comparing the comparison image created by the image creation unit with the converted image converted by the conversion unit, Search support device.

2. a captured image acquisition unit that acquires a captured image; A setting unit that sets a search target image including a search target object; an object information acquisition unit that acquires object information indicating an object included in the captured video; a determination unit that determines whether or not a part of a search target candidate is hidden by a surrounding object by using the search target image or the object information; an image creation unit that creates a comparison image to be compared in order to search for the search target object, using an image showing the peripheral object from among the captured images, when the determination unit determines that a part of the search target candidate is hidden by a peripheral object; and a conversion unit that converts the search target image set by the setting unit into a converted image viewed from an arbitrary viewpoint, the image creation unit includes a synthesis unit that creates a synthesis image by synthesizing an image showing the peripheral object that hides a part of the search target candidate with the converted image, as the comparison image; a search unit that searches for the search target object by comparing an object included in the captured video with the composite image created by the synthesis unit; Search support device.

3. a captured image acquisition unit that acquires a captured image; A setting unit that sets a search target image including a search target object; an object information acquisition unit that acquires object information indicating an object included in the captured video; a determination unit that determines whether or not a part of a search target candidate is hidden by a surrounding object by using the search target image or the object information; an image creation unit that creates a comparison image to be compared in order to search for the search target object, using an image showing the peripheral object from among the captured images, when the determination unit determines that a part of the search target candidate is hidden by a peripheral object; and a conversion unit that converts the search target image set by the setting unit into a converted image viewed from an arbitrary viewpoint, the setting unit includes a target image acquisition unit that acquires, as the search target image, an object image designated based on an operation by a searcher from among object images included in the captured video; The conversion unit converts the object image acquired by the target image acquisition unit into a plurality of converted images, the determination unit detects, as the search target candidate, an object included in the captured video that matches one of the converted images by comparing the converted image converted by the conversion unit with an object included in the captured video, and determines whether or not a part of the detected search target candidate is hidden by a surrounding object; When a part of the search target candidate is hidden by a surrounding object, the image creation unit transmits the comparison image to a searcher's terminal device, and when a part of the search target candidate is not hidden by a surrounding object, a communication unit transmits the converted image that matches the captured image to the searcher's terminal device. Search support device.

4. A step in which a search support device sets a search target image including a search target object; a step of the exploration support device acquiring a captured image and object information indicating an object included in the captured image; a step of determining by the search support device whether or not a part of a search target candidate is hidden by a surrounding object by using the search target image or the object information; creating a comparison image to be compared in order to search for the search target object using an image showing the peripheral object among the captured images when the search support device determines that a part of the search target candidate is hidden by a peripheral object; a step of converting the set search target image into a converted image viewed from an arbitrary viewpoint by the search support device; The search support device, Acquire the search target image from another device; Transforming the search target image into the transformed image; detecting an object included in the captured video that matches one of the transformed images as the search target candidate by comparing the transformed image with an object included in the captured video, and determining whether or not a part of the detected search target candidate is hidden by a surrounding object; The search support method includes comparing the created comparison image with the converted image to search for the search target object.

5. A step in which a search support device sets a search target image including a search target object; a step of the exploration support device acquiring a captured image and object information indicating an object included in the captured image; a step of determining by the search support device whether or not a part of a search target candidate is hidden by a surrounding object by using the search target image or the object information; creating a comparison image to be compared in order to search for the search target object using an image showing the peripheral object among the captured images when the search support device determines that a part of the search target candidate is hidden by a peripheral object; a step of converting the set search target image into a converted image viewed from an arbitrary viewpoint by the search support device; a step of the search support device synthesizing an image showing the peripheral object hiding a part of the search target candidate with the converted image to create a composite image as the comparison image; a step of searching for the search target object by comparing an object included in the captured video with the created synthetic image by the exploration support device; A search support method comprising:

6. A step in which a search support device sets a search target image including a search target object; a step of the exploration support device acquiring a captured image and object information indicating an object included in the captured image; a step of determining by the search support device whether or not a part of a search target candidate is hidden by a surrounding object by using the search target image or the object information; creating a comparison image to be compared in order to search for the search target object using an image showing the peripheral object among the captured images when the search support device determines that a part of the search target candidate is hidden by a peripheral object; a step of converting the set search target image into a converted image viewed from an arbitrary viewpoint by the search support device; The search support device, acquiring, as the search target image, an object image designated based on an operation by a searcher from among object images included in the captured video; Transforming the acquired object image into a plurality of transformed images; detecting an object included in the captured video that matches one of the transformed images as the search target candidate by comparing the transformed image with an object included in the captured video, and determining whether or not a part of the detected search target candidate is hidden by a surrounding object; When a part of the search target candidate is hidden by a surrounding object, the comparison image is transmitted to a searcher's terminal device, and when a part of the search target candidate is not hidden by a surrounding object, the converted image that matches the captured image is transmitted to the searcher's terminal device. Search support method.

7. The search support device is equipped with a computer. A step of setting a search target image including a search target object; acquiring a captured image and object information indicating an object included in the captured image; determining whether a part of the search target candidate is hidden by a surrounding object by using the search target image or the object information; creating a comparison image to be compared in order to search for the search target object using an image showing the peripheral object from among the captured images when it is determined that a part of the search target candidate is hidden by a peripheral object; converting the set search target image into a converted image viewed from an arbitrary viewpoint; Acquire the search target image from another device; Transforming the search target image into the transformed image; detecting an object included in the captured video that matches one of the transformed images as the search target candidate by comparing the transformed image with an object included in the captured video, and determining whether or not a part of the detected search target candidate is hidden by a surrounding object; searching for the search target object by comparing the created comparison image with the converted image; A program that executes a process.

8. The search support device is equipped with a computer. A step of setting a search target image including a search target object; acquiring a captured image and object information indicating an object included in the captured image; determining whether a part of the search target candidate is hidden by a surrounding object by using the search target image or the object information; creating a comparison image to be compared in order to search for the search target object using an image showing the peripheral object from among the captured images when it is determined that a part of the search target candidate is hidden by a peripheral object; converting the set search target image into a converted image viewed from an arbitrary viewpoint; creating a composite image as the comparison image by combining an image showing the peripheral object that hides a part of the search target candidate with the converted image; searching for the search target object by comparing an object included in the captured video with the created synthetic image; A program to execute.

9. The search support device is equipped with a computer. A step of setting a search target image including a search target object; acquiring a captured image and object information indicating an object included in the captured image; determining whether a part of the search target candidate is hidden by a surrounding object by using the search target image or the object information; creating a comparison image to be compared in order to search for the search target object using an image showing the peripheral object from among the captured images when it is determined that a part of the search target candidate is hidden by a peripheral object; converting the set search target image into a converted image viewed from an arbitrary viewpoint; acquiring, as the search target image, an object image designated based on an operation by a searcher from among object images included in the captured video; Transforming the acquired object image into a plurality of transformed images; detecting an object included in the captured video that matches one of the transformed images as the search target candidate by comparing the transformed image with an object included in the captured video, and determining whether or not a part of the detected search target candidate is hidden by a surrounding object; When a part of the search target candidate is hidden by a surrounding object, the comparison image is transmitted to a searcher's terminal device, and when a part of the search target candidate is not hidden by a surrounding object, the converted image that matches the captured image is transmitted to the searcher's terminal device. A program that executes a process.

10. an imaging device that generates an image; an object information generating device that generates object information indicating an object included in the captured video; a determination unit that determines whether or not a part of a search target candidate is hidden by a peripheral object using the search target image or the object information; an image creation unit that creates a comparison image to be compared with the search target object by using an image of the captured image showing the peripheral object when the determination unit determines that a part of the search target candidate is hidden by the peripheral object; and a conversion unit that converts the search target image set by the setting unit into a converted image seen from an arbitrary viewpoint, wherein the setting unit acquires the search target image from another device, the conversion unit converts the search target image into the converted image, and the determination unit detects an object included in the captured image that matches one of the converted images as the search target candidate, by comparing the converted image converted by the conversion unit with an object included in the captured image, and determines whether or not a part of the detected search target candidate is hidden by a peripheral object, and searches for the search target object by comparing the comparison image created by the image creation unit with the converted image converted by the conversion unit; A search support system comprising:

11. an imaging device that generates an image; an object information generating device that generates object information indicating an object included in the captured video; a setting unit that sets a search target image including a search target object; a determination unit that uses the search target image or the object information to determine whether or not a part of a search target candidate is hidden by a peripheral object; an image creation unit that creates a comparison image to be compared in order to search for the search target object using an image of the captured image that shows the peripheral object when the determination unit determines that a part of the search target candidate is hidden by the peripheral object; and a conversion unit that converts the search target image set by the setting unit into a converted image viewed from an arbitrary viewpoint, wherein the image creation unit includes a synthesis unit that creates a composite image as the comparison image by synthesizing an image showing the peripheral object that hides a part of the search target candidate with the converted image, and a search unit that searches for the search target object by comparing an object included in the captured image with the composite image created by the synthesis unit; A search support system comprising:

12. an imaging device that generates an image; an object information generating device that generates object information indicating an object included in the captured video; a setting unit which sets a search target image including a search target object; a determination unit which determines whether or not a part of a search target candidate is hidden by a peripheral object using the search target image or the object information; an image creation unit which creates a comparison image to be compared in order to search for the search target object using an image showing the peripheral object in the captured image when the determination unit determines that a part of the search target candidate is hidden by the peripheral object; and a conversion unit which converts the search target image set by the setting unit into a converted image viewed from an arbitrary viewpoint, wherein the setting unit comprises a target image acquisition unit which acquires an object image designated based on an operation by a searcher from among object images included in the captured image as the search target image, a conversion unit converts the object image acquired by the target image acquisition unit into a plurality of converted images, the determination unit detects an object included in the captured image that matches one of the converted images by comparing the converted image converted by the conversion unit with an object included in the captured image as the search target candidate, and determines whether or not a part of the detected search target candidate is hidden by a peripheral object, and if a part of the search target candidate is hidden by a peripheral object, the image creation unit transmits the comparison image to a searcher's terminal device, and if a part of the search target candidate is not hidden by a peripheral object, a communication unit transmits the converted image that matches the captured image to the searcher's terminal device; A search support system comprising:

Images