Search support device, search support system, search support method, and program

Abstract

A search support device according to a first embodiment is provided with: a captured image acquisition unit that acquires a captured image; an object information acquisition unit that acquires object information indicating an object included in the captured image; a setting unit that sets a search target image including a search target object; a determination unit that determines whether or not a portion of the search target candidate is occluded by a surrounding object using the search target image or the object information; and a video creation unit that, when it is determined that a portion of the search target candidate is occluded by the surrounding object, creates a comparison video that is compared to search for the search target object using an image indicating the surrounding object in the captured video.

Description

Technical Field

[0001] The present disclosure relates to a search assistance device, a search assistance system, a search assistance method, and a program. Background Art

[0002] Conventionally, a technique for searching for an object or a person existing in the distance has been known. For example, Patent Document 1 describes the following technique: a multi-eye camera is used to photograph a subject, and a vector medium filter is applied to remove unnecessary objects from an image in which a part of a character is blocked. Patent Document 2 describes the following technique: a person who becomes an object is extracted from an image, and by using images at multiple locations, even when a part of the person's image is missing due to an obstacle, the person is identified. Non-Patent Document 1 describes the following technique: visible light camera images and invisible light camera images are integrated to improve the visual recognition of an object.

[0003] Prior Art Documents

[0004] Patent Documents

[0005] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2019-24258

[0006] Patent Document 2: Japanese Unexamined Patent Application Publication No. 2018-93423

[0007] Non-Patent Documents

[0008] Non-Patent Document 1: NEC Corporation, Tokyo Institute of Technology, "NEC and Tokyo Institute of Technology jointly develop 'Multi-modal Image Fusion Technology' that significantly improves visual recognition under harsh conditions ~ Automatically synthesize visible light and invisible light images ~", [online], June 5, 2017, [searched on June 20, 2022], Internet <URL: https: / / www.titech.ac.jp / news / pdf / webtokyotechpr20170605_okutomi.pdf> Summary of the Invention

[0009] Problems to be Solved by the Invention

[0010] In the above-described techniques, in order to search for an object existing in the distance, an image obtained by photographing with a multi-eye camera, images at multiple locations, or multiple types of images are used. However, due to the number of cameras in the distance, etc., sometimes only an image obtained by photographing from a specific direction can be obtained. When searching for an object whose object to be searched is blocked by surrounding objects in an image obtained by photographing from a specific direction, the object to be searched may not be detected.

[0011] The present disclosure has been completed to solve the above-described problems, and an object thereof is to provide a search assistance device, a search assistance system, a search assistance method, and a program that can highly probably detect a search target object even when the search target object is blocked by surrounding objects.

[0012] Means for solving the problems

[0013] The search assistance device according to the first aspect includes: an imaging image acquisition unit that acquires an imaging image; an object information acquisition unit that acquires object information indicating an object included in the imaging image; a setting unit that sets a search target image including a search target object; a determination unit that determines whether a part of a search target candidate is blocked by surrounding objects using the search target image or the object information; and an image production unit that, when the determination unit determines that a part of the search target candidate is blocked by surrounding objects, produces a comparison image to be compared for searching the search target object using an image representing the surrounding objects in the imaging image.

[0014] The search assistance method according to the second aspect includes the following steps: setting a search target image including a search target object; acquiring an imaging image and object information indicating an object included in the imaging image; determining whether a part of a search target candidate is blocked by surrounding objects using the search target image or the object information; and when it is determined that a part of the search target candidate is blocked by surrounding objects, producing a comparison image to be compared for searching the search target object using an image representing the surrounding objects in the imaging image.

[0015] The program according to the third aspect causes a computer mounted on a search assistance device to execute the following steps: setting a search target image including a search target object; acquiring an imaging image and object information indicating an object included in the imaging image; determining whether a part of a search target candidate is blocked by surrounding objects using the search target image or the object information; and when it is determined that a part of the search target candidate is blocked by surrounding objects, producing a comparison image to be compared for searching the search target object using an image representing the surrounding objects in the imaging image.

[0016] The search assistance system according to the fourth mode includes: a photographing device that generates a photographed image; an object information generation device that generates object information representing an object included in the photographed image; and a search assistance device that includes a setting unit, a determination unit, and an image production unit. The setting unit sets a search target image including a search target object, the determination unit determines whether a part of a search target candidate is blocked by a surrounding object using the search target image or the object information, and the image production unit, when it is determined by the determination unit that a part of the search target candidate is blocked by a surrounding object, produces a comparison image for comparison in searching for the search target object using an image representing the surrounding object in the photographed image.

[0017] Advantageous Effects of the Invention

[0018] According to the present disclosure, even if the search target object is blocked by a surrounding object, the search target object can be detected with a high probability. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram showing an example of the search assistance system according to the first embodiment.

[0020] Figure 2 is a block diagram showing an example of the functional configuration of the search assistance device according to the first embodiment.

[0021] Figure 3 is a diagram for explaining the processing of the search assistance device according to the first embodiment.

[0022] Figure 4 is a flowchart showing an example of the processing steps in the search assistance device according to the first embodiment.

[0023] Figure 5 is a block diagram showing an example of the functional configuration of the search assistance device according to the second embodiment.

[0024] Figure 6 is a diagram for explaining the processing of the search assistance device according to the second embodiment.

[0025] Figure 7 is a flowchart showing an example of the processing steps in the search assistance device according to the second embodiment.

[0026] Figure 8 is a block diagram showing an example of the functional configuration of the search assistance device according to the third embodiment.

[0027] Figure 9 is a diagram for explaining the processing of the search assistance device according to the third embodiment.

[0028] Figure 10It is a flowchart showing an example of the processing steps in the search assistance device in the third embodiment.

[0029] Figure 11 It is a diagram for explaining a modified example of the search assistance system.

[0030] Figure 12 It is a block diagram showing an example of the functional structure of the search assistance device in the fourth embodiment.

[0031] Figure 13 It is a diagram for explaining the processing of the search assistance device in the fourth embodiment.

[0032] Figure 14 It is a flowchart showing an example of the processing steps in the search assistance device in the fourth embodiment.

[0033] Figure 15 It is a diagram for explaining a modified example of the search assistance device in the fourth embodiment. Detailed Embodiment

[0034] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In addition, the scope of the present disclosure is not limited to the following embodiments and can be arbitrarily changed within the scope of the technical idea of the present disclosure. For example, the image in the embodiment can be replaced with a video, and the video in the embodiment can be replaced with an image. In addition, in the following drawings, in order to facilitate understanding of each structure, the scale and quantity of each structure may sometimes be different from those of the actual structure.

[0035] Figure 1 It is a block diagram showing an example of the search assistance system in the first embodiment. The search assistance system has, for example, a search assistance device 100, a camera system 200, and a searcher's terminal device 300. The search assistance device 100 and the searcher's terminal device 300 are provided, for example, in a management room or the like that manages search objects such as lost items in a search area. The camera system 200 is disposed in a distant search area separated from the search assistance device 100 and the searcher's terminal device 300. The search assistance device 100 and the camera system 200 are connected via a network NW such as the Internet. In addition, in the present embodiment, the search assistance device 100, the camera system 200, and the searcher's terminal device 300 are shown using different devices, but this is not limited thereto. The function of the search assistance device 100 and the function of the searcher's terminal device 300 may be provided in one information processing device, and the function of the search assistance device 100 may be provided in the camera system 200.

[0036] The search assistance device 100 performs processing for assisting in searching for a search target object. The search assistance device 100 can be configured, for example, to be connected to a plurality of camera systems 200 and a plurality of searcher terminal devices 300, and provide a cloud service for assisting in searching for a search target object. The search target object is, for example, a lost item in a store, but is not limited thereto, and can also be a missing person or the like.

[0037] The searcher terminal device 300 is an information processing device such as a personal computer or a smartphone operated by a searcher who searches for a search target object. The searcher terminal device 300 has an operation interface for accepting the operations of the searcher, a display interface for presenting various information to the searcher, a storage device, and a control unit such as a CPU (Central Processing Unit) that performs various processes.

[0038] The camera system 200 is provided in a search area. The search area is, for example, a store such as a shopping mall or a restaurant, but is not limited thereto. The camera system 200 includes a plurality of camera devices 210, an object information generation device 216, and a control unit 220. The object information generation device 216 is, for example, an infrared camera device 212 and a radar device 214. The camera device 210 generates a captured image in the search area. The camera device 210 generates a captured image by incident visible light. For example, one or more camera devices 210 are provided in a search area. The camera devices 210 respectively generate captured images of the search area seen from each installation location. In addition, in the present embodiment, the camera device 210 generates images that are continuous in time, but is not limited thereto, and may also generate images that are not continuous in time. Further, in the present embodiment, the object information generation device 216 includes an infrared camera device 212 and a radar device 214, but is not limited thereto. As described later, as long as object information can be generated, other configurations may also be used.

[0039] The object information generation device 216 obtains object information indicating an object included in the captured image. The infrared camera device 212 generates an infrared image as object information by incident infrared light. The radar device 214 measures the distance information to the object as object information by irradiating a laser pulse on the search area and measuring the reflected light.

[0040] In addition, the object information is information representing the two-dimensional or three-dimensional shape of the object to be searched. Specifically, the object information is information representing the two-dimensional or three-dimensional shape of the object to be searched or a surrounding object that obscures a part of the object to be searched or a group of surrounding objects composed of a plurality of surrounding objects. For example, when the object information generation device 216 is the infrared camera device 212, the object information is information representing a two-dimensional shape. When the object information generation device 216 is the radar device 214, the object information is, for example, information representing a three-dimensional shape obtained by vibrating the transmission direction of the radar signal in the vertical and horizontal directions. For example, when the object information generation device 216 is the infrared camera device 212, the object information acquisition unit 108 of the search assistance device 100 described later differentiates the object to be retrieved and the surrounding objects that obscure the object to be retrieved from the perspective image, thereby acquiring the object information of the object to be retrieved. When the object information generation device 216 is the radar device 214, the object information acquisition unit 108 of the search assistance device 100 described later differentiates the object to be retrieved and the surrounding objects that obscure the object to be retrieved based on the difference in distance information, thereby acquiring the object information of the object to be retrieved.

[0041] The control unit 220 controls the camera device 210, the infrared camera device 212, and the radar device 214. The control unit 220 causes the camera device 210, the infrared camera device 212, and the radar device 214 to operate synchronously in time. Thereby, the control unit 220 acquires the captured image and the object information that are synchronized in time. The control unit 220 sends the captured image and the object information that are synchronized in time to the search assistance device 100. The control unit 220 can, for example, change the imaging direction and perform zooming of the imaging range of the camera device 210, the infrared camera device 212, or the radar device 214 according to the operation of the searcher.

[0042] Figure 2 It is a block diagram showing an example of the functional structure of the search assistance device 100 in the first embodiment. Figure 3 It is a diagram for explaining the processing of the search assistance device 100 in the first embodiment.

[0043] The search assistance device 100 includes, for example, a search area side communication unit 102, a terminal side communication unit 104, a captured image acquisition unit 106, an object information acquisition unit 108, a setting unit 110, a conversion unit 112, a determination unit 114, an image production unit 116, and a search unit 118. The search area side communication unit 102 is a communication interface such as a NIC (Network Interface Card) or a wireless communication module for connecting to the network NW. The terminal side communication unit 104 is, for example, a communication interface that communicates via a LAN line. Each of the units such as the captured image acquisition unit 106, the object information acquisition unit 108, the setting unit 110, the conversion unit 112, the determination unit 114, the image production unit 116, and the search unit 118 is implemented, for example, by a computer such as a CPU mounted on the search assistance device 100 executing a program stored in a program memory.

[0044] The captured image acquisition unit 106 acquires a captured image P10. The captured image P10 is supplied via the search area side communication unit 102. The captured image acquisition unit 106, for example, stores in a storage device (not shown) the search area information indicating the search area, the camera information uniquely identifying the camera device 210, and the captured image in association with each other.

[0045] The object information acquisition unit 108 acquires object information indicating an object included in the captured image P10. The object information is supplied via the search area side communication unit 102. The object information includes the object information of the retrieval target object and the object information of the surrounding objects. The object information acquisition unit 108, for example, stores in a storage device (not shown) the search area information indicating the search area, the device information uniquely identifying the infrared camera device 212 or the radar device 214, and the object information in association with each other. The object information acquisition unit 108 can store the acquired object information in association with the search area information or the device information.

[0046] The setting unit 110 sets a search target image P12 including a search target object input via the terminal side communication unit 104. The search target image P12 is, for example, a static image seen from a specific viewpoint provided by a person (search requester) who requests a search for the search target object, such as an image of a lost item or a facial image of a missing person. The setting unit 110 can, for example, acquire an image uploaded from another device such as the search requester's smartphone to the search assistance device 100 as the search target image P12.

[0047] The conversion unit 112 converts the search target image P12 set by the setting unit 110 into a viewpoint conversion image P14 seen from an arbitrary viewpoint. The conversion unit 112 can generate one viewpoint conversion image P14 based on the search target image P12, or can generate a plurality of viewpoint conversion images P14. For example, the conversion unit 112 can crop the region including the search target object included in the search target image P12, and perform image processing such as angle correction, rotation, and size change of the region including the search target object, thereby producing a viewpoint conversion image P14 after converting the viewpoint of the search target object. For example, the conversion unit 112 can predict the parts where the search target object does not appear in the search target image P12, and combine the region including the search target object included in the search target image P12 and the predicted parts, thereby producing a viewpoint conversion image P14 after converting the viewpoint of the search target object. For example, the conversion unit 112 can retrieve objects similar to the search target object in a search site or the like based on the product name information of the search target object, thereby predicting the shape of the search target object and the like.

[0048] The determination unit 114 determines whether a part of the search target candidate is occluded by surrounding objects using the search target image P12 or the object information. The search target candidate may be an object included in the captured image P10. The determination unit 114 compares the viewpoint conversion image P14 obtained by converting the search target image P12 with the objects included in the captured image P10, thereby detecting the object included in the captured image P10 that matches one of the viewpoint conversion images P14 as the search target candidate, and determining whether a part of the detected search target candidate is occluded by surrounding objects. The determination unit 114 can determine whether the search target candidates overlap by using the object information, thereby determining whether a part of the objects included in the captured image P10 is occluded by surrounding objects. The determination unit 114 can determine whether a part of the object information is missing, thereby determining whether a part of the objects included in the captured image P10 is occluded by surrounding objects. The search target candidate is, for example, an object included in the objects included in the captured image P10 that matches at least one of the search target image P12 and one or more viewpoint conversion images P14.

[0049] When the imaging production unit 116 determines that a part of the search target candidate determined by the determination unit 114 is blocked by a surrounding object, it uses the image representing the surrounding object in the captured image P10 to produce a comparison image for comparison in searching for the search target object. For example, the imaging production unit 116 produces an overall image P16 representing the whole of the search target candidate as the comparison image. For example, the imaging production unit 116 uses the infrared image obtained by the infrared camera device 212 to penetrate the surrounding object, thereby interpolating the part blocked by the surrounding object. The imaging production unit 116 uses the distance image obtained by the radar device 214 to determine the position where the surrounding object is located, and interpolates the position where the surrounding object is located. Thus, the imaging production unit 116 produces the overall image P16.

[0050] The search unit 118 compares the overall image P16 produced by the imaging production unit 116 with the viewpoint conversion image P14 converted by the conversion unit 112, thereby searching for the search target object. The search unit 118 sends the search result to the searcher's terminal device 300 via the terminal-side communication unit 104. For example, when the search unit 118 finds a search target candidate, it sends the captured image P10 including the search target candidate and the position information of the search target candidate, etc. as the search result. Thus, the searcher can visually identify the search target object by displaying the captured image P10, confirm the image of the camera device 210 that captures the position of the search target candidate, and thereby search for the search target object.

[0051] Figure 4It is a flowchart showing an example of the processing steps in the search assistance device 100 in the first embodiment. First, the terminal-side communication unit 104 acquires the search target image P12 from the searcher's terminal device 300, and the setting unit 110 sets the search target image P12 (step S100). Next, the conversion unit 112 creates a viewpoint conversion image P14 based on the search target image P12 (step S102). Next, the captured image acquisition unit 106 acquires the captured image P10 of the search area from the camera system 200, and the object information acquisition unit 108 acquires object information (step S104). Next, the determination unit 114 determines whether an object in the captured image P10 that matches the viewpoint conversion image P14 (search target candidate) is detected (step S106). In addition, when the search target candidate matches the viewpoint conversion image P14 with a match degree equal to or higher than a specified reference value, the determination unit 114 may determine that a search target candidate that matches the viewpoint conversion image P14 is detected. That is, the determination unit 114 is not limited to determining whether the entirety of the images match, and may also determine whether a part of the images match. The reference value is preferably a value that can identify the object included in the captured image P10 and the viewpoint conversion image P14 as the same object, but may also be a value appropriately changed by the searcher. When the determination unit 114 does not detect an object that matches the viewpoint conversion image P14 (step S106: No), the process returns to step S104 to continue the search.

[0052] When the determination unit 114 detects a search target candidate that matches the viewpoint conversion image P14 (step S106: Yes), it determines whether a part of the search target candidate is blocked by surrounding objects (step S108). When the determination unit 114 determines that a part of the search target candidate is not blocked by surrounding objects (step S108: No), it notifies the search result (step S116). When the determination unit 114 determines that a part of the search target candidate is blocked by surrounding objects (step S108: Yes), the image production unit 116 cuts out and extracts an image (surrounding object image) that includes an image representing the surrounding objects (step S110), and creates an overall image P16 that can confirm the entirety of the search target candidate using the surrounding object image (step S112). The image production unit 116 identifies the part of the image representing the search target candidate that is blocked by the surrounding objects based on the surrounding object image, and interpolates the part of the image representing the search target candidate that is blocked by the surrounding objects using the object information.

[0053] Next, the search unit 118 determines whether the viewpoint conversion image P14 matches the overall image P16 (step S114). When the search unit 118 does not detect a viewpoint conversion image P14 that matches the overall image P16 (step S114: No), the process returns to step S104. When the search unit 118 detects a viewpoint conversion image P14 that matches the overall image P16 (step S114: Yes), the search result is notified (step S116). In step S116, the search unit 118 transmits, as the search result, for example, an image representing a search object candidate that is the basis of the overall image P16, the position information of the search object candidate, etc. to the searcher's terminal device 300.

[0054] As described above, according to the search assist device 100 of the first embodiment, it is possible to acquire the captured image P10 and the object information, determine whether a part of the search object candidate is blocked by surrounding objects, and in the case where it is determined that a part of the search object candidate is blocked by surrounding objects, use the object information to create an overall image P16 representing the whole of the search object candidate. When the search object candidate is blocked by surrounding objects, it may not be possible to search for the true search object with a high probability. In addition, when the number of camera devices 210 is limited, the possibility of not being able to search for the true search object becomes high. In contrast, according to the search assist device 100 of the first embodiment, in the case where the search object candidate detected by comparing the captured image P10 and the viewpoint conversion image P14 is blocked by surrounding objects, it is possible to create the overall image P16 using the object information. Thus, the search assist device 100, for example, transmits the overall image P16 to the searcher's terminal device 300 so that the searcher can view the overall image P16, thereby enabling the search for whether the search object candidate is the true search object. As a result, according to the search assist device 100 of the first embodiment, even if the search object is blocked by surrounding objects, it is possible to detect the search object with a high probability.

[0055] In the first embodiment, the search assist system includes an infrared camera device 212 or a radar device 214 to acquire object information, and can create the overall image P16 using the object information acquired by the infrared camera device 212 or the radar device 214. Thus, according to the search assist system and the search assist device 100, it is possible to create the overall image P16 by performing processing such as interpolation on the part of the search object candidate that is blocked by surrounding objects.

[0056] According to the search assistance device 100 of the first embodiment, the search assistance device 100 can convert the search target image P12 into a viewpoint conversion image P14 seen from an arbitrary viewpoint, compare the objects included in the viewpoint conversion image P14 and the captured image P10, and thereby detect the object included in the captured image P10 that matches one of the viewpoint conversion images P14 as a search target candidate. According to the search assistance device 100 of the first embodiment, since it has a search unit 118 that searches for the search target object by comparing the overall image P16 and the viewpoint conversion image P14, even if the search target object is blocked by surrounding objects, the search target object can be searched for.

[0057] Next, a second embodiment will be described. Figure 5 FIG. is a block diagram showing an example of the functional configuration of the search assistance device 100B in the second embodiment. Figure 6 FIG. is a diagram for explaining the processing of the search assistance device 100B in the second embodiment. The difference between the search assistance device 100B of the second embodiment and the first embodiment is that a composite image P20 is produced by synthesizing an image of a surrounding object that blocks a part of the search target candidate with the viewpoint conversion image P14, and the produced composite image P20 and the objects included in the captured image P10 are compared to thereby search for the search target object. Next, the description will focus on the differences.

[0058] In the search assistance device 100B, as Figure 5 shown, the image production unit 116 has a synthesis unit 120 that produces a composite image obtained by synthesizing an image of a surrounding object that blocks a part of the search target candidate with the conversion image as a comparison image. The composite image in the second embodiment corresponds to the comparison image that is compared in order to search for the search target object in the first embodiment. Furthermore, the search assistance device 100B of the second embodiment has a search unit 122 that compares the objects included in the captured image and the composite image produced by the synthesis unit 120 to thereby search for target candidates.

[0059] As Figure 6As shown, the synthesizing unit 120 creates a plurality of viewpoint-converted images P14 based on the search target image P12, and synthesizes the surrounding object image with each of the created viewpoint-converted images P14, thereby creating a plurality of synthesized images P20 as comparison images. Additionally, the synthesized image P20 can be an image or a video. For example, it can be a static image or a video containing a plurality of images that are continuous in time. The synthesizing unit 120, for example, uses the object information of the surrounding object, cuts out and extracts an image containing the surrounding object image from the captured video P10, and overlaps the extracted surrounding object image with the viewpoint-converted image P14, thereby creating the synthesized image P20. The searching unit 118 compares the synthesized image P20 and the captured video P10. Thus, the searching unit 122 can retrieve the search target candidate when at least one of the synthesized images P20 matches the search target candidate included in the captured video P10.

[0060] Figure 7 FIG. 4 is a flowchart showing an example of the processing steps in the search assistance device 100B according to the second embodiment. First, the terminal-side communication unit 104 and the setting unit 110 acquire the search target image P12 from the searcher's terminal device 300 (step S200). Next, the conversion unit 112 creates the viewpoint-converted image P14 based on the search target image P12 (step S202). Next, the search area-side communication unit 102 and the object information acquisition unit 108 acquire the captured video P10 of the search area and the object information from the camera system 200 (step S204). Next, the determination unit 114 determines whether an object matching the viewpoint-converted image P14 is detected among the objects included in the captured video P10 (step S206). When the determination unit 114 does not detect an object matching the viewpoint-converted image P14 (step S206: No), the process returns to step S204 to continue the search.

[0061] When the determination unit 114 detects an object (search target candidate) matching the viewpoint-converted image P14 (step S206: Yes), the determination unit 114 determines whether the search target candidate is blocked by the surrounding object (step S208). When the search target candidate is not blocked by the surrounding object (step S208: No), the search result is notified (step S216). When the search target candidate is blocked by the surrounding object (step S208: Yes), the synthesizing unit 120 cuts out and extracts an image containing an image (surrounding object image) representing the surrounding object (step S210), and creates a synthesized video P20 obtained by synthesizing the surrounding object image and the viewpoint-converted image P14 (step S212).

[0062] Next, the search unit 122 determines whether the composite image P20 matches the captured image P10 (step S214). When the composite image P20 does not match the captured image P10 (step S214: No), the search unit 122 returns the process to step S204. When the composite image P20 matches the captured image P10 (step S214: Yes), the search result is notified (step S216). The search unit 122 sends, for example, an image representing a search object candidate that matches the composite image P20, position information of the search object candidate, etc. as the search result to the terminal device 300 for the searcher.

[0063] The search assistance device 100B according to the second embodiment includes: a composite unit 120 that creates a composite image P20 obtained by compositing an image of a surrounding object that obscures a part of the search object candidate and the viewpoint-converted image P14; and a search unit 122 that compares an object included in the captured image P10 with the composite image P20 created by the composite unit 120, thereby searching for the search object. Thus, according to the search assistance device 100B, even when the search object is obscured by a surrounding object, the search object can be detected with a high probability.

[0064] Next, a third embodiment will be described. Figure 8 FIG. is a block diagram showing an example of the functional configuration of the search assistance device 100C in the third embodiment. Figure 9 FIG. is a diagram for explaining the process of the search assistance device 100C in the third embodiment. The search assistance device 100C in the third embodiment performs both the process of comparing the entire image P16 and the viewpoint-converted image P14 as in the first embodiment and the process of comparing the composite image P20 and the captured image P10 as in the second embodiment. Thus, the search assistance device 100C in the third embodiment can detect the search object with a higher probability.

[0065] The image production unit 130 produces an overall image P16 that can identify the overall search object candidate based on the captured image P10 and the object information. Further, the image of the surrounding object included in the captured image P10 is combined with the viewpoint conversion image P14 to produce a combined image P20a. The search unit 132 compares the overall image P16 and the viewpoint conversion image P14 as in the first embodiment, and compares the combined image P20a and the captured image P10 as in the second embodiment. The search unit 132 searches for the search object based on the comparison result between the overall image P20a and the viewpoint conversion image P14 and the comparison result between the combined image P20a and the captured image P10. For example, the search unit 132 can determine that the search object has been found when both the similarity between the overall image P20 and the viewpoint conversion image P14 and the similarity between the combined image P20a and the captured image P10 are above a certain standard. For example, the search unit 132 can notify that the search object may have been found when either the similarity between the overall image P20 and the viewpoint conversion image P14 or the similarity between the combined image P20a and the captured image P10 is above a certain standard. Thus, the search assist device 100C can search for the search object with higher accuracy.

[0066] Figure 10 It is a flowchart showing an example of the processing steps in the search assist device 100C in the third embodiment. The search assist device 100C performs steps S200 to S206 as described above. When the determination unit 114 detects an object (search object candidate) that matches the viewpoint conversion image P14 (step S206: Yes), it determines whether the search object candidate is blocked by surrounding objects (step S208). When the search object candidate is not blocked by surrounding objects (step S208: No), the search result is notified (step S216). When the search object candidate is blocked by surrounding objects (step S208: Yes), the image production unit 130 cuts out and extracts an image (surrounding object image) that includes an image representing the surrounding object, produces a combined image P20 obtained by combining the surrounding object image and the viewpoint conversion image P14, and produces the overall image P16 using the surrounding object image (step S300).

[0067] Next, the search unit 132 determines whether the composite image P20 matches the captured image P10, and determines whether the viewpoint conversion image P14 matches the overall image P16 (step S302). When the composite image P20 does not match the captured image P10 and the viewpoint conversion image P14 does not match the overall image P16 (step S302: No), the search unit 132 returns the process to step S204. When the composite image P20 matches the captured image P10 or the viewpoint conversion image P14 matches the overall image P16 (step S302: Yes), the search unit 122 notifies the search result (step S304). When the composite image P20 matches the captured image P10, the search unit 132 sends, as the search result, an image indicating a search target candidate, position information of the search target candidate, etc. to the searcher's terminal device 300. When the viewpoint conversion image P14 matches the overall image P16, the search unit 132 sends, as the search result, an image indicating a search target candidate that is the basis of the overall image P16, position information of the search target candidate, etc. to the searcher's terminal device 300.

[0068] Next, modified examples of the first embodiment, the second embodiment, and the third embodiment will be described. Figure 11 FIG. is for explaining a modified example of the search assistance system. The camera system 200 of the search assistance system in the modified example may include a mobile robot 230 having a camera device. The mobile robot 230 moves, for example, according to control information from the search assistance device 100 or an operation of the searcher's terminal device 300. The mobile robot 230 moves within the search area, acquires a captured image, and sends it to the search assistance device 100. The search assistance device 100 acquires the captured image received from the mobile robot 230 as object information. The search assistance device 100 uses the object information received from the mobile robot 230 to create an overall image P16 of search target candidates included in the captured image received from the camera device 210. Thus, the search assistance device 100 can create the overall image P16, for example, using a captured image obtained by capturing from a shooting direction different from the shooting direction of the camera device 210.

[0069] The mobile robot 230 can move according to the operation of the searcher and send the captured image P10 acquired while moving to the search assistance device 100. Thus, the mobile robot 230 can provide the captured image P10 to the search assistance device 100 instead of the camera device 210. As a result, for example, in a place where no camera device such as the camera device 210 or the infrared camera device 212 is provided, the mobile robot 230 can also assist in searching for a search target.

[0070] The search assistance device 100 can, for example, control the movement of the mobile robot 230 to obtain a captured image P10 for creating a 360-degree free viewpoint image. The search assistance device 100 detects search object candidates through the fixed camera device 210. When it is desired to more closely confirm the captured image P10 of the search object candidates, the control information for moving the mobile robot 230 to the location where the search object candidates are found is sent to the mobile robot 230. When the mobile robot 230 moves near the search object candidates, the search assistance device 100 controls the zoom and the imaging direction of the mobile robot 230, thereby enabling the acquisition of an image for creating a free viewpoint image.

[0071] Next, a description will be given of the fourth embodiment. The search assistance device 100D in the fourth embodiment is different from the above-described embodiments in that it searches for a search object by the searcher designating an object in the captured image P10.

[0072] Figure 12 FIG. is a block diagram showing an example of the search assistance device 100D in the fourth embodiment. Figure 13 FIG. is a diagram for explaining the processing of the search assistance device 100D in the fourth embodiment. The setting unit 110# of the search assistance device 100D in the fourth embodiment includes an object image acquisition unit 110a that acquires, as a search object image, the object image P30 specified by the searcher's operation from among the object images included in the captured image P10. The setting unit 110# sets the object image P30 acquired by the object image acquisition unit 110a as a search object image including the search object.

[0073] Figure 14 FIG. is a flowchart showing an example of the processing steps in the search assistance device 100D in the fourth embodiment. The search assistance device 100D sends the captured image P10 received from the camera system 200 by the search area side communication unit 102 to the searcher terminal device 300 via the terminal side communication unit 104# (step S400). Thereby, the searcher terminal device 300 presents the captured image P10 to the searcher. The searcher terminal device 300 sends designation information indicating an object included in the captured image P10 designated by the searcher's operation to the search assistance device 100D. The terminal side communication unit 104# of the search assistance device 100D receives the designation information (step S402). The object image acquisition unit 110a acquires the object image P30 designated by the searcher as a search object image based on the designation information (step S404).

[0074] The conversion unit 112# performs a viewpoint conversion on the object image P30 obtained as the search target image by the object image acquisition unit 110a, thereby producing a viewpoint-converted image P32 (step S406). Next, the captured image acquisition unit 106 acquires a captured image P10, and the object information acquisition unit 108 acquires object information (step S408). Next, the determination unit 114# compares the objects included in the viewpoint-converted image P32 and the captured image P10, and thereby detects, as a search target candidate, the object included in the captured image P10 that matches one of the viewpoint-converted images P32. When the determination unit 114# detects a search target candidate by comparing the viewpoint-converted image P32 and the captured image P10 (step S412: Yes), it determines whether the search target candidate is occluded by surrounding objects (step S414). When the determination unit 114# does not detect an object that matches the viewpoint-converted image P32 (step S412: No), it returns the process to step S408.

[0075] In the case where a part of the search target candidate is occluded by surrounding objects (step S414: Yes), the overall image production unit 116# cuts out and extracts an image including the surrounding object image (step S418), and uses the surrounding object image to produce an overall image P34 that enables confirmation of the entirety of the search target candidate based on the viewpoint-converted image P32 (step S420). The terminal-side communication unit 104# transmits the overall image P34 to the searcher's terminal device 300 (step S422). When a part of the search target candidate is not occluded by surrounding objects (step S414: No), the search assistance device 100D transmits the viewpoint-converted image P32 that matches the object image P30 to the searcher's terminal device 300 via the terminal-side communication unit 104# (step S316). Thereby, the search assistance device 100D can search for a search target based on an object specified by the searcher in the captured image P10.

[0076] Next, a modification of the fourth embodiment will be described. Figure 15This is a diagram for explaining a modification of the fourth embodiment. The camera system 200 of the search assistance system in the modification includes a mobile robot 240 having a camera device. The mobile robot 240 moves, for example, according to control information from the search assistance device 100D or operations of the searcher's terminal device 300. The mobile robot 240 moves within the search area according to the operations of the searcher, acquires a captured image P10, and transmits it to the search assistance device 100D. The search assistance device 100D acquires the captured image P10 received from the mobile robot 240. As a result, the searcher can search for the search target by moving the mobile robot 240. Consequently, for example, in a place where a camera device such as the camera device 210 or the infrared camera device 212 is not installed, the mobile robot 240 can also assist in searching for the search target.

[0077] The search assistance device 100D can acquire the captured image P10 from the camera device 210 and acquire the captured image obtained from the mobile robot 240 as object information. The search assistance device 100D uses the captured image received from the mobile robot 240 to create an overall image of the search target candidates included in the captured image P10 received from the camera device 210. Thus, the search assistance device 100D can create an overall image using, for example, a captured image obtained by capturing from a shooting direction different from the shooting direction of the camera device 210.

[0078] The search assistance device 100D can, for example, control the movement of the mobile robot 240 to acquire a captured image in order to create a 360-degree free viewpoint image. The search assistance device 100D detects search target candidates through the fixed camera device 210. When it is desired to more closely confirm the captured image of the search target candidates, the search assistance device 100D transmits control information for moving the mobile robot 240 to the place where the search target candidates are found to the mobile robot 240. When the mobile robot 240 moves near the search target candidates, the search assistance device 100D controls the zoom and shooting direction of the mobile robot 240, thereby enabling acquisition of an image for creating a free viewpoint image.

[0079] Next, another modification of the fourth embodiment will be described. The searcher's terminal device 300 may include a goggle-type display device (not shown) worn on the searcher's head, and the goggle-type display device displays the captured image P10. Preferably, the captured image P10 displayed on the goggle-type display device is a 360-degree free-viewpoint image so that the searcher can freely change the viewpoint. However, it is not limited thereto. The captured image P10 may be obtained by controlling the imaging direction and imaging range of the camera system 200 in combination with the searcher's viewpoint, or may be obtained by controlling the movement of the mobile robot 240 in combination with the searcher's viewpoint. It may also be a virtual reality image created by virtualizing the image of the search range.

[0080] The searcher's terminal device 300 is provided with a detection device that detects the searcher's gestures, detects the movement of the fingers as the searcher's gestures, generates designation information, and transmits it to the search assistance device 100D. The terminal-side communication unit 104# of the search assistance device 100D receives the designation information, and the object image acquisition unit 110a acquires the object image P30 designated by the searcher as the search target image according to the designation information. The conversion unit 112# performs viewpoint conversion on the object image P30 designated by the searcher, thereby creating a viewpoint-converted image P32. The determination unit 114# compares the viewpoint-converted image P32 with the captured image P10, thereby detecting search target candidates and determining whether the search target candidates are blocked by surrounding objects. When the search target candidates are blocked by surrounding objects, the overall image creation unit 116# creates an overall image P34 based on the viewpoint-converted image P32 using the object information of the search target object and the object information of the surrounding objects. The overall image creation unit 116# transmits the overall image P34 to the searcher's terminal device 300 via the terminal-side communication unit 104#. When the search target candidates are not blocked by surrounding objects, the search assistance device 100D transmits the viewpoint-converted image P32 corresponding to the object image P30 to the searcher's terminal device 300 via the terminal-side communication unit 104#. Thus, the search assistance device 100D can search for the search target using the overall image P34 or the viewpoint-converted image P32. In addition, the search assistance device 100D allows the searcher to wear a goggle-type display device, which can give the feeling of actually going to the scene to search.

[0081] As described above, the embodiments of the present disclosure have been described in detail with reference to the accompanying drawings. However, the specific structure is not limited to the above embodiments and also includes designs that do not depart from the gist of the present disclosure. The various structures described in the above embodiments can be arbitrarily combined.

[0082] Reference Signs

[0083] 100, 100B, 100C, 100D: Search assistance device; 102: Search area side communication unit; 104: Terminal side communication unit; 106: Camera image acquisition unit; 108: Object information acquisition unit; 110: Setting unit; 110a: Object image acquisition unit; 112: Conversion unit; 114: Determination unit; 116: Image production unit; 116#: Overall image production unit; 118: Search unit; 120: Composition unit; 122: Search unit; 130: Image production unit; 132: Search unit; 200: Camera system; 210: Camera device; 212: Infrared camera device; 214: Radar device; 216: Object information generation device; 220: Control unit; 230, 240: Mobile robot; 300: Searcher's terminal device.

Claims

1. A search assistance device, the search assistance device having: An imaging image acquisition unit that acquires an imaging image; A setting unit that sets a search target image including a search target object; An object information acquisition unit that acquires object information indicating an object included in the imaging image; A determination unit that determines whether a part of a search target candidate is blocked by a surrounding object using the search target image or the object information; and An image production unit that, when the determination unit determines that a part of the search target candidate is blocked by a surrounding object, uses an image representing the surrounding object in the imaging image to produce a comparison image for comparison in searching for the search target object.

2. The search assistance device according to claim 1, wherein The image production unit produces an overall image representing the entire search target candidate as the comparison image.

3. The search assistance device according to claim 2, wherein The object information acquisition unit is an infrared camera or a radar device, The image production unit produces the overall image using an infrared camera image or radar information obtained by the infrared camera or the radar device.

4. The search assistance device according to claim 1, wherein The search assistance device has a conversion unit that converts the search target image set by the setting unit into a converted image seen from an arbitrary viewpoint.

5. The search assistance device according to claim 4, wherein The setting unit acquires the search target image from another device, The conversion unit converts the search target image into the converted image, The determination unit compares the converted image converted by the conversion unit with the objects included in the imaging image, detects an object included in the imaging image that matches one of the converted images as the search target candidate, and determines whether a part of the detected search target candidate is blocked by a surrounding object, The search assistance device has a search unit that searches for the search target object by comparing the comparison image produced by the image production unit with the converted image converted by the conversion unit.

6. The search assistance device according to claim 4, wherein The image production unit has a synthesis unit that produces a synthesized image as the comparison image, and the synthesized image is obtained by synthesizing an image representing the surrounding object that blocks a part of the search target candidate with the converted image. There is a search unit that searches for the search target object by comparing the objects included in the imaging image with the synthesized image produced by the synthesis unit.

7. The search assistance device according to claim 4, wherein The setting unit has an object image acquisition unit that acquires, as the search target image, an object image specified according to the operation of a searcher from among the object images included in the imaging image, The conversion unit converts the object image acquired by the object image acquisition unit into a plurality of the converted images, The determination unit compares the converted image converted by the conversion unit with the objects included in the captured image, and thereby detects the objects included in the captured image that match one of the converted images as candidates for the search target, and determines whether a part of the detected candidates for the search target is blocked by surrounding objects. It has a communication unit. When a part of the search target candidate is blocked by surrounding objects, the image production unit sends the comparison image to the terminal device for the searcher. When a part of the search target candidate is not blocked by surrounding objects, the communication unit sends the converted image that matches the captured image to the terminal device for the searcher.

8. A search assistance method, the search assistance method comprising the following steps: Set a search target image including a search target object; Obtain a captured image and object information indicating the objects included in the captured image; Use the search target image or the object information to determine whether a part of the search target candidate is blocked by surrounding objects; and When it is determined that a part of the search target candidate is blocked by surrounding objects, use the image representing the surrounding objects in the captured image to produce a comparison image for comparing in order to search for the search target object.

9. A program, the program causes a computer incorporated in a search assistance device to execute the following steps: Set a search target image including a search target object; Obtain a captured image and object information indicating the objects included in the captured image; Use the search target image or the object information to determine whether a part of the search target candidate is blocked by surrounding objects; and When it is determined that a part of the search target candidate is blocked by surrounding objects, use the image representing the surrounding objects in the captured image to produce a comparison image for comparing in order to search for the search target object.

10. A search assistance system, the search assistance system having: A camera device that generates a captured image; An object information generation device that generates object information indicating the objects included in the captured image; and A search assistance device having a setting unit, a determination unit, and an image production unit. The setting unit sets a search target image including a search target object. The determination unit uses the search target image or the object information to determine whether a part of the search target candidate is blocked by surrounding objects. When the determination unit determines that a part of the search target candidate is blocked by surrounding objects, the image production unit uses the image representing the surrounding objects in the captured image to produce a comparison image for comparing in order to search for the search target object.

Images