Image search device and program

Abstract

To provide an image search device with which it is possible for a user to simply acquire an image in which him or herself is seen from a continuous and wide-area video data.SOLUTION: Provided is an image search device 20 for retrieving images from databases 5, 6 in which captured videos and imaging range histories regarding a plurality of cameras arranged in a field are preserved. The image search device 20 receives from a user a designation of range information from the position information history of the user in the field (range designation unit 9), and executes first processing to search for images having been captured at the time of the designated range information by cameras whose imaging range histories are overlapping in the designated range history information among the position information histories (search unit 7).SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image search device and a program. [Background technology]

[0002] There are various conventional techniques for video search, for example, Patent Documents 1 to 3 can be mentioned.

[0003] Patent Document 1 discloses a technology for searching for driving video along a route traveled by a vehicle, which allows a user to acquire video of the content desired by the user, such as "cherry blossom viewing" or "swimming in the sea," by specifying tags during a search. Patent Document 2 discloses a technology for lowering the object determination threshold under poor visibility conditions when recognizing a person from a camera mounted on a vehicle, so that the person or bicycle can be recognized even if the person or bicycle suddenly appears from behind an object with poor visibility and only a part of the person or bicycle is captured, rather than the entire person or bicycle. Patent Document 3 discloses a technology for improving tracking accuracy and face recognition accuracy by combining tracking processing and face recognition processing in face tracking technology from video. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-170299 A [Patent Document 2] JP 2022-42283 A [Patent Document 3] JP 2022-112831 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, with conventional technology, each camera takes continuous images over a certain period of time, and in an environment where multiple such cameras are present and distributed over a wide area over a certain spatial range, it is not possible for a user to simply obtain an image of themselves being photographed from the continuous, wide-area video data obtained by multiple cameras.

[0006] That is, in daily life, it is impossible to know when an event that one would like to capture in a photograph will occur, making it difficult to record "that moment," but it is unrealistic in terms of the effort involved to constantly point a camera at the subject (the user himself) and keep taking pictures, so it is possible to use continuous, wide-area video data from multiple cameras as described above. However, it is expected that the task of manually selecting data that corresponds to "that moment" from the continuous, wide-area video data will place a heavy burden on the user. Conventional technologies have not been able to handle such a heavy burden.

[0007] That is, Patent Document 1 is a method for searching for a road on which a user wants to travel from past driving videos, and is not intended for people, and does not have the concept of time as a search key, so it could not handle the load. Patent Document 2 is an object recognition method for recognizing people or bicycles ridden by people and notifying the driver, particularly in poor visibility conditions, for automatic driving of vehicles, and is intended for people, but focuses on real-time video processing during driving, and does not perform searches from continuous and wide-area video data, so it could not handle the load. Patent Document 3 is a technology that continues face recognition and tracking of the same person in a single video and continues to assign information such as the person's name, and tracks a specific person continuously over time, but aims to track within a single video that may involve scene changes, and does not perform searches from continuous and wide-area video data, so it could not handle the load.

[0008] In view of the problems with the conventional techniques described above, it is an object of the present invention to provide an image search device and program that enable a user to easily obtain an image of himself or herself from continuous, wide-area video data. [Means for solving the problem]

[0009] In order to achieve the above object, the present invention is an image search device that performs an image search from a database that stores images and shooting range history of a plurality of cameras arranged in a field, and receives a designation of range information from the user's position information history in the field from the user, and executes a first process to search for images taken at the time of the designated range information by a camera whose shooting range history overlaps with the designated range information in the position information history.The present invention is also characterized by a program that causes a computer to function as the image search device. Effect of the Invention

[0010] According to the present invention, a user can easily obtain an image of himself / herself from continuous and wide-area video data. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a configuration diagram of an image search system according to an embodiment. [Diagram 2] FIG. 1 is a functional block diagram of each element of an image search system according to an embodiment. [Diagram 3] 1 is a flowchart of an operation of an image search system according to an embodiment. [Figure 4] FIG. 2 is a diagram showing each piece of data in a table format in a DB (database) constructed in step S1. [Diagram 5] 1A and 1B are diagrams showing schematic examples in which a shooting range can be determined as a predetermined range according to a camera. [Figure 6] FIG. 2 is a diagram illustrating a search process performed by a search unit. [Figure 7]13 is a flowchart showing details of a search process by a search unit in step S2 according to one embodiment. [Figure 8] FIG. 13 is a diagram showing a schematic example of a screen example presenting searched images in order of time (date and time). [Figure 9] FIG. 13 is a diagram illustrating an example illustrating the presentation of images from a search result in order of cameras appearing in order of angles around a subject. [Figure 10] 1 is a flowchart showing search results presented in a step-by-step manner at two time granularities. [Figure 11] FIG. 1 is a diagram illustrating a hardware configuration of a typical computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] 1 is a configuration diagram of an image search system 100 according to an embodiment. The image search system 100 includes an image search device 20, a terminal 30 configured as a mobile device such as a smartphone carried by each user U moving within a field F, and a large number of cameras C1, C2, C3, ..., CN arranged within the field F.

[0013] In addition, in Fig. 1, only C1, C2, C3, C4, C5, and C6 are shown as an example as a part of the N number of cameras. For the sake of explanation, any one of the N number of cameras will be an arbitrary k-th camera Ck (k = 1, 2, ..., N). Each camera Ck may be fixedly installed in the field F, or may be installed on a moving body M and move within the field F. In Fig. 1, camera C5 is illustrated as being installed on a moving body M such as a cleaning robot.

[0014] The field F is a closed space such as a theme park, which can be controlled by a business operator, each user U can obtain permission to shoot a video of himself / herself, and is an environment in which a large number of video cameras can be installed as cameras Ck (k=1, 2, ..., N), and the video is shot and stored by the large number of cameras Ck. In FIG. 1, only one user U is depicted as having a terminal 30, but in reality, an unspecified number of users each have a terminal 30 and exist in the field F. In order to simplify the drawing, FIG. 1 depicts only one of the many. In other words, the user U and the terminal 30 are any one person / one device among such an unspecified number of people.

[0015] At least between the terminal 30 and the image retrieval device 20, and between each camera Ck and the image retrieval device 20, mutual information communication is possible via a network NW such as the Internet or a LAN.

[0016] 2 is a functional block diagram of each element of an image search system 100 according to an embodiment. The terminal 30 includes a terminal position information acquisition unit 31 that acquires position information of the terminal 30 by any existing method such as a global positioning system (GPS) (outdoor GPS) and / or an indoor GPS based on radio wave strength from a WiFi (registered trademark) access point. Each camera Ck includes a camera position information acquisition unit k1 that acquires position information of the camera Ck by any existing method similar to the terminal position information acquisition unit 31, and an image acquisition unit k2 that acquires images by shooting as a video camera.

[0017] The image search device 20 includes a user information collection unit 1, a photography information collection unit 2, a DB 3 including a user information DB (database) 4, a camera information DB 5, and a video information DB 6, a search unit 7, a user designation unit 8, a range designation unit 9, and an image output unit 10. Here, modifications from the configuration of FIG. 2 are possible, and the image search device 20 only needs to include at least the search unit 7. For example, the DB 3 may be realized in an external server separate from the image search device 20, and the user designation unit 8, the range designation unit 9, and the image output unit 10 may be configured to be realized in the terminal 30.

[0018] Fig. 3 is a flowchart showing the operation of the image search system 100 according to an embodiment. Below, the operation of each functional block of the image search device 20 in Fig. 2 will be described in detail while explaining each step in Fig. 3.

[0019] In step S1, the user information collection unit 1 collects a large amount of user information history over a certain period of time, and the photography information collection unit 2 collects a large amount of photography information over a certain period of time, thereby constructing DB3, and then the process proceeds to step S2.

[0020] In step S1, the user information collection unit 1 collects the history (time change) of the location information of the terminal 30 acquired from the terminal location information acquisition unit 31 of each terminal 30, by linking it with the ID (identifier) ​​of the user U who uses the terminal 30, and stores it in the user information DB4, thereby constructing a user information DB. In step S1, the shooting information collection unit 2 collects the history of the shooting range obtained from the history of the location information acquired from the camera location information acquisition unit k1 of each camera Ck, by linking it with the ID of the camera Ck, and stores it in the camera information DB5, thereby constructing the camera information DB5, and collects the video acquired from the video acquisition unit k2 of each camera, by linking it with the ID of the camera Ck, and stores it in the video information DB6, thereby constructing the video information DB6.

[0021] FIG. 4 is a diagram showing each data of DB3 constructed in step S1 in a table format. Data example D4 is an example of data stored in user information DB4, and the history of the location information is stored in a form linked to the ID (A, B, C, ..., etc.) of each user U. For example, for a certain user A, the location information p = p1, p2, ... is stored at time (date and time) t = t1, t2, .... Data example D5 is an example of data stored in camera information DB5, and for each camera Ck, information on the shooting range R (k, t) at each time (date and time) t = t1, t2, ... is stored as history. Data example D6 is an example of data stored in video information DB6, and for each camera Ck, information on the frame image F (k, t) in the video at each time (date and time) t = t1, t2, ... is stored, thereby storing the video of each camera Ck.

[0022] Regarding the camera information DB5, information on the shooting range R(k,t) can be determined by using information on a predetermined relative shooting range (angle of view) R of the camera Ck that is acquired in advance, and the shooting range R can be defined as the shooting range R(k,t) centered (based) on the position p(k,t) of the camera Ck on the map information (coordinate information) in the field F at time t acquired by the camera position information acquisition unit k1.

[0023] FIG. 5 is an example of the imaging range R(k,t) that can be determined. As in Example EX1, if the camera Ck is a normal camera with an angle of view θ, the angle of view range (a triangle or sector centered on the position p(k,t) of the camera Ck) is set as the imaging range R(k,t). If the camera Ck is a 180° camera as in Example EX2, the 180° range (for example, a semicircle centered on the position p(k,t) of the camera Ck, but may be a rectangle or the like) is set as the imaging range R(k,t). If the camera Ck is a 360° camera as in Example EX3, the 360° range (for example, a circle centered on the position p(k,t) of the camera Ck, but may be a rectangle or a square) may be set as the imaging range R(k,t). As shown in FIG. 5, if the map information of the field F is configured in two dimensions, the imaging range R(k,t) may also be a two-dimensional range. If the map information of the field F is configured in three dimensions, the imaging range R(k,t) may also be a three-dimensional range.

[0024] Returning to the explanation of Fig. 3, in step S2, a search request is received from a user U (a user U who has moved within a field F using a certain terminal 30) for the DB3 constructed in step S1, and the DB3 is searched, and images of the user U obtained as a search result are provided to the user U, and the flow of Fig. 3 ends. The search can be performed as follows (step 1) to (step 3).

[0025] Specifically, (Step 1) when the user designation unit 8 accepts the designation of a user ID from the user U, the search unit 7 searches the user information DB4 and obtains the location information history (equivalent to movement history) of the user U within field F.

[0026] Furthermore, (step 2) in the range designation unit 9, by accepting designation of range information from the user U for which the user U wishes to obtain a captured image within the field F, the search unit 7 searches the camera information DB5 and acquires the ID of the camera Ck whose capturing range overlaps with the part of the user U's location information history that corresponds to the designated range information.

[0027] Here, the range information can be specified by either time (period) or position (spatial range), but the case of specifying by time (period) will be described first below. For example, if DB3 construction in step S1 is constructed by data collection over the past month, and a certain user U visited field F during the day on Sunday one week ago, user U can specify as range information the 10-minute period from 13:00 to 13:10 on that Sunday one week ago, based on the memory that some memorable event or the like occurred.

[0028] Finally, (step 3), the search unit 7 retrieves from the video information DB 6 the video of the history portion of the specified range information among the video of the acquired camera Ck ID, outputs it to the image output unit 10 as a search result, and provides it to the user U as an image of the search result.

[0029] FIG. 6 is a diagram showing a schematic diagram of the search process in step S2 by the search unit 7 (in the case where the range information is specified by time (period)). When the movement history in the field F during the period specified by the user U as the search condition is shown as a path Path, and the position information of the user U on the path Path at time t1 is p1, the cameras C1 and C2 (k=1,2) that are photographing the user U at time t1 can be obtained as cameras (p1∈R(1,t1), p1∈R(2,t1)) with the position p1 being within the photographing range at time t1, and the images can be provided as the search results. Similarly, the cameras C3 and C4 (k=3,4) that are photographing the user U at time t2 can be obtained as cameras (p2∈R(3,t2), p2∈R(4,t2)) with the position p2 being within the photographing range at time t2, and the images can be provided as the search results. Similarly, cameras C5 and C6 (k=5,6) can be obtained as the cameras capturing the image of user U at time t3, and position p3 can be obtained as being within the capturing range at time t3 (p3∈R(5,t3), p3∈R(6,t3)), and the images can be provided as search results.

[0030] 7 is a flowchart showing details of the search process in step S2 by the search unit 7 according to one embodiment. Note that the above (Procedure 1) corresponds to steps S12 and S14, (Procedure 2) corresponds to steps S16 and S18, and (Procedure 3) corresponds to steps S20 and S22.

[0031] In step S10, the user designation unit 8 acquires the ID of the user U as a search key input by the user U, and the range designation unit 9 acquires range information of the user U as a period, and then the process proceeds to step S12. (Note that the range information can be accepted not only as a period, but also as spatial information in the form of a position or range within the field F, and an embodiment of this will be described later.)

[0032] For the sake of explanation below, the specified period as a search key is assumed to be a period of length M {tn|n=1, 2, ..., M}, and any time (date and time) within the specified period is assumed to be tn.

[0033] In step S12, the search unit 7 acquires the location history of the user U corresponding to the user ID from the user information DB4, acquires information on the location pn at time tn in the location history, and then proceeds to step S14. In step S14, it is determined whether or not the acquisition in step S12 has been completed for all times tn in the target period {tn|n=1,2,...,M}, and if so, proceeds to step S16, and if not, returns to step S14 to continue acquiring information on the location pn. In other words, if the determination in step S14 is positive, the location history {pn|n=1,2,...,M} of the user U throughout the entire specified period {tn|n=1,2,...,M} has been acquired.

[0034] In step S16, the search unit 7 searches the camera information DB 5 to obtain information on the camera Ck that captures the position pn at time tn in the position history {pn|n=1,2,...,M} (i.e., information on the camera Ck such that the position pn is included in the capture range R(k,tn) of the camera Ck at time tn and "pn∈R(k,tn)" is satisfied), and then the process proceeds to step S18. In step S18, it is determined whether the information on the camera Ck has been searched and obtained for all times tn in the position history {pn|n=1,2,...,M} within the target period, and if it has been obtained, the process proceeds to step S20, and if it has not been obtained, the process returns to step S16, thereby continuing the search process until the information on all times tn has been obtained.

[0035] In step S20, for a camera Ck (which may be a plurality of cameras) capturing a position pn at time tn in the position history {pn|n=1, 2, ..., M}, the search unit 7 searches the video information DB 6 to acquire a frame image F(k, tn) at the time tn for the camera Ck (i.e., the camera searched and acquired in step S18), and then proceeds to step S22. In step S22, it is determined whether images F(k, tn) of all the cameras Ck corresponding to all the times tn in the position history {pn|n=1, 2, ..., M} within the target period have been searched and acquired, and if they have been acquired, the process proceeds to step S24, and if they have not been searched, the process returns to step S20, thereby continuing the search process until images have been acquired for all the cameras Ck corresponding to all the times tn.

[0036] In step S24, all images F(k,tn) of the camera Ck at all times tn acquired in the repeated step S20 are output from the search unit 7 to the image output unit 10, and are provided to the user U as search results, after which the flow of Fig. 7 ends. There are various possible interfaces for providing the search result images to the user U in the image output unit 10, which will be described later.

[0037] As described above, according to the embodiment of the present invention, a user can select a preferred scene from a chronological list of images in which the user is captured, without being aware of the location and time of the user's stay, or which of the many cameras Ck (k = 1, 2, ..., N) in a field F configured as a theme park or the like that captured the image of the user, i.e., without having to take the burden of manually selecting the camera in which the user is captured from among the many cameras.

[0038] Various supplementary examples, modifications, alternative examples, and the like relating to the embodiments of the present invention will be described below.

[0039] (1) According to an embodiment of the present invention, by using the records of a large number of cameras installed, for example, for security purposes in a field F such as a theme park, it is possible to generate new content with high personal value for the user, not limited to the initial use for security purposes, and thus promote innovation in the content industry and other industries. This makes it possible to contribute to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "build resilient infrastructure, promote sustainable industrialization and foster innovation."

[0040] (2) A function for excluding users with malicious intent (such as voyeurism) may be provided. For example, when the terminal location information acquisition unit 31 is realized by a GPS or the like, if a malicious user U hides and leaves the terminal 30 consisting of a smartphone or the like in the field F, and the user U specifies the period during which the terminal 30 was left in a search, an image of the position where the terminal 30 was left may be provided to the user U, regardless of the actual behavior of the user U, and may be used for voyeurism. For this reason, if the location history during the period specified by the user U in the search is determined to be unnatural, for example, if it is determined by a threshold value that the terminal 30 has been stationary for a certain period of time or more, it may be determined that the search is for an illegal purpose, and the search unit 7 may not perform a search and the search results may not be provided to the user U. Regarding location history, location history when there is no fraudulent purpose and location history when there is a fraudulent purpose can be prepared as learning data, and a model can be constructed using deep learning, etc., and the model can determine whether the user-specified location history is unnatural and may indicate a fraudulent purpose, not just when the user remains stationary, and if it is determined that there may be a fraudulent purpose, searches and search results cannot be provided.

[0041] (3) The location information history of the user U may be acquired by a method other than using the terminal location information acquisition unit 31 using GPS or the like. A face or whole-body image of the user U may be acquired in advance as tracking data (for example, upon entering a facility constituting the field F), and based on the tracking data, the user may be identified in the video recorded in the video information DB6, and the location may be calculated backwards assuming that the user is within the shooting range of the camera Ck that is shooting the identified video at the identified time.

[0042] When performing this location calculation, rather than attempting location calculation for all frames of the video in accordance with the frame rate of the video recorded in the video information DB6, the frame rate of the video may be made rougher (for example, one frame per minute) and location calculation may be attempted discretely, and the user's location history may be estimated by moving in a constant straight line between the discretely estimated positions.

[0043] (4) For a certain time tn within a period specified in the search, rather than presenting to the user via image output unit 10 all images found by search unit 7, search unit 7 may further select from the found images those that are determined to have high content value, and then present only the selected images to the user.

[0044] For example, images with high content value may be determined by acquiring a user's face photo in advance with the user's permission, detecting the user's face image area from the retrieved images using existing face detection technology (OpenFace, etc.), and determining whether the image has high content value using all or part of the following criteria: whether the face is determined to be facing the camera, whether the facial expression is determined to be good (e.g., smiling), whether the facial expression is determined to be bad (e.g., eyes closed, strange expression, etc.), and whether the size of the face area detected in the image is within a certain range based on the entire image (e.g., not too small, not too close up).

[0045] Here, the user's facial image is not acquired in advance, but rather, personal data is extracted from all images searched for in a period specified by the user whose shooting range matches the user's location information history (all images searched in the repeated step S20), and individuals are identified and classified. The person with the highest appearance rate is assumed to be the relevant user, and the facial image of that person is automatically extracted.

[0046] (5) As shown in FIG. 1 as an example in which camera C5 is installed on a moving body M, all or some of the cameras Ck may be mobile. The mobile camera Ck may be realized, for example, as a neck camera for security purposes, or a camera provided on a security robot or a cleaning robot. In this embodiment, in order to enable the video of the mobile camera Ck to be searched, as described above, the shooting range of the camera Ck is stored in the camera information DB5 as history information of the shooting range R(k, tn) that may change over time according to each time tn.

[0047] In addition, when it is known that the camera Ck is not mobile but is fixedly installed, it may be assumed that the camera Ck falls within the same shooting range R(k, tn) = R(k) that does not change over time at all times tn, thereby reducing the amount of data recorded in the camera information DB5.

[0048] (6) When images searched by the search unit 7 are presented to the user on the image output unit 10, they may be sorted by time (date and time) tn and presented in the order of time tn, as shown in Fig. 8. Fig. 8 shows an example EXa in which search result images displayed in a program such as a browser as web page data in a format such as HTML are listed from top to bottom as image search results S(t1), S(t2), ..., S(tn), ..., S(tM) at each of times t1, t2, ..., tn, ..., tM, and presented to the user in the order of time tn. Furthermore, as an example EXb in which the same content as in example EXa is presented in a different form in a format such as HTML, a seek bar SB is provided for specifying time t from all times t1, t2, ..., tn, ..., tM, and when a user sets time t to, for example, t=tn on the seek bar SB, the image search result S(tn) at the time tn is displayed.

[0049] (7) When sorting by time tn as in (6) above and presenting a plurality of images constituting a search result S(tn) for a certain time tn to a user, images F(k,tn) of a camera Ck appearing in the order of angles may be presented in order of the angles around the time tn position pn of the user U (subject) as shown in the example of Fig. 9. From the standpoint of the user U receiving the images, the images presented as the search results for time tn are presented in order around the user U himself, rather than being presented in an irregular order from various angles, so that the user U can smoothly check the search results.

[0050] For this reason, as shown in Fig. 9, by predetermining a reference direction RefD such that the angle θ of rotation on a (two-dimensional) map with respect to the user position pn is θ=0°, the angle θ(k,tn) of the position p(k,tn) of the camera Ck of the image F(k,tn) constituting the search result S(tn) is calculated, and assuming that this angle θ(k,tn) is 0° or more and less than 360°, the image F(k,tn) of the camera Ck with the smallest angle θ(k,tn) is presented in order. In the example of Fig. 9, the cameras C1, C2, C3, and C4 are in order from the smallest angle to the largest angle, so four images (F1, F2, F3, and F4, respectively) are presented in this order.

[0051] As an interface for presenting search result images in the order of angle θ, images may be switched one by one starting from the image with the smallest angle θ, or the screen may be divided to display multiple thumbnails, and the thumbnails may be arranged in the order of angle θ. FIG. 9 shows, as Example EX9, an example of a screen in which thumbnail images are arranged in the order of "F1→F2→F3→F4" for cameras C1, C2, C3, and C4 with the smallest angle θ. In this example, if the horizontal length of the screen display area is long enough to display two thumbnail images, the thumbnail images are arranged in the first row in the order of "F1→F2" and then arranged in the second row in the order of "F3→F4", so that the thumbnail images are arranged and presented in the order of "F1→F2→F3→F4" overall.

[0052] (8) As a method for enabling the user U to perform a search using the image search device 20 and smoothly find an image equivalent to a best shot, the granularity of the time interval of the period constituting the search result may be gradually reduced. FIG. 10 is a flowchart showing an example of the method for gradually presenting the search result with two time granularities. In step S30, the image search device 20 performs a search after setting a time interval (a time interval for extracting frame images from a video) of a first granularity (a granularity coarser than the second granularity) for the specified period, and presents the search result to the user. In step S31, the designation of one image is accepted from the user U to whom the search result of step S30 is presented. In step S32, the image search device 20 performs a search after setting a time interval of the second granularity (a granularity finer than the first granularity) only for videos belonging to a period near the image time designated in step S31, and presents the search result to the user U. It is also possible to accept an image designation from the user U for the search result of step S32, and to perform a search with a finer third granularity in the same manner, and so on.

[0053] For example, in step S30, a four-hour period (12:00 to 16:00) on a certain day when user U stayed in field F is specified as the period, and search results are presented in 240 / 20+1 = 13 time steps divided into 20-minute intervals as the first granularity. If user U specifies an image from 15:00 in step S31, the second search can be performed in step S32, and search results can be presented in 40 / 2+1 = 21 time steps divided into 2-minute intervals as the second granularity over the 40-minute range from 14:40 to 15:20.

[0054] (9) Upon receiving an image specification in step S31 in FIG. 10, the search unit 7 may immediately execute the reading of the search target data in the nearby time range to be searched in the next step S32 at the second granularity, thereby speeding up the obtaining of search results.

[0055] (10) The image intervals that constitute the search results, such as those exemplified as the first granularity and the second granularity in FIG. 10, may be determined by accepting a specification from a user, or a predetermined setting may be established by an administrator of the image search device 20.

[0056] (11) As in the case of the second granularity in step S32 in Fig. 10, when it is determined that the granularity of the time intervals constituting the search results has become finer by the threshold judgment, the search results may be presented in a manner other than the manner of presenting the search results from various cameras Ck at a certain time tn. In the case where there is a camera Ck in the search result image at a certain time tn, the camera Ck may be fixed, and search result images at times tn±1, tn±2, tn±3, ... within a predetermined range near the time tn of the camera Ck may be presented in order of time. When the granularity of the time intervals is fine, if the time tn is fixed and presented, the angle of view may be changed one after another, making it difficult for the user U to select an image, so the method of fixing the camera Ck in this way may be used.

[0057] (12) In the above explanation, the range specification unit 9 accepts input of a period as a search key in the image search device 20, but the specification of range information may be accepted not by period (time) but by position (spatial range) in the field F. In this case, the specification of the spatial range is accepted, and the search unit 7 searches the position history of the user information DB4 for the relevant user for a period during which the user was located within the specified spatial range, and then, as explained above, a search can be performed in the same manner as when the user specifies the period from the beginning.

[0058] For example, if there are multiple attraction facilities within field F and their spatial range is given in advance, the user can specify the range information as "staying at attraction facility α", and the search unit 7 can use the specified range information of attraction facility α to search for the period during which the user's location information is within attraction facility α, for example, from 1:00 p.m. to 2:00 p.m. on a certain day, and then perform a similar search for footage from 1:00 p.m. to 2:00 p.m. during that period.

[0059] (13) FIG. 10 is a diagram showing an example of a hardware configuration in a general computer. Each part of the image search system 100 (including the image search device 20 and the terminal 30) can be realized as one or more computer devices 70 having such a configuration. When each part is realized by two or more computer devices 70, information required for processing may be transmitted and received via a network. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, a GPU (Graphics Processing Unit) 72 as a dedicated processor that executes some or all of the execution instructions of the CPU 71 in place of the CPU 71 or in cooperation with the CPU 71, a RAM 73 as a main storage device (memory) that provides a work area for the CPU 71 (and the GPU 72), a ROM 74 as an auxiliary storage device (storage), a communication interface 75, a display 76, an input interface 77 that accepts user input via a mouse, a keyboard, a touch panel, etc., a camera 81, a sensor 82, and a bus BS for transmitting and receiving data between them.

[0060] Each unit of the image search system 100 can be realized by a CPU 71 and / or a GPU 72 that reads a predetermined program corresponding to the function of each unit from a ROM 74 and executes it. Both the CPU 71 and the GPU 72 are a type of computing device (processor). Here, when display-related processing is performed, a display 76 also operates in conjunction with the CPU 71 and the GPU 72, and when communication-related processing related to data transmission and reception is performed, a communication interface 75 also operates in conjunction with the CPU 71 and the GPU 72.

[0061] The image acquisition unit k2 may be realized by a camera 81, and the processing of the outdoor GPS and indoor GPS constituting the terminal position information acquisition unit 31 and the camera position information acquisition unit k1 may be realized by a sensor 82 configured as a position sensor. [Explanation of symbols]

[0062] 100... image search system, 20... image search device, 30... terminal, C1, C2, ..., Ck, ... CN... camera 31: terminal position information acquisition unit, k1: camera position information acquisition unit, k2: image acquisition unit 1...user information collecting unit, 2...photography information collecting unit, 3...DB, 4...user information DB, 5...camera information DB, 6...video information DB, 7...search unit, 8...user designation unit, 9...condition designation unit, 10...image output unit

Claims

1. An image search device that performs an image search from a database that stores images and shooting range history of a plurality of cameras arranged in a field, An image search device characterized in that it accepts from a user a specification of range information from the user's location information history within the field, and executes a first process to search for images taken at the time of the specified range information by a camera whose shooting range history overlaps with the specified range information in the location information history.

2. 2. The image search device according to claim 1, wherein the range information specified by the user is accepted as a time range or a spatial range.

3. 2. The image search device according to claim 1, further comprising a second process for presenting the images searched for in the first process to the user.

4. 4. The image search device according to claim 3, wherein when the location information history in the range information specified by the user is determined to be unnatural, presentation to the user by the second process is not performed.

5. 4. The image search device according to claim 3, wherein in the second process, images determined to have high content value are selected from the images searched in the first process and then presented to the user.

6. 6. The image search device according to claim 5, wherein the content value is determined to be high based on at least one of the orientation of the user's face in the image, facial expression, and size of a face area.

7. 4. The image search device according to claim 3, wherein the second process performs the presentation in the order of the image shooting times of the images.

8. The image search device according to claim 3, characterized in that in the second process, when images taken at the same time are presented, the images taken by the cameras are presented in the order of the cameras surrounding the position in the user's location information history at that time.

9. 9. The image search device according to claim 8, wherein images captured by the surrounding cameras are arranged as thumbnails and presented in the order of the surrounding cameras.

10. A first iteration of the first process is performed by extracting images from the video in the database at a time interval of a first granularity as search targets; receiving, from the user who has received a presentation result of the second process executed in relation to the first process, a selection of one image from among images in the presentation result; The image search device described in claim 3, characterized in that, for the second time of the first process, images extracted from the images in the database at a time interval of a second granularity finer than the first granularity are executed as search targets for images within a time range surrounding the shooting time of the selected image.

11. 11. A program for causing a computer to function as the image search device according to claim 1.

Images