Information processing device, detection method, and detection program

Abstract

To make it easier to find lost items that have been placed in another item.SOLUTION: An information processing device 10 detects a first object appearing in a first object-searched image 1 from images taken at multiple times in a camera video 11a. The information processing device 10 outputs a first detected image 2 in which the first object is detected. The information processing device 10 identifies a second object-searched image 3 included in the first detected image 2. The information processing device 10 detects a second object appearing in the second object-searched image 3 from images taken at multiple times in the camera video 11a. The information processing device 10 outputs a second detected image 4 in which the second object is detected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a detection method, and a detection program. [Background technology]

[0002] In daily life, we handle many items such as wallets, keys, glasses, etc., and we may lose them. In order to find lost items, it is possible to detect the items from images of the room taken with a camera.

[0003] As a technology for assisting in the discovery of lost items, for example, an image information processing device that provides a means for a user to easily search for any lost accessory has been proposed, and for example, an object monitoring device that can monitor an object carried by a person to prevent it from being lost has also been proposed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-118464 [Patent Document 2] Japanese Patent Publication No. 2021-111033 Summary of the Invention [Problem to be solved by the invention]

[0005] In the method of taking a picture of a room to search for a lost item, an image of the lost item from the video is shown to the user, and the user can find the lost item by searching for the location in the room where the lost item was captured. However, if the lost item is carried inside another item, the lost item will not be captured in the video and will not be able to be found.

[0006] In one aspect, the present case aims to facilitate the discovery of lost items placed inside other items. [Means for solving the problem]

[0007] In one example, an information processing device is provided that includes a storage unit and a processing unit. The storage unit stores camera images. The processing unit detects a first object captured in a first search image from the camera images captured at multiple times, outputs a first detected image in which the first object is detected, identifies a second search image included in the first detected image, detects a second object captured in the second search image from the camera images captured at multiple times, and outputs a second detected image in which the second object is detected. [Effects of the Invention]

[0008] According to one embodiment, it is possible to facilitate the discovery of a lost item that has been placed in another item. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of an information processing device according to a first embodiment. [Figure 2] FIG. 10 illustrates an example of an information processing system according to a second embodiment. [Figure 3] FIG. 2 illustrates an example of a hardware configuration of a server. [Figure 4] FIG. 2 is a block diagram illustrating an example of the functions of a server. [Figure 5] FIG. 10 is a diagram illustrating an example of member information. [Figure 6] FIG. 10 is a diagram illustrating an example of registered item information. [Figure 7] FIG. 10 is a diagram illustrating an example of detection information. [Figure 8] FIG. 10 is a diagram illustrating an example of a lost item detection process. [Figure 9] FIG. 10 is a diagram illustrating an example of a setting process for a tracking search. [Figure 10] FIG. 10 is a diagram illustrating an example of a tracking search process. [Figure 11] 10 is a flowchart illustrating an example of a procedure for a pre-detection process. [Figure 12] 10 is a flowchart (part 1) illustrating an example of a procedure for a lost item detection process. [Figure 13]10 is a flowchart (part 2) illustrating an example of a procedure for a lost item detection process. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present embodiment will be described below with reference to the drawings. Note that each embodiment can be implemented in combination with a plurality of other embodiments within a range that does not contradict each other. [First embodiment] First, the first embodiment will be described.

[0011] FIG. 1 is a diagram showing an example of an information processing device according to a first embodiment. The first embodiment is for supporting the discovery of lost items. The information processing device 10 is a computer that detects objects from video. For example, the information processing device 10 is a server computer that provides a service that supports the discovery of lost items. The information processing device 10 has a storage unit 11 and a processing unit 12.

[0012] The memory unit 11 stores data used in processing executed by the information processing device 10. The memory unit 11 is, for example, a memory or storage device included in the information processing device 10. The memory unit 11 has camera video 11a. The camera video 11a is video captured by a camera installed in a predetermined location. For example, the camera video 11a is video captured by a camera installed inside the home of a user who uses a service to assist in finding lost items. Note that the camera video 11a may be a combination of videos captured by multiple cameras.

[0013] The processing unit 12 controls the information processing device 10 and is capable of executing required processing. The processing unit 12 is, for example, a processor or an arithmetic circuit included in the information processing device 10. The processing unit 12 executes the following processing in response to a detection request for an item being searched for (first object). For example, the processing unit 12 receives a detection request from a user terminal, such as a smartphone, a PC (Personal Computer), or a tablet terminal, operated by a user.

[0014] The processing unit 12 acquires a first missing object image 1 that depicts a first object. The first missing object image 1 may be registered in advance in association with the first object, or may be included in the detection request. The processing unit 12 detects the first object depicted in the first missing object image 1 from images of the camera video 11a taken at multiple times. For example, the processing unit 12 extracts features of the first object from the first missing object image 1. Then, the processing unit 12 detects the features of the first object for each of the images of the camera video 11a taken at multiple times.

[0015] The processing unit 12 outputs a first detected image 2 in which the first object is detected. Here, the processing unit 12 outputs the image with the most recent photographing time among the images in which the first object is detected as the first detected image 2. For example, the processing unit 12 transmits the first detected image 2 to the user terminal. Note that the processing unit 12 may output an image obtained by extracting a predetermined period of time including the photographing time of the first detected image 2 from the camera video 11a. The processing unit 12 identifies a second search image 3 included in the first detected image 2. Here, the processing unit 12 identifies the second search image 3 based on a range selection operation of the first detected image 2 by the user. For example, the processing unit 12 identifies a portion cut out from the first detected image 2 by the range selection operation on the user terminal as the second search image 3. Note that the processing unit 12 may automatically recognize an object into which a person has placed the first object, and identify a portion containing the object as the second search image 3.

[0016] The processing unit 12 detects the second object appearing in the second object-searched image 3 from the images of the camera video 11a taken at multiple times. For example, the processing unit 12 extracts features of the second object from the second object-searched image 3. Then, the processing unit 12 detects the features of the second object for each of the images of the camera video 11a taken at multiple times.

[0017] The processing unit 12 outputs a second detected image 4 in which the second object is detected. Here, the processing unit 12 outputs the image with the most recent photographed time among the images in which the second object is detected as the second detected image 4. For example, the processing unit 12 transmits the second detected image 4 to the user terminal. Note that the processing unit 12 may also output an image obtained by extracting a predetermined period of time including the photographed time of the second detected image 4 from the camera video 11a.

[0018] According to the first embodiment, the processing unit 12 of the information processing device 10 detects a first object appearing in the first object-searching image 1 from images of the camera video 11a taken at multiple times, and outputs a first detected image 2 in which the first object is detected. The processing unit 12 identifies a second object-searching image 3 included in the first detected image 2, detects a second object appearing in the second object-searching image 3 from images of the camera video 11a taken at multiple times, and outputs a second detected image 4 in which the second object is detected.

[0019] As a result, even if the first object that the user is looking for is placed inside a second object and carried therein, the information processing device 10 can detect the second object from the camera image 11a and assist the user in finding the first object. Thus, the information processing device 10 can easily find a lost item that has been placed inside another object.

[0020] Furthermore, the processing unit 12 identifies the second object-searched image 3 based on the user's range selection operation on the first detected image 2. This allows the information processing device 10 to accurately identify the second object that has contained the first object.

[0021] The processing unit 12 may detect a predetermined human behavior from images of the camera video 11a at multiple times, output a first detected image 2 in which a first object is detected and the predetermined behavior is detected, and output a second detected image 4 in which a second object is detected and the predetermined behavior is detected. This allows the information processing device 10 to present to the user the situation when the first object and the second object move away from the user.

[0022] Furthermore, the processing unit 12 may execute, at predetermined time intervals, a process of acquiring an image from the camera and updating the camera image 11a, detecting a first object from the image of the camera image at the current time, and recording a stamp indicating whether the first object has been detected in association with the current time. Then, upon receiving a request to detect the first object, the processing unit 12 may output the image of the camera image 11a at the time the stamp was recorded as the first detected image 2. In this way, the information processing device 10 can reduce the time from the detection request to the output of the first detected image 2 by detecting the first object in advance.

[0023] Second Embodiment Next, a second embodiment will be described, which provides a service to assist in finding an item lost at home.

[0024] 2 is a diagram illustrating an example of an information processing system according to a second embodiment. The information processing system according to the second embodiment includes a camera 22, a smartphone 23, and a server 100. The camera 22, the smartphone 23, and the server 100 are connected to a network 30. The network 30 is, for example, a wide area network such as the Internet.

[0025] The camera 22 is installed in the house 20. The house 20 is the home of a user 21 who uses the lost item discovery support service provided by the server 100. The camera 22 is installed so as to be able to capture images of the interior of the house 20. The camera 22 transmits the captured camera images to the server 100 in real time. The smartphone 23 is a smartphone operated by the user 21. The smartphone 23 transmits a request to the server 100 to detect the lost item in response to an operation by the user 21. Note that the server 100 may also provide the lost item discovery support service to users other than the user 21, and the information processing system of the second embodiment may have a camera and a smartphone for each user to whom the service is provided.

[0026] The server 100 is a server computer that provides a service to assist in finding lost items. The server 100 acquires in real time camera images captured by the camera 22 from the camera 22. When the server 100 acquires the camera images from the camera 22, it detects objects (registered objects) that are registered in association with the user 21 from the acquired camera images.

[0027] When the server 100 receives a request to detect a registered object from the smartphone 23, the server 100 transmits to the smartphone 23 an image obtained by extracting a portion in which the registered object is detected from the camera image acquired from the camera 22. The smartphone 23 displays the image acquired from the server 100. Here, when the smartphone 23 receives a range selection operation from the displayed image, the smartphone 23 transmits to the server 100 a request to track and search for an object that appears in an image (cut-out image) obtained by cutting out the selected range.

[0028] When the server 100 receives a tracking search request from the smartphone 23, the server 100 detects an object appearing in the cutout image from the camera video acquired from the camera 22. Then, the server 100 transmits to the smartphone 23 a video in which a portion of the camera video acquired from the camera 22 in which the object appearing in the cutout image has been detected is extracted.

[0029] 3 is a diagram showing an example of a hardware configuration of a server. The entire server 100 is controlled by a processor 101. A memory 102 and multiple peripheral devices are connected to the processor 101 via a bus 109. The processor 101 may be a multiprocessor. The processor 101 is, for example, a central processing unit (CPU), a micro processing unit (MPU), or a digital signal processor (DSP). At least some of the functions realized by the processor 101 executing a program may be realized by an electronic circuit such as an application specific integrated circuit (ASIC) or a programmable logic device (PLD).

[0030] The memory 102 is used as a main storage device of the server 100. The memory 102 temporarily stores at least a part of the OS (Operating System) programs and application programs to be executed by the processor 101. The memory 102 also stores various data used in processing by the processor 101. As the memory 102, for example, a volatile semiconductor storage device such as a RAM (Random Access Memory) is used.

[0031] The peripheral devices connected to the bus 109 include a storage device 103, a GPU (Graphics Processing Unit) 104, an input interface 105, an optical drive device 106, a device connection interface 107, and a network interface 108.

[0032] The storage device 103 electrically or magnetically writes and reads data to and from a built-in recording medium. The storage device 103 is used as an auxiliary storage device for a computer. The storage device 103 stores an OS program, application programs, and various data. Note that the storage device 103 may be, for example, a hard disk drive (HDD) or a solid state drive (SSD).

[0033] The GPU 104 is connected to a monitor 31. The GPU 104 displays an image on the screen of the monitor 31 in accordance with an instruction from the processor 101. The monitor 31 may be a display device using an organic EL (Electro Luminescence) display device, a liquid crystal display device, or the like.

[0034] The input interface 105 is connected to a keyboard 32 and a mouse 33. The input interface 105 transmits signals sent from the keyboard 32 and the mouse 33 to the processor 101. The mouse 33 is an example of a pointing device, and other pointing devices can also be used. Examples of other pointing devices include a touch panel, a tablet, a touch pad, and a trackball.

[0035] The optical drive device 106 uses a laser beam or the like to read data recorded on the optical disc 34. The optical disc 34 is a portable recording medium on which data is recorded so that it can be read by reflected light. The optical disc 34 includes a DVD (Digital Versatile Disc), a DVD-RAM, a CD-ROM (Compact Disc Read Only Memory), a CD-R (Recordable) / RW (Rewritable), etc.

[0036] The device connection interface 107 is a communication interface for connecting peripheral devices to the server 100. For example, a memory device 35 or a memory reader / writer 36 can be connected to the device connection interface 107. The memory device 35 is a recording medium equipped with a function for communicating with the device connection interface 107. The memory reader / writer 36 is a device for writing data to the memory card 37 or reading data from the memory card 37. The memory card 37 is a card-type recording medium.

[0037] The network interface 108 is connected to the network 30. The network interface 108 transmits and receives data to and from other computers or communication devices via the network 30.

[0038] The server 100 can realize the processing functions of the second embodiment with the hardware configuration described above. The information processing device 10 shown in the first embodiment can also be realized with the same hardware as the server 100 shown in FIG. 3. The processor 101 is an example of the processing unit 12 shown in the first embodiment. The memory 102 or the storage device 103 is an example of the storage unit 11 shown in the first embodiment.

[0039] The server 100 realizes the processing functions of the second embodiment by executing a program recorded on, for example, a computer-readable recording medium. The program describing the processing to be executed by the server 100 can be recorded on various recording media. For example, the program to be executed by the server 100 can be stored in a storage device 103. The processor 101 loads at least a portion of the program in the storage device 103 into the memory 102 and executes the program. The program to be executed by the server 100 can also be recorded on a portable recording medium such as an optical disk 34, a memory device 35, or a memory card 37. The program stored on the portable recording medium becomes executable after being installed on the storage device 103 under the control of, for example, the processor 101. The processor 101 can also read and execute the program directly from the portable recording medium. Next, the functions of the server 100 will be described in detail.

[0040] 4 is a block diagram showing an example of the functions of the server. The server 100 has a storage unit 120, a transmission / reception unit 130, and an image processing unit 140. The storage unit 120 is realized using a storage area of ​​the memory 102 or the storage device 103. The transmission / reception unit 130 and the image processing unit 140 are realized by the processor 101 executing a program stored in the memory 102.

[0041] The memory unit 120 stores member information 121, camera images 122-1, 122-2,..., registered object information 123-1, 123-2,..., and detection information 124-1, 124-2,.... The member information 121 is information about members who receive services provided by the server 100. The camera images 122-1, 122-2,... are images acquired from cameras corresponding to the respective users. For example, the camera image 122-1 is an image acquired from the camera 22 corresponding to the user 21.

[0042] The registered object information 123-1, 123-2, etc. is information about registered objects for each user. For example, the registered object information 123-1 is information about registered objects for user 21. The detection information 124-1, 124-2, etc. is information about the detection of objects from each of the camera images 122-1, 122-2, etc. For example, the detection information 124-1 is information about the detection of an object from the camera image 122-1.

[0043] The transmitting / receiving unit 130 transmits and receives data to and from the smartphone 23. The transmitting / receiving unit 130 acquires video captured by the camera 22 in real time, and in the advance detection process for detecting registered objects, acquires video from the camera 22 at predetermined time intervals. For example, the transmitting / receiving unit 130 acquires video from the camera 22 from the time when video was last acquired from the camera 22 to the current time. The transmitting / receiving unit 130 adds the acquired video to the camera video 122-1 to update it.

[0044] In a lost object detection process for detecting an object in response to a detection request, the transmitting and receiving unit 130 receives a detection request and a tracking search from the smartphone 23, and transmits an image extracted by the image processing unit 140 from the camera image 122-1 to the smartphone 23. The transmitting and receiving unit 130 receives a detection request from the smartphone 23. For example, the transmitting and receiving unit 130 receives a detection request specifying an image of the object to be detected and a time period for detection from the smartphone 23. In a detection request for a registered object, information specifying the registered object registered in the registered object information 123-1 is set in the detection request, and in a detection request for an object other than the registered object (additional object), an image of the additional object is set in the detection request.

[0045] The transmitting / receiving unit 130 transmits to the smartphone 23 data for displaying a screen including the image extracted by the image processing unit 140 from the camera image 122-1 in response to the detection request, and causes the smartphone 23 to display the screen. The transmitting / receiving unit 130 receives from the smartphone 23 a tracking search request including a clipped image clipped from the image transmitted in response to the detection request. The transmitting / receiving unit 130 transmits to the smartphone 23 data for displaying a screen including the image extracted by the image processing unit 140 from the camera image 122-1 in response to the tracking search request, and causes the smartphone 23 to display the screen.

[0046] Image processing unit 140 performs image processing on camera image 122-1. In advance detection processing, when transmitting / receiving unit 130 updates camera image 122-1, image processing unit 140 detects registered objects and actions of a person placing an object from the image at the current time. Image processing unit 140 detects each registered object registered in registered object information 123-1 from the image at the current time of camera image 122-1. For example, image processing unit 140 detects features of objects shown in each image of the registered objects registered in registered object information 123-1 from the image at the current time of camera image 122-1.

[0047] Then, for the detected registration object, image processing unit 140 records a registration object stamp for each time that indicates that the object was detected from camera image 122-1. For example, image processing unit 140 records a registration object stamp corresponding to the registration object detected from the image from among the registration object stamps corresponding to the current time in detection information 124-1.

[0048] Furthermore, image processing unit 140 detects the action of a person placing an object from the image of camera image 122-1 at the current time. For example, image processing unit 140 inputs the image of camera image 122-1 at the current time into a trained model that outputs whether or not the input image shows the action of a person placing an object. Note that image processing unit 140 may also detect other actions in which a person lets go of an object, such as putting an object into another object. Then, when image processing unit 140 detects the action of a person placing an object, it records an action stamp for each time indicating that the action was detected from camera image 122-1. For example, image processing unit 140 records an action stamp corresponding to the current time in detection information 124-1.

[0049] In the lost item detection process, image processing unit 140 extracts from camera image 122-1 a portion that captures the object specified in the detection request. In the registered object detection request, image processing unit 140 identifies the time at which the action stamp and registered object stamp were recorded within the time period specified in the detection request. Image processing unit 140 extracts from camera image 122-1 the image of the most recent time at which the action stamp and registered object stamp were recorded. For example, image processing unit 140 extracts from camera image 122-1 an image that includes the time at which the identified action stamp and registered object stamp were recorded (e.g., an image for a predetermined time from that time).

[0050] In response to a request to detect an additional object, image processing unit 140 detects the additional object that appears in the image acquired in response to the detection request from the image at each time in camera video 122-1. For example, image processing unit 140 detects the object that appears in the image acquired in response to the detection request from the image at each time set in detection information 124-1 in camera video 122-1. Image processing unit 140 extracts from camera video 122-1 the image at the most recent time at which the additional object was detected from the image and an additional object stamp was recorded.

[0051] When the transmitting / receiving unit 130 receives a tracking search request, the image processing unit 140 detects the object appearing in the clipped image from the image at each time in the camera video 122-1. For example, the image processing unit 140 detects the object appearing in the clipped image from the image at each time set in the detection information 124-1 in the camera video 122-1. The image processing unit 140 extracts from the camera video 122-1 the image at the most recent time at which the object appearing in the clipped image was detected and an action stamp was recorded.

[0052] 4 indicate part of the communication paths, and communication paths other than those shown in the figure can also be set. Next, the information stored in storage unit 120 will be described in detail.

[0053] 5 is a diagram showing an example of member information. Member information 121 is information about members who receive services provided by server 100. A member ID and a member name are registered in member information 121. The member ID is an ID for identifying a member. The member name is the name of the member.

[0054] 6 is a diagram showing an example of registered item information. Registered item information 123-1, 123-2, ... is information related to registered items for each user. Note that, although the following description will be given of registered item information 123-1 corresponding to user 21, registered item information 123-2, ... also registers the same content as registered item information 123-1.

[0055] An object ID, a name, and an image ID are registered in the registered object information 123-1. The object ID is an ID for identifying a registered object. The name is the name of the registered object. The image ID is an ID for identifying an image in which the registered object appears. For example, an object is registered as a registered object in the registered object information 123-1 in response to a registration request from the smartphone 23. For example, the smartphone 23 transmits to the server 100 an image of the object to be registered as a registered object and the name of the object. Then, an unused object ID, the name transmitted from the smartphone 23, and an image ID for identifying the image transmitted from the smartphone 23 are registered in association with each other in the registered object information 123-1.

[0056] Furthermore, for example, an unused object ID, a name of the additional object, and an image ID that identifies an image of the additional object are associated and registered in the registered object information 123-1 when a detection request for an additional object is received from the smartphone 23. Furthermore, for example, when a tracking search request is received from the smartphone 23, an object that appears in the cut-out image is registered as an additional object in the registered object information 123-1.

[0057] 7 is a diagram showing an example of detection information. Detection information 124-1, 124-2, ... is information relating to the detection of objects from camera images 122-1, 122-2, ... respectively. Note that, although the detection information 124-1 corresponding to camera image 122-1 will be described below, the detection information 124-2, ... also registers the same content as the detection information 124-1.

[0058] The detection information 124-1 records a date, a time, a registered object stamp, an action stamp, and an added object stamp. The date is the date when the video was acquired from the camera 22. The time is the time when the video was acquired from the camera 22. The registered object stamp is recorded for each object ID of the registered object. The registered object stamp has a default value of "0" indicating that no registered object was detected, and if the corresponding registered object is detected at the corresponding time in the camera video 122-1, a default value of "1" indicating that the registered object was detected is recorded. The action stamp has a default value of "0" indicating that no action was detected, and if the action of a person placing an object is detected at the corresponding time in the camera video 122-1, a default value of "1" indicating that the action was detected is recorded.

[0059] An additional item stamp is recorded for each object ID of the additional item. The initial value of the additional item stamp is "0", which indicates that no additional item was detected, and if the corresponding additional item is detected at the corresponding time in camera image 122-1, "1", which indicates that the additional item was detected, is recorded. Next, a process for detecting an item lost by user 21 will be described.

[0060] 8 is a diagram showing an example of a lost item detection process. When server 100 receives a request to detect a lost item from smartphone 23, server 100 transmits an image of a portion of camera image 122-1 in which the lost item is captured to smartphone 23. Note that the following describes a case in which the lost item is a registered item that has been registered in advance in registered item information 123-1.

[0061] First, the smartphone 23, in response to an operation by the user 21, notifies the server 100 of the detection request data 41 and sends a detection request. The detection request data 41 includes an image, a date, a time period (start time), and a time period (end time). An image of the lost item or information identifying the image is set in the image of the detection request data 41. For example, the object ID of the lost item is set in the image of the detection request data 41. The date, time period (start time), and time period (end time) of the detection request data 41 are set to the date and time period when the user 21 believes the lost item was lost.

[0062] The image processing unit 140 refers to the detection information 124-1 to identify the time when the user 21 performed the action of placing the lost item. The image processing unit 140 refers to the registered object stamp and action stamp corresponding to the object ID specified in the detection request data 41 of the detection information 124-1 for the time period specified in the detection request data 41. The image processing unit 140 then identifies the most recent time among the times at which both the registered object stamp and the action stamp were "1". The image processing unit 140 extracts an image including the identified time (e.g., an image for a predetermined time from the identified time) from the camera image 122-1. The transmission / reception unit 130 then causes the smartphone 23 to display a screen 51 including the extracted image.

[0063] Screen 51 is a screen for notifying the user of the results of lost item detection. Screen 51 includes a message indicating that it is a screen for notifying the user of the results of lost item detection, such as "Search Results." Screen 51 also includes an image (search target image) extracted by image processing unit 140 from camera image 122-1. Screen 51 also includes the date and time period set in the detection request. Screen 51 includes a back button. When the back button is pressed, smartphone 23 ends display of screen 51. Screen 51 also includes a track and search button. When the track and search button is pressed, smartphone 23 sends a track and search request to server 100.

[0064] In this way, when the server 100 receives a detection request from the smartphone 23, it refers to the detection information 124-1 and identifies the time when the lost item and the action of the person placing the item were detected. Then, the server 100 extracts an image including the identified time from the camera image 122-1 and transmits it to the smartphone 23. This allows the server 100 to present to the user 21 an image of when the user 21 placed the lost item. The user 21 can search for the lost item based on the image on the screen 51.

[0065] However, if the image on screen 51 is an image of the lost item being placed in another item (for example, putting glasses in an eyeglass case), the location of the item containing the lost item cannot be identified from the image on screen 51, and the lost item cannot be found. Therefore, server 100 performs a tracking search to detect the item containing the lost item from camera image 122-1. Next, the setting process for the tracking search will be described.

[0066] 9 is a diagram showing an example of the setting process for tracking and searching. In response to a tracking and searching request from the smartphone 23, the server 100 acquires a cutout image obtained by cutting out a portion of the image on the screen 51 that shows the object containing the lost item.

[0067] When the smartphone 23 receives a press of the tracking search button on the screen 51, it displays an image 51a, which is a frame image of the video on the screen 51. For example, the image 51a is a frame image captured when the user 21 pauses playback of the video on the screen 51. The smartphone 23 receives an operation to select a range from within the image 51a. For example, the smartphone 23 receives a selection of coordinates of four points that form a rectangle within the image 51a. The smartphone 23 generates a cutout image by cutting out the selected range from the image 51a and transmits the cutout image to the server 100. The transmitter / receiver 130 acquires the cutout image from the smartphone 23 and additionally registers the acquired cutout image in the registered object information 123-1. The server 100 may acquire the time of the image 51a and information indicating the range selected from the image 51a from the smartphone 23, and generate the cutout image.

[0068] In this way, the server 100 acquires from the smartphone 23 a cutout image of the part of the image on the screen 51 where the lost item is placed. Here, the server 100 can accurately identify the object where the lost item is placed by acquiring a cutout image of the range selected by the user 21 from the image 51a. Next, a tracking search using the cutout image will be described.

[0069] 10 is a diagram showing an example of the tracking search process. In the tracking search, the server 100 extracts an image of when the user 21 places an object that appears in the cutout image from the camera image 122-1 and transmits the extracted image to the smartphone 23.

[0070] First, the image processing unit 140 generates tracking and retrieval request data 42 based on the tracking and retrieval request from the smartphone 23. The tracking and retrieval request data 42 includes an image, a date, a time period (start time), and a time period (end time). A clipped image or information identifying the clipped image is set in the image of the tracking and retrieval request data 42. For example, the object ID used when the clipped image was registered in the registered object information 123-1 is set in the image of the tracking and retrieval request data 42. Furthermore, the date, time period (start time), and time period (end time) of the tracking and retrieval request data 42 are set to the date and time period set in the detection request data 41.

[0071] First, image processing unit 140 detects an object appearing in the cutout image from camera video 122-1 and records the detection result in detection information 124-1. Image processing unit 140 detects an object appearing in the cutout image from images of camera video 122-1 at each time set in detection information 124-1. Then, image processing unit 140 sets the additional object stamp of the object ID of the cutout image corresponding to the time at which the object appearing in the cutout image was detected to "1."

[0072] The image processing unit 140 refers to the detection information 124-1 to identify the time when the user 21 placed the object shown in the clipped image. The image processing unit 140 refers to the additional object stamp and action stamp of the detection information 124-1 corresponding to the object ID specified in the tracking search request data 42 for the time period specified in the tracking search request data 42. The image processing unit 140 then identifies the most recent time at which both the additional object stamp and the action stamp were "1". The image processing unit 140 extracts an image including the identified time (e.g., an image for a predetermined time from the identified time) from the camera image 122-1. The transmission / reception unit 130 then causes the smartphone 23 to display a screen 52 including the extracted image.

[0073] Screen 52 is a screen for notifying the results of the tracking search. Screen 52 includes the same content as screen 51. Screen 52 also includes, as the search target image, an image extracted from camera image 122-1 by image processing unit 140 in the tracking search.

[0074] In this way, when the server 100 receives a tracking search request from the smartphone 23, it detects an object captured in the clipped image from the camera video 122-1 and identifies the time when the object and the person's action of placing an object were detected. Then, the server 100 extracts an image including the identified time from the camera video 122-1 and transmits it to the smartphone 23.

[0075] This allows the server 100 to present to the user 21 an image of the moment when the user 21 places the item containing the lost item. For example, the server 100 can present to the user 21 an image of the moment when the user 21 places the eyeglasses case containing the lost eyeglasses. The user 21 can search for the item containing the lost item based on the image on the screen 52. Thus, the server 100 can easily find the lost item that has been placed inside another item. Next, the procedure of the process executed by the server 100 will be described. First, the advance detection process will be described.

[0076] 11 is a flowchart showing an example of the procedure of the advance detection process. The process shown in FIG. 11 will be described below in order of step number. [Step S11] The transmitter / receiver 130 waits for a predetermined time.

[0077] [Step S12] The transmitter / receiver 130 acquires video from the camera 22. For example, the transmitter / receiver 130 acquires video from the camera 22 from the time when video was last acquired from the camera 22 to the current time. The transmitter / receiver 130 adds the acquired video to the camera video 122-1 to update it.

[0078] [Step S13] The image processing unit 140 detects each registered object registered in the registered object information 123-1 from the image of the camera video 122-1 at the current time. For example, the image processing unit 140 detects the features of objects appearing in each image whose image ID is registered in the registered object information 123-1 from the image of the camera video 122-1 at the current time.

[0079] [Step S14] The image processing unit 140 records the registration stamp of the current time for the registration detected in step S13. For example, the image processing unit 140 sets to "1" the registration stamp corresponding to the registration detected from the image in step S13 among the registration stamps corresponding to the current time in the detection information 124-1.

[0080] [Step S15] The image processing unit 140 detects an action of a person placing an object from the image of the camera image 122-1 at the current time. For example, the image processing unit 140 inputs the image of the camera image 122-1 at the current time to a trained model that outputs whether or not the input image shows an action of a person placing an object.

[0081] [Step S16] The image processing unit 140 determines whether or not a person's action of placing an object was detected in step S15. If the image processing unit 140 determines that a person's action of placing an object was detected, the process proceeds to step S17. If the image processing unit 140 determines that a person's action of placing an object was not detected, the process proceeds to step S11.

[0082] [Step S17] The image processing unit 140 records the action stamp of the current time. For example, the image processing unit 140 sets the action stamp corresponding to the current time of the detection information 124-1 to “1.” Then, the process proceeds to step S11.

[0083] In this way, the server 100 detects registered objects and the action of a person placing an object in advance. The server 100 acquires images from the camera 22 at predetermined time intervals to update the camera image 122-1, and detects each registered object from the image at the current time in the camera image 122-1. The server 100 then records the detection result for each registered object as a registered object stamp in the detection information 124-1. This allows the server 100 to shorten the time required to identify the time when a lost object, which is a registered object, was detected from the camera image 122-1 when a detection request is made. Next, the lost object detection process will be described.

[0084] 12 is a flowchart (part 1) showing an example of the procedure of the item-searching process. The process shown in FIG. 12 will be explained below in order of step number. [Step S21] The transmitter / receiver 130 receives a detection request from the smartphone 23. For example, the transmitter / receiver 130 receives, from the smartphone 23, detection request data 41 that specifies an image of an object to be detected and a time period during which the object is to be detected.

[0085] [Step S22] The transmitting / receiving unit 130 determines whether the detection request acquired in step S21 is a detection request for a registered object. For example, if the detection request data 41 includes an object ID, the transmitting / receiving unit 130 determines that it is a detection request for a registered object, and if it does not include an object ID but includes an image, the transmitting / receiving unit 130 determines that it is a detection request for an additional object. If the transmitting / receiving unit 130 determines that the detection request is a detection request for a registered object, the processing proceeds to step S23. If the transmitting / receiving unit 130 determines that the detection request is a detection request for an additional object, the processing proceeds to step S25.

[0086] [Step S23] The image processing unit 140 identifies the time at which the action stamp and the registration stamp were recorded during the time period specified in the detection request. For example, the image processing unit 140 refers to the registration stamp and the action stamp corresponding to the object ID specified in the detection request data 41 of the detection information 124-1 for the time period specified in the detection request data 41. Then, the image processing unit 140 identifies the time at which both the registration stamp and the action stamp were "1."

[0087] [Step S24] The image processing unit 140 extracts from the camera image 122-1 the image of the most recent time at which the action stamp and the registered object stamp are recorded. For example, the image processing unit 140 extracts from the camera image 122-1 the image including the time at which the action stamp and the registered object stamp identified in step S23 are recorded (for example, the image for a predetermined time from that time). Then, the process proceeds to step S30.

[0088] [Step S25] The image processing unit 140 additionally registers the image acquired in response to the detection request in the registered object information 123-1. For example, the image processing unit 140 associates an unused object ID, the name of the additional object, and an image ID that identifies the image of the additional object included in the detection request data 41, and registers them in the registered object information 123-1.

[0089] [Step S26] The image processing unit 140 detects additional objects from the images of the camera video 122-1 at each time. For example, the image processing unit 140 detects objects that appear in the images acquired in response to the detection request from the images of the camera video 122-1 at each time set in the detection information 124-1.

[0090] [Step S27] The image processing unit 140 records an additional object stamp for the time when the additional object was detected from the image. For example, the image processing unit 140 adds an additional object stamp corresponding to the object ID of the image of the additional object to the detection information 124-1. Then, the image processing unit 140 sets to "1" the additional object stamp that corresponds to the time when the object shown in the image acquired in response to the detection request in step S26 was detected from the image of the camera video 122-1, among the added additional object stamps.

[0091] [Step S28] The image processing unit 140 identifies the time at which the action stamp and additional object stamp were recorded within the time period specified in the detection request. For example, the image processing unit 140 references the additional object stamp and action stamp corresponding to the object ID of the image of the additional object in the detection information 124-1 for the time period specified in the detection request data 41. The image processing unit 140 then identifies the time at which both the additional object stamp and the action stamp were "1."

[0092] [Step S29] The image processing unit 140 extracts from the camera image 122-1 the image of the most recent time at which the action stamp and additional stamp were recorded. For example, the image processing unit 140 extracts from the camera image 122-1 the image including the most recent time at which the action stamp and additional stamp identified in step S28 were recorded (for example, the image for a predetermined time from that time).

[0093] [Step S30] The transmitter / receiver 130 transmits data for displaying the screen 51 including the video extracted in step S24 or step S29 to the smartphone 23, and causes the smartphone 23 to display the screen 51.

[0094] [Step S31] The transmitting / receiving unit 130 determines whether or not a tracking search request has been received from the smartphone 23. If the transmitting / receiving unit 130 determines that a tracking search request has been received, the processing proceeds to step S32. If the transmitting / receiving unit 130 determines that a tracking search request has not been received, the processing ends.

[0095] 13 is a flowchart (part 2) showing an example of the procedure of the item-searching process. The process shown in FIG. 13 will be explained below in order of step number. [Step S32] The image processing unit 140 acquires the clipped image included in the tracking search request.

[0096] [Step S33] The image processing unit 140 additionally registers the clipped image in the registered object information 123-1. For example, the image processing unit 140 associates an unused object ID, the name of the object appearing in the clipped image specified in the tracking search request, and an image ID that identifies the clipped image, and registers them in the registered object information 123-1.

[0097] [Step S34] The image processing unit 140 detects an object appearing in the cutout image from the image of the camera video 122-1 at each time. For example, the image processing unit 140 detects an object appearing in the cutout image from the image of the camera video 122-1 at each time set in the detection information 124-1.

[0098] [Step S35] The image processing unit 140 records an additional object stamp for the time when the object shown in the clipped image was detected from the image. For example, the image processing unit 140 adds an additional object stamp corresponding to the object ID of the clipped image to the detection information 124-1. Then, the image processing unit 140 sets to "1" the additional object stamp among the added additional object stamps that corresponds to the time when the object shown in the clipped image was detected from the image of the camera video 122-1 in step S34.

[0099] [Step S36] The image processing unit 140 identifies the time at which the action stamp and the additional object stamp were recorded within the time period specified in the detection request. For example, the image processing unit 140 references the additional object stamp and the action stamp corresponding to the object ID of the clipped image in the detection information 124-1 for the time period specified in the detection request data 41. The image processing unit 140 then identifies the time at which both the additional object stamp and the action stamp were "1."

[0100] [Step S37] The image processing unit 140 extracts from the camera image 122-1 the image of the most recent time at which the action stamp and additional stamp are recorded. For example, the image processing unit 140 extracts from the camera image 122-1 the image including the most recent time at which the action stamp and additional stamp identified in step S36 are recorded (for example, the image for a predetermined time from that time).

[0101] [Step S38] The transmitter / receiver 130 transmits data for displaying the screen 52 including the video extracted in step S37 to the smartphone 23, and causes the smartphone 23 to display the screen 52. Then, the process proceeds to step S31.

[0102] In this way, the server 100 detects the object shown in the image specified in the detection request from the image taken in the time period specified in the detection request of the camera video 122-1. Then, the server 100 transmits to the smartphone 23 a video including the object shown in the image specified in the detection request and an image in which the action of a person placing an object is detected. This allows the server 100 to present to the user 21 the situation when the lost item was separated from the user 21.

[0103] Note that server 100 detects a registered object in advance from camera image 122-1 in advance detection processing. Therefore, when a registered object is specified in a detection request, server 100 can identify the time when the object specified in the detection request was detected simply by referring to the registered object stamp in detection information 124-1, thereby shortening the time it takes to transmit the image to smartphone 23. Furthermore, even if the object specified in the detection request is not a registered object, server 100 can detect the object from camera image 122-1 when receiving the detection request, and can also transmit an image to smartphone 23 to assist in finding the unregistered object.

[0104] When the server 100 receives the tracking and search request, it acquires a clipped image clipped from the video transmitted to the smartphone 23. The server 100 detects the object shown in the clipped image from the image captured in the camera video 122-1 during the time period specified in the detection request. The server 100 then transmits to the smartphone 23 a video including an image in which the object shown in the clipped image and an action of a person placing an object are detected. This allows the server 100 to present to the user 21 the situation when the item containing the lost item that the user 21 is searching for was removed from the user 21. Therefore, the server 100 can assist in finding the lost item even if the lost item was placed in an object shown in the clipped image and carried away, and can facilitate the finding of a lost item that has been placed in another item.

[0105] The server 100 also acquires a cutout image generated by the user 21 selecting a range of the image 51a included in the video transmitted to the smartphone 23. This allows the user 21 to specify the object, so the server 100 can accurately identify the other object into which the lost item was placed.

[0106] Although the embodiments have been described above, the configuration of each part shown in the embodiments can be replaced with other parts having similar functions. Also, any other components or processes may be added. Furthermore, any two or more configurations (features) of the above-described embodiments may be combined. [Explanation of symbols]

[0107] 1. First Search Image 2. First detected image 3. Second Search Image 4 Second detected image 10. Information processing equipment 11 Storage section 11a Camera footage 12 Processing section

Claims

1. a storage unit for storing camera images; a processing unit that detects a first object appearing in a first search image from images of the camera video at a plurality of times, outputs a first detected image in which the first object is detected, identifies a second search image included in the first detected image, detects a second object appearing in the second search image from images of the camera video at a plurality of times, and outputs a second detected image in which the second object is detected; An information processing device having the above.

2. the processing unit identifies the second object-searched image based on a range selection operation of the user on the first detected image.

2. The information processing device according to claim 1.

3. the processing unit detects a predetermined motion of a person from images of the camera video at a plurality of times, outputs the first detected image in which the first object is detected and the predetermined motion is detected, and outputs the second detected image in which the second object is detected and the predetermined motion is detected.

2. The information processing device according to claim 1.

4. the processing unit executes a process of acquiring an image from a camera and updating the camera image, detecting the first object from an image of the camera image at a current time, and recording a stamp indicating whether the first object has been detected in association with the current time at predetermined time intervals, and upon receiving a request to detect the first object, outputs the image of the camera image at the time the stamp was recorded as the first detected image.

2. The information processing device according to claim 1.

5. The computer Detecting a first object appearing in a first object-searching image from images captured by a camera at a plurality of times; outputting a first detected image in which the first object is detected, and identifying a second object-searched image included in the first detected image; Detecting a second object appearing in the second object-searching image from images of the camera video taken at a plurality of times; outputting a second detected image in which the second object is detected; Detection method.

6. On the computer, Detecting a first object appearing in a first object-searching image from images captured by a camera at a plurality of times; outputting a first detected image in which the first object is detected, and identifying a second object-searched image included in the first detected image; Detecting a second object appearing in the second object-searching image from images of the camera video taken at a plurality of times; outputting a second detected image in which the second object is detected; The detection program that causes the process to run.

Images