Suspicious person detection device and suspicious person detection method

Abstract

To provide a technique for detecting a suspicious person even in a situation where his / her face is not easily recognized.SOLUTION: A video acquisition section 310 acquires a video obtained by photographing the periphery. An extraction section 312 extracts a person 10 from the acquired video. An identification information acquisition section 322 acquires identification information to identify a terminal device 100 being a transmission source of a signal from the signal received by wireless communication at video acquisition timing. A storage section 340 stores first information of the person 10 and the identification information by association. When a processing section 330 comes to be in a communication impossible state with the terminal device 100 of the stored identification information and then comes to be in a communication possible state again, the processing section acquires the identification information being the same as the identification information stored in the storage section 340 again. In this case, the processing section compares second information of the person 10 extracted from a video photographed after the communication state becomes possible with the first information, so as to identify at least one person between the person 10 of the first information and the person 10 of the second information as a suspicious person.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a suspicious person detection technique, and more particularly to a suspicious person detection device and a suspicious person detection method for identifying a person in a captured video as a suspicious person. [Background technology]

[0002] A loiter detection system uses a surveillance camera to monitor the surroundings of a space such as an apartment building that is used mainly by specific people. The loiter detection system detects face images from input video and registers features extracted from the face images in a database. The loiter detection system also searches for features similar to the features of the face image to be judged among the features registered in the database, and judges whether the person in the face image to be judged is a suspicious person or not based on the number of features with a similarity higher than a predetermined value (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO 14 / 132841 Summary of the Invention [Problem to be solved by the invention]

[0004] Even if a person is captured by a surveillance camera or a drive recorder, it is difficult to detect a suspicious person if the person is wearing sunglasses or a mask that makes it difficult to see the face. Therefore, it is necessary to detect suspicious people even in situations where the face is difficult to see.

[0005] The present invention has been made in view of the above circumstances, and has as its object to provide a technique for detecting a suspicious person even in a situation in which the face is difficult to discern. [Means for solving the problem]

[0006] In order to solve the above problems, a suspicious person detection device of one embodiment of the present invention comprises an image acquisition unit that acquires an image of the surroundings of a protected object, an extraction unit that extracts a person from the image acquired by the image acquisition unit, an identification information acquisition unit that acquires, from a signal received via wireless communication at the time the image acquisition unit acquires the image, identification information for identifying a wireless device that is the sender of the signal, a memory unit that associates and stores first information of the person extracted by the extraction unit and the identification information acquired by the identification information acquisition unit, and a processing unit that, when the identification information acquisition unit again acquires identification information identical to the identification information stored in the memory unit by causing a communication connection with the wireless device of the identification information stored in the memory unit to become unavailable after the wireless device becomes available for communication again, compares the second information of the person extracted by the extraction unit from the image acquired by the video acquisition unit with the first information, and identifies at least one of the person in the first information and the person in the second information as a suspicious person depending on the comparison result.

[0007] Another aspect of the present invention is a method for detecting a suspicious person, which includes the steps of: acquiring a video of the periphery of a protection target; extracting a person from the acquired video; acquiring identification information for identifying a wireless device that is a source of a signal from a signal received by wireless communication at the timing of acquiring the video; storing first information of the extracted person and the acquired identification information in association with each other; and, when a communication connection with the wireless device of the stored identification information becomes unavailable after the wireless device becomes available for communication again and thereby the same identification information as the stored identification information is acquired again, comparing second information of the person extracted from the video captured after the wireless device becomes available for communication again with the first information, and identifying at least one of the person of the first information and the person of the second information as a suspicious person according to the comparison result.

[0008] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also effective as aspects of the present invention. Effect of the Invention

[0009] According to the present invention, a suspicious person can be detected even in a situation where the face is difficult to discern. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing a configuration of a vehicle according to an embodiment. [Diagram 2] 2 is a diagram showing a configuration of the suspicious person detection device shown in FIG. 1; [Diagram 3] 3(a) and 3(b) are diagrams showing images acquired by the image acquisition unit of FIG. [Figure 4] 3 is a diagram showing a data structure of a database stored in a storage unit in FIG. 2. [Diagram 5] 3 is a flowchart showing a storage procedure performed by the suspicious person detection device of FIG. 2. [Figure 6] 3 is a flowchart showing a detection procedure performed by the suspicious person detection device of FIG. 2. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Before describing the present invention in detail, an outline will be given first. The embodiment of the present invention relates to a suspicious person detection device that is mounted on a vehicle as an example of a protected object and detects a suspicious person in the vehicle. A suspicious person is, for example, a person who may cause harm to the vehicle. As described above, even if a person is extracted from an image captured by an imaging device, it is difficult to detect the suspicious person if the person is wearing sunglasses or a mask that makes it difficult to see the face of the person.

[0012] In order to deal with such a situation, the suspicious person detection device according to the present embodiment utilizes the fact that people generally carry a terminal device such as a smartphone, and that suspicious people tend to inspect the vehicle they are going to harm and then return to the vehicle to harm it. The suspicious person detection device extracts the person from the captured video, obtains the identification information of the terminal device from a signal received by wireless communication, and stores the person information and the identification information in association with each other.

[0013] If the person is suspicious, he or she will leave the vehicle and return. Therefore, when the suspicious person detection device becomes unable to connect to a terminal device and then becomes able to connect to a terminal device again, the suspicious person detection device receives a signal from the terminal device. When the identification information contained in the signal received after the terminal device becomes able to connect to a terminal device again matches the stored identification information, the suspicious person detection device identifies the person carrying the terminal device as a suspicious person. In this case, the suspicious person detection device compares the person extracted from the video captured after the terminal device becomes able to connect to a terminal device again with the stored people, and identifies at least one person as a suspicious person according to the comparison result.

[0014] FIG. 1 shows a configuration of a vehicle 1000. The vehicle 1000 is parked in a parking lot, for example, and includes an imaging device 200 and a suspicious person detection device 300. A person 10 carries a terminal device 100 such as a smartphone, and is present around the vehicle 1000. The terminal device 100 can also be said to be a wireless device. The person 10 may be a suspicious person who may harm the vehicle 1000, or a person other than a suspicious person (hereinafter referred to as a "normal person"). The terminal device 100 is capable of performing wireless communication by a mobile phone communication system such as 4G or 5G, and is also capable of performing wireless communication by a short-range wireless communication system such as Bluetooth (registered trademark). The terminal device 100 may also be capable of performing wireless communication by a wireless LAN (Local Area Network).

[0015] The imaging device 200 of the vehicle 1000 captures an image of the surroundings of the vehicle 1000. In FIG. 1, the imaging device 200 captures an image of the area in front of the vehicle 1000, but it may capture an image in another direction. The imaging device 200 outputs an image of the surroundings to the suspicious person detection device 300. The suspicious person detection device 300 acquires the image from the imaging device 200. The suspicious person detection device 300 is also capable of performing wireless communication using a short-range wireless communication system, and receives a signal from the terminal device 100 via wireless communication. The suspicious person detection device 300 judges whether or not the person 10 is a suspicious person based on the acquired image and the received signal.

[0016] When the suspicious person detection device 300 determines that the person is a suspicious person, it outputs an alarm sound to the outside. The suspicious person detection device 300 can also perform wireless communication by a mobile phone communication system, and can communicate with the server 600 via the base station device 400 and the network 500 by wireless communication. The suspicious person detection device 300 transmits a report of the suspicious person to the server 600.

[0017] 2 shows the configuration of the suspicious person detection device 300. The suspicious person detection device 300 includes a video acquisition unit 310, an extraction unit 312, a first communication unit 320, an identification information acquisition unit 322, a processing unit 330, a storage unit 340, an alarm unit 350, and a second communication unit 360.

[0018] The image acquisition unit 310 is connected to the imaging device 200, and acquires an image of the surroundings of the vehicle 1000 from the imaging device 200. If a person 10 is present around the vehicle 1000, the person 10 will be included in the image. FIGS. 3(a)-(b) show images acquired by the image acquisition unit 310. FIG. 3(a) is an image captured in the daytime, for example, and includes the person 10 as described above. FIG. 3(b) will be described later, and we will return to FIG. 2.

[0019] The extraction unit 312 extracts the person 10 from the video by executing edge processing of image recognition on the video acquired by the video acquisition unit 310. A known technique may be used for the edge processing of image recognition. Furthermore, the feature amounts of the eyes, nose, and mouth are also acquired by image recognition. When the extraction unit 312 extracts the person 10, it outputs the video including the person 10 and the feature amounts of the eyes, nose, and mouth to the processing unit 330 as information of the person 10 (hereinafter referred to as "person information").

[0020] The first communication unit 320 executes wireless communication by a short-range wireless communication system, for example, Bluetooth (registered trademark) 5.1. The first communication unit 320 receives a signal, for example, an advertising signal, from the terminal device 100 by wireless communication. The first communication unit 320 also includes a plurality of antennas and estimates the strength and direction of the received advertising signal. Since a known technique may be used for estimating the strength and direction, a description thereof will be omitted here.

[0021] The identification information acquisition unit 322 receives the advertising signal received by the first communication unit 320 and the information on the intensity and direction estimated by the first communication unit 320. The identification information acquisition unit 322 acquires, from the advertising signal, identification information for identifying the terminal device 100 that is the transmission source of the advertising signal, at the timing when the video acquisition unit 310 acquires the video. The identification information is, for example, a public address, but may also be a random address (static device address). The identification information acquisition unit 322 outputs the identification information and the information on the intensity and direction to the processing unit 330.

[0022] The processing unit 330 controls the operation of the suspicious person detection device 300. The processing unit 330 receives the person information from the extraction unit 312, and also receives the identification information and the intensity and direction information from the identification information acquisition unit 322. The processing unit 330 associates the identification information with the person information and stores them in the storage unit 340. Fig. 4 shows the data structure of the database stored in the storage unit 340. As shown in the figure, the imaging time, the identification information, and the person information are stored in an associated manner. Return to Fig. 2.

[0023] When the extraction unit 312 extracts multiple persons 10 from the video, the processing unit 330 may execute the following process. Since the imaging device 200 is fixed to the vehicle 1000, the position in the video is associated with the relative distance and direction with respect to the vehicle 1000. The processing unit 330 specifies the relative distance and direction of each person 10 with respect to the vehicle 1000 from the position of each of the multiple persons 10 in the video. Furthermore, the processing unit 330 links the person 10 with the identification information based on the specified relative distance and direction and the strength and direction information received together with the identification information.

[0024] As described above, if the person 10 is a suspicious person, he or she tends to return to the vehicle 1000 after previewing the vehicle 1000. For example, if the person 10 leaves the vehicle 1000 after previewing the vehicle 1000 during the day, the first communication unit 320 of the suspicious person detection device 300 stops receiving advertising signals from the terminal device 100. If the identification information acquisition unit 322 does not receive advertising signals from the terminal device 100 that was receiving advertising signals for a certain period of time, it determines that the state has shifted from a state in which communication connection with the terminal device 100 is possible to a state in which communication connection is not possible.

[0025] In such a situation, when the person 10 returns to the vehicle 1000, for example, at night, the first communication unit 320 receives the advertising signal from the terminal device 100 again. When the identification information acquisition unit 322 acquires again the identification information from the terminal device 100 that has been determined to have transitioned to a communication connection unavailable state, it determines that the terminal device 100 has transitioned from a communication connection unavailable state to a communication connection available state. The identification information acquisition unit 322 outputs the identification information, the strength and direction information to the processing unit 330, and also outputs a flag to the processing unit 330 to indicate that the terminal device 100 has returned to a communication connection available state again. In order to output the flag, the identification information acquisition unit 322 may store the identification information or may check the identification information stored in the storage unit 340. In other words, the identification information acquisition unit 322 acquires the identification information again by becoming a communication connection available state again after becoming a communication connection unavailable state with the terminal device 100 of the identification information stored in the storage unit 340.

[0026] The video acquisition unit 310 acquires video from the imaging device 200. Fig. 3(b) shows video captured at night, for example. Return to Fig. 2. The extraction unit 312 extracts the person 10 from the video, as in the previous cases, and also acquires feature amounts of the eyes, nose, and mouth. After the extraction unit 312 becomes capable of communication connection again, it outputs the person information of the extracted person 10 to the processing unit 330.

[0027] The processing unit 330 receives the person information from the extraction unit 312, and also receives the identification information, the flag, and the information on the strength and direction from the identification information acquisition unit 322. When the processing unit 330 receives the flag, it searches the database stored in the storage unit 340 for identification information identical to the received identification information. When the processing unit 330 detects the identification information, it extracts the person information associated with the identification information from the database stored in the storage unit 340. Here, when the person information extracted from the database is called "first information", the person information received by the processing unit 330 after the terminal device 100 becomes in a communication connectable state again is called "second information". In other words, the processing unit 330 acquires the second information of the person 10 after acquiring the same identification information as the identification information stored in the storage unit 340 again by becoming in a communication connectable state again after becoming in a communication connectable state with the terminal device 100 of the identification information stored in the storage unit 340 in an inaccessible state.

[0028] When the processing unit 330 extracts from the database the same identification information as the received identification information, it determines that the person 10 carrying the terminal device 100 that transmitted the identification information is a suspicious person. In addition, the processing unit 330 compares the person 10 of the second information with the person 10 of the first information, and identifies at least one of them as a suspicious person according to the comparison result. For example, the processing unit 330 compares the sum of the feature amounts of the eyes, nose, and mouth of the person 10 of the first information (hereinafter referred to as the "first sum") with the sum of the feature amounts of the eyes, nose, and mouth of the person 10 of the second information (hereinafter referred to as the "second sum"). When the first sum is equal to or greater than the second sum, the processing unit 330 identifies the person 10 of the first information as a suspicious person. On the other hand, when the first sum is smaller than the second sum, the processing unit 330 identifies the person 10 of the second information as a suspicious person. The processing unit 330 may identify both the person 10 of the first information and the person 10 of the second information.

[0029] When the processing unit 330 determines that the person 10 is a suspicious person, it causes the alarm unit 350 to output an alarm sound. The alarm unit 350 may also output an announcement such as "Recording has started." Furthermore, the processing unit 330 outputs to the second communication unit 360 a video including the identified person 10 and information including the identification information (hereinafter referred to as "suspicious person information").

[0030] The second communication unit 360 can also perform wireless communication by a mobile phone communication system, and can communicate with the base station device 400 via wireless communication. As shown in Fig. 1, the base station device 400 can communicate with a server 600 via a network 500. The second communication unit 360 transmits suspicious person information, i.e., a report of a suspicious person, to the base station device 400 and the server 600 via the network 500 via wireless communication.

[0031] Here, identification information of a person 10 who is determined in advance not to be a suspicious person, for example, the user of the vehicle 1000, a family member, an acquaintance, or the like, on the terminal device 100, may be registered in advance in the suspicious person detection device 300. The storage unit 340 does not store identification information that has been registered in advance.

[0032] This configuration can be realized in hardware terms by the CPU, memory, and other LSIs of any computer, and in software terms by programs loaded into memory, but here we depict functional blocks realized by the cooperation of these. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by hardware alone, software alone, or a combination of both.

[0033] The operation of the suspicious person detection device 300 configured as above will be described. Fig. 5 is a flowchart showing a storage procedure by the suspicious person detection device 300. The processing unit 330 acquires person information from the extraction unit 312 (S10). The processing unit 330 acquires identification information from the identification information acquisition unit 322 (S12). The processing unit 330 associates the person information with the identification information and stores them in the storage unit 340 (S14).

[0034] 6 is a flowchart showing a detection procedure by the suspicious person detection device 300. When the state transitions from a communication connectable state to a communication unconnectable state to a communication connectable state with the terminal device 100 in this order (Y in S50), the processing unit 330 acquires person information from the extraction unit 312 (S52). The processing unit 330 acquires identification information from the identification information acquisition unit 322 (S54). When the same identification information has been acquired (Y in S56), the processing unit 330 detects a suspicious person (S58). When the same identification information has not been acquired (N in S56), step 58 is skipped. When the state transitions from a communication connectable state to a communication unconnectable state to a communication connectable state with the terminal device 100 in this order (N in S50), steps 52 to 58 are skipped.

[0035] According to this embodiment, the first information of a person and the identification information of a terminal device are linked and stored, and the second information of the person and the identification information of the terminal device are obtained when the communication connection with the terminal device becomes possible after the communication connection with the terminal device becomes impossible. If the identification information is the same, the person is identified as a suspicious person, so that a suspicious person can be detected even in a situation where the face is difficult to see. In addition, since the first information of the person and the identification information of the terminal device, and the second information of the person and the identification information of the terminal device are used, a person who returns after inspecting a vehicle can be detected as a suspicious person. In addition, since a person who returns after inspecting a vehicle is detected as a suspicious person, the detection accuracy can be improved.

[0036] In addition, the person in the second information is compared with the person in the first information based on the feature amount, and at least one person is identified as a suspicious person according to the comparison result, so that the face of the suspicious person can be easily identified. Furthermore, the person information and the identification information are linked based on the strength and direction of the received signal and the position of the person in the video, so that even if multiple people are included in the video, the person information and the identification information can be linked. Furthermore, the identification information registered in advance is excluded from the storage unit, so that processing can be omitted for people who cannot be suspicious. Furthermore, since processing is omitted for people who cannot be suspicious, the amount of processing can be reduced. Furthermore, the first information of the person and the second information of the person for the suspicious person are compared, and at least one of them is sent to the server according to the comparison result, so that the suspicious person can be notified.

[0037] The present invention has been described above based on the embodiments. These embodiments are merely illustrative, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention.

[0038] In this embodiment, a short-distance wireless communication system is used between the terminal device 100 and the suspicious person detection device 300. However, the present invention is not limited to this, and for example, a wireless LAN is used between the terminal device 100 and the suspicious person detection device 300. In this case, the identification information acquisition unit 322 acquires identification information included in a wireless LAN signal and for identifying the terminal device 100. According to this modification, the degree of freedom of the configuration can be improved.

[0039] In this embodiment, the imaging device 200 and the suspicious person detection device 300 are mounted on the vehicle 1000. However, the present invention is not limited to this, and for example, the imaging device 200 and the suspicious person detection device 300 may be installed in a place other than the vehicle 1000. The object to be protected is not limited to a vehicle, but also includes objects that require protection such as houses and cultural properties. According to this modified example, the degree of freedom of configuration can be improved. [Explanation of symbols]

[0040] 10 person, 100 terminal device, 200 imaging device, 300 suspicious person detection device, 310 video acquisition unit, 312 extraction unit, 320 first communication unit, 322 identification information acquisition unit, 330 processing unit, 340 memory unit, 350 alarm unit, 360 second communication unit, 400 base station device, 500 network, 600 server, 1000 vehicle.

Claims

1. an image acquisition unit that acquires an image of the surroundings of the protection object; an extraction unit that extracts a person from the image acquired by the image acquisition unit; an identification information acquisition unit that acquires, at a timing when the image acquisition unit acquires an image, from a signal received by wireless communication, identification information for identifying a wireless device that is a transmission source of the signal; a storage unit that stores first information of the person extracted by the extraction unit and the identification information acquired by the identification information acquisition unit in association with each other; a processing unit that, when the identification information acquisition unit acquires again the same identification information as the identification information stored in the storage unit by the identification information acquisition unit being in a communication-connectable state after the communication-connectable state with the wireless device having the identification information stored in the storage unit becomes unavailable, compares second information of the person extracted by the extraction unit from the video acquired by the video acquisition unit with the first information, and identifies at least one of the person in the first information and the person in the second information as a suspicious person according to a comparison result; A suspicious person detection device comprising:

2. 2. The suspicious person detection device according to claim 1, wherein the first information and the identification information of the person stored in the memory unit are linked based on the strength and direction of a received signal and the position of the person in the video.

3. The suspicious person detection device according to claim 1 , wherein the storage unit excludes identification information registered in advance from the storage unit.

4. Further comprising a communication unit capable of communicating with the server, 4. The suspicious person detection device according to claim 1, wherein the communication unit transmits at least one of first person information and second person information for the suspicious person identified by the processing unit to the server.

5. acquiring an image of the surroundings of the object to be protected; Extracting a person from the captured video; acquiring, from a signal received via wireless communication at a timing when the image is acquired, identification information for identifying a wireless device that is a transmission source of the signal; storing the extracted first information of the person in association with the acquired identification information; a step of comparing second information of a person extracted from a video captured after the communication connection with the wireless device of the stored identification information becomes unavailable and then becomes available for communication again, thereby acquiring the same identification information as the stored identification information again, with the first information, and identifying at least one of the person of the first information and the person of the second information as a suspicious person according to a comparison result; A method for detecting a suspicious person comprising the steps of:

Images