Methods, information processing systems, and programs

Abstract

To more reliably determine whether a visitor is a desirable visitor or a suspicious visitor.SOLUTION: A method according to the present disclosure causes a processor to analyze a captured image obtained by capturing an image of a subject who is a visitor, identify the visitor corresponding to the subject based on an analysis result of the captured image and obtain information on the identified visitor, which includes information on the evaluation of the visitor, and display the captured image and the information on the evaluation of the visitor together on a display unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a method, an information processing system, and a program.

Background Art

[0002] From the viewpoints of crime prevention and security, technologies have been developed to analyze images captured by cameras to detect whether a person is suspicious or not. For example, Patent Document 1 discloses a technology for determining in real time a suspicious behavior of a monitoring target projected onto a video captured by a video camera.

Prior Art Documents

Patent Documents

[0003] [[ID= 21]] [[ID=2 3]]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, for example, when a visitor comes to the entrance of an apartment building, it is difficult to determine whether the visitor is a preferred visitor or a suspicious person based only on the video displayed on an intercom.

[0005] Therefore, the present disclosure has been made in view of the above problems, and an object thereof is to provide a method, an information processing system, and a program capable of more reliably determining whether a visitor is a preferred visitor or a suspicious person.

Means for Solving the Problems

[0006] According to this disclosure, a method is provided which includes: a processor analyzing an image obtained by imaging a subject that is a visitor; identifying the visitor corresponding to the subject based on the results of the analysis of the image, and obtaining information about the identified visitor; and displaying the image and the visitor information together on a display unit, with the visitor information including evaluation information about the visitor.

[0007] Furthermore, according to this disclosure, an information processing system is provided, comprising: an analysis unit that analyzes an image obtained by capturing a subject that is a visitor; a visitor information acquisition unit that identifies the visitor corresponding to the subject based on the analysis results of the image and acquires information about the identified visitor; and a display control unit that displays the image and the visitor evaluation information together on a display unit, with the visitor information including evaluation information about the visitor.

[0008] Furthermore, according to this disclosure, a program is provided that causes a computer to function as: an analysis unit that analyzes captured images obtained by imaging a subject that is a visitor; a visitor information acquisition unit that identifies the visitor corresponding to the subject based on the analysis results of the captured images and acquires information about the identified visitor; and a display control unit that displays the captured images and the visitor evaluation information together on a display unit, with the visitor information including evaluation information about the visitor. [Effects of the Invention]

[0009] According to this disclosure, it is possible to more reliably determine whether a visitor is a desirable visitor or a suspicious person. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example configuration of System 1 according to one embodiment of the present disclosure. [Figure 2] This block diagram shows the configuration of server 10 according to the same embodiment. [Figure 3]This figure shows an example of the software configuration of the server 10 according to the same embodiment. [Figure 4] The figure shows an example of the processing conditions by the visitor information acquisition unit 102 according to the same embodiment. [Figure 5] This figure shows an example of the display mode on the display device by the display control unit 103 according to the same embodiment. [Figure 6] This flowchart shows an example of the information processing flow by the server 10 according to the same embodiment. [Modes for carrying out the invention]

[0011] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0012] Figure 1 is a diagram showing an example configuration of System 1 according to one embodiment of the present disclosure. As shown in the figure, System 1 according to this embodiment comprises a cloud server 10 (hereinafter referred to as Server 10), a display device 20, and a monitoring device 30. Server 10, display device 20, and monitoring device 30 are connected to each other via a network so as to be communicative. A user terminal 21, such as a smartphone, used by a resident HR of a room equipped with the display device 20 can communicate with Server 10, etc. Note that this configuration is just an example, and one configuration may combine other configurations, or other configurations may be included. In addition, in this embodiment, Server 10 has a configuration having the main functions described later, but some or all of these main functions may be provided by the intercom (display device) 20. That is, System 1 according to this embodiment may consist of only Server 10 or Display Device 20, or it may be a configuration in which Server 10 and Display Device 20 are jointly configured.

[0013] In this embodiment, the network NW is assumed to be the Internet. The network NW is constructed using, for example, a public telephone network, a mobile phone network, a wireless communication network, Ethernet (registered trademark), etc.

[0014] Server 10 is a computer that, for example, manages information about visitors to an apartment building or office building, or processes information about visitors to an apartment building or office building that outputs information about visitors to an apartment building or office building that is used to display information about residents. Such server 10 may be managed, for example, by the manager of the apartment building or office building, or by an organization that provides services to such manager.

[0015] Furthermore, for example, server 10 may be a general-purpose computer such as a workstation or personal computer, or it may be logically implemented through cloud computing.

[0016] The display device 20 is, for example, a display terminal installed in the living room of resident HR. Similar to a typical intercom, it is installed to communicate with an intercom or monitoring device 30 installed in, for example, the entrance of an apartment building or other building, and displays captured images, including the face of the visitor VR, acquired by the monitoring device 30, and allows for communication with the visitor VR. Furthermore, the display device 20 according to this embodiment can display information about the visitor VR that is processed and output by the server 10. For example, the display device 20 may be a typical intercom equipped with a processor, display, microphone or speaker, or it may be a smartphone, tablet computer or personal computer, etc. Alternatively, a user terminal 21 such as a smartphone, tablet computer or personal computer owned by resident HR may perform the functions of the display device 20.

[0017] The monitoring device 30 is a device that may include an imaging device such as a camera and a sound collection device such as a microphone. For example, the monitoring device 30 may be an intercom provided at the entrance of an apartment or a building, or may be a security camera (CCTV) or the like.

[0018] The outline of the system 1 according to this embodiment will be described. The monitoring device 30 captures an image of a visitor VR who has visited an apartment or the like, and the obtained captured image is transmitted to the server 10. The server 10 analyzes such a captured image to identify the visitor VR and obtains information on the visitor VR by introducing the visitor database. The information on the visitor VR, although details will be described later, may include, for example, the number of visits of the visitor VR, the evaluation (rating) of the visitor VR, the attributes of the visitor VR (such as a delivery company, business, individual friend, etc.), text information regarding the visitor VR, and the like. Such information on the visitor VR may be information registered by other residents living in the same apartment, for example, via the user terminal 21. On the display device 20 in the living room of the resident HR, information P2 regarding the above-mentioned visitor VR can be displayed together with the captured image P1. Thereby, for example, even if the visitor VR is a person unknown to the resident HR, the resident HR can determine whether to invite the visitor VR, and security can be improved. Hereinafter, the details of this embodiment will be described.

[0019] FIG. 2 is a block diagram showing the configuration of the server 10 according to this embodiment. As shown in the figure, the server 10 includes a control unit 11.

[0020] The processor 11a is an arithmetic unit that controls the operation of the control unit 11 and performs controls for data transmission and reception between each element and processes necessary for program execution. In this embodiment, this processor 11a is, for example, a CPU (Central Processing Unit), and executes a program stored in the storage 11c described later and developed in the memory 11b to perform each process.

[0021] Memory 11b includes a main memory device composed of a volatile memory device such as DRAM (Dynamic Random Access Memory), and an auxiliary memory device composed of a non-volatile memory device such as a flash memory or an HDD (Hard Disc Drive). This memory 11b is used as the working area of the processor 11a, while storing a bootloader executed at the startup of the control unit 11 and various setting information.

[0022] Storage 11c stores programs and information used for various processes. For example, a program for displaying information may be stored in the storage 11c.

[0023] The transceiver unit 11d connects the control unit 11 to a network such as the Internet, and may be equipped with a communication interface such as a local area network (LAN), a wide area network (WAN), infrared, wireless, WiFi, a point-to-point (P2P) network, a telecommunications network, cloud communication, LTE, Bluetooth (registered trademark), or BLE (Bluetooth Low Energy).

[0024] The input / output unit 11e is an interface to which input / output devices are connected, and may be provided so as to be connectable to, for example, an interphone (display device) 20.

[0025] The bus 11f transmits, for example, address signals, data signals, and various control signals among the connected processor 11a, memory 11b, storage 11c, transceiver unit 11d, and input / output unit 11e.

[0026] The display device 20 is an example of a display unit, and displays videos and images. The display device 20 may be a device such as an intercom, monitor, tablet, mobile terminal display, or touch panel. The display device 20 may also consist of a projector and a projection surface. The hardware configuration of the display device 20 may be the same as that of the server 10 shown in Figure 2.

[0027] Figure 3 shows an example of the software configuration of the server 10 according to this embodiment. The server 10 includes an analysis unit 101, a visitor information acquisition unit 102, a display control unit 103, an evaluation information input unit 104, and a visitor DB (database) 105. The analysis unit 101, the visitor information acquisition unit 102, the display control unit 103, and the evaluation information input unit 104 can be realized by the control unit 11 reading a program stored in the storage 13 into the memory 12 and executing it. The visitor DB 105 can also be realized as part of the storage area provided by the memory 12 and the storage 13. Note that the visitor DB 105 may also be realized as part of an external storage area provided by one or more servers other than the server 10.

[0028] The analysis unit 101 has the function of analyzing captured images obtained by capturing images of a subject, which is a visitor VR. The captured images are images captured by the monitoring device 30. The server 10 acquires the captured images from the monitoring device 30. The analysis unit 101 performs an analysis of the subject in the captured image, for example, using a known image analysis method. The analysis of the subject may include, for example, the analysis of the visitor VR's face or the visitor VR's clothing. The analysis unit 101 may output the analysis results of the captured images.

[0029] The visitor information acquisition unit 102 identifies visitors to the subject based on the analysis results of the captured image and acquires information about the identified visitors. The process of identifying visitors to the subject may include, for example, the process of identifying the visitor VR's personal information (face, name, etc.) and / or the visitor VR's attributes (occupation, age, gender, etc.).

[0030] Visitor DB105 is a database that stores information identified based on analysis results, such as the visitor VR's face and clothing, linked to information such as the visitor VR's identification information and attributes. For example, visitor data stored in Visitor DB105 may include image information (or its analysis results) of the visitor VR's face and clothing, the number of visits by the visitor VR, attributes related to the visitor VR, and information on the visitor VR's evaluation. The information stored in Visitor DB105 is information shared within a single residential area. In other words, the visitor VR information is information that all resident HRs living in a single residential area can commonly obtain. A single residential area will be described later.

[0031] Information regarding the evaluation of the visitor VR may include, for example, comments from the resident HR about the visitor (e.g., "a pleasant delivery person") and evaluation scores, which are entered by the evaluation information input unit 104 described later. Information regarding the attributes of the visitor VR may also include information about the attributes (for example, if the attribute is police officer, "There have been many crimes recently committed by people impersonating police officers, so let's check if this person is a real police officer"). Visitor data may not be linked to a specific visitor VR (for example, data not linked to the visitor VR's face). For example, visitor data may include attributes such as delivery person or police officer, and information about clothing linked to such attributes. This allows for estimation of the visitor VR's occupation, etc., even if visitor data is not stored in the visitor DB 105 (i.e., if it is the first time the visitor VR is recognized), and information based on such attributes can be presented to the resident HR.

[0032] The visitor information acquisition unit 102 can query the visitor database 105 using the analysis results of the captured image as a query, and acquire visitor data (visitor information) corresponding to the analysis results. For example, the analysis results of the captured image may include analysis results of the face, clothing, etc., of the subject in the captured image. If the analysis results of the captured image are the analysis results of the subject's face, visitor data having data corresponding to such face will be identified. Also, if the analysis results of the captured image are the analysis results of the subject's clothing, visitor data having data corresponding to such clothing (for example, data indicating that the visitor has attributes related to their job, such as a delivery person or police officer) may be identified.

[0033] Furthermore, if, as a result of analyzing the captured image, there is no visitor data that has data corresponding to the face identified in the analysis results, the visitor information acquisition unit 102 may identify visitor data (information) related to the attributes corresponding to the clothing based on the analysis results of the parts corresponding to the clothing. Also, if the number of visitor data stored in the visitor DB 105 does not meet a predetermined condition (for example, if the accuracy of the analysis of the visitor's face by machine learning does not reach a predetermined threshold), the visitor information acquisition unit 102 may identify visitor data (information) related to the attributes corresponding to the clothing based only on the analysis results of the parts corresponding to the clothing. This makes it possible to identify visitor VR even if sufficient visitor data has not been accumulated in the visitor DB 105 (see Table 1001 in Figure 4). In addition, the visitor information acquisition unit 102 may change the method of analyzing the captured image depending on the degree of evaluation of the information regarding the evaluation of visitor VR included in the visitor data. For example, if the analysis result of the visitor VR's face is positive, the visitor information acquisition unit 102 does not need to use the analysis results of the parts corresponding to the clothing. On the other hand, if the facial analysis results of the visitor VR are negative or not stored in the visitor DB105, the analysis results of the clothing portion may be used (see Table 1002 in Figure 4).

[0034] The display control unit 103 has the function of displaying the captured image and evaluation information for the visitor VR on the display device 20. Figure 5 is a diagram showing an example of the display mode on the display device by the display control unit 103 according to this embodiment. As shown in Figure 5, for example, in the example shown on screen 1101 of the display device 20, the captured image when the visitor VR is a delivery person, the number of visits by the delivery person, a score (an example of information regarding the evaluation of the visitor), and a comment (an example of information regarding the evaluation of the visitor) are displayed. Such a visitor VR can be identified, for example, by identifying and analyzing the face region F1 and the clothing region C1 from the captured image displayed on screen 1101. The display control unit 103 can display the above information based on the visitor data acquired by the visitor information acquisition unit 102. Note that the information regarding the visitor VR displayed on screen 1101 is not particularly limited.

[0035] Furthermore, in the example shown on screen 1102, the captured image and comments about the police officer are displayed when the visitor VR is a police officer. Here, it is assumed that the visitor is not identified based on the police officer's face because there is no face data corresponding to the face region F2 in the visitor DB 105. In this case, for example, if the visitor information acquisition unit 102 cannot identify the face of the visitor VR, it will identify and acquire visitor data that has the attributes of a police officer by analyzing the clothing region C2. Then, the display control unit 103 may control the system to display information indicating that the visitor is a suspicious police officer as information regarding the evaluation of such a visitor. On the other hand, if the face of the visitor VR can be identified, visitor data corresponding to such a face can be acquired even if the clothing analysis results in the attribute being determined to be that of a police officer. This makes it possible to display different evaluations depending on whether the face can be identified or not, even for visitor data with the same attributes.

[0036] The evaluation information input unit 104 has a function to output evaluation information about visitors, which are subjects included in the captured images displayed on the display device 20. The display for inputting evaluation information about visitors may be, for example, the display device 20, or it may be displayed on a user terminal 21 such as a smartphone owned by the resident HR. Such visitor information may be information input by each of several households included in a single collective residential area. Here, a single collective residential area may be, for example, an area composed of a single multi-unit dwelling such as an apartment building or condominium, or an area composed of a collection of individual houses included in a single neighborhood association. Alternatively, a single collective residential area may be an area composed of a collection of houses included in any administrative division (for example, a street number, district, block, town name, administrative district, city, town, etc.). For example, visitor information may be information input by residents of a single apartment building.

[0037] For example, after a resident HR interacts with a visitor VR, they input comments and scores for the visitor VR via the user terminal 21. The evaluation information input unit 104 obtains information about the visitor VR from the user terminal 21 via an input screen displayed on the user terminal 21. The evaluation information input unit 104 then records the acquired information in the visitor DB 105. The visitor data stored in the visitor DB 105 is updated based on this input information. As evaluations from other residents accumulate, evaluations of the visitor VR can be shared. In addition, comments entered by other residents may be evaluated by other residents. This can improve the reliability of the comments.

[0038] The evaluation information input unit 104 may also convert the text information of the input comment and obtain summarized information. Furthermore, the evaluation information input unit 104 may analyze the text information of the input comment and determine whether the comment is positive or negative. Such techniques for conversion to summary and text analysis may be publicly known technologies as appropriate.

[0039] Furthermore, the evaluation information input unit 104 may acquire information not only from resident HRs but also from the condominium management company, security company, public institutions, etc. For example, the evaluation information input unit 104 may input information on suspicious persons, crimes, and crime prevention information related to the area of ​​the condominium, linking it to attributes associated with the corresponding visitors and their clothing. Such input information is processed as appropriate and reflected in the visitor data stored in the visitor DB 105.

[0040] Figure 6 is a flowchart showing an example of the information processing flow by the server 10 according to this embodiment. First, when a visitor VR calls a resident HR using the intercom at the entrance of the apartment building, the monitoring device 30 captures an image of the visitor VR, and the server 10 acquires the generated image (step S101).

[0041] Next, the analysis unit 101 analyzes the acquired image (step S103), and the visitor information acquisition unit 102 acquires visitor information from the visitor DB 105 based on the analysis results (step S105).

[0042] Next, the display control unit 103 displays information related to evaluations, such as comments and scores, included in the acquired visitor information on the display device 20 (step S107).

[0043] According to System 1 of this embodiment, the resident HR can obtain information about visitors, as shown in Figure 4. This allows the resident HR to determine whether to respond to the visitor VR and whether to unlock the entrance.

[0044] Furthermore, after interacting with a visitor, resident HR can input comments and scores via the display device 20 or user terminal 21. Other residents can view these comments and scores upon subsequent visits. By sharing evaluations among residents of the same apartment building in this way, it is possible to avoid visits from visitors who could cause various problems, such as malicious sales visits, in addition to improving security.

[0045] Furthermore, the evaluation information input unit 104 may have a function to adjust the weight of the reliability of the entered comments based on indicators such as the residency history of the condominium resident HR and the degree of participation in the management association. For example, comments entered by a resident HR who has just moved into the condominium may not be displayed to other residents, or may be displayed with a lower priority (for example, at the bottom of the list of comments). Also, the weight of evaluations for comments from resident HRs who have just moved into the condominium may be adjusted to be lower.

[0046] Furthermore, the evaluation information input unit 104 may perform statistical processing on comments and evaluations of visitor VRs, and evaluate the attributes and trends of visitor VRs for a condominium based on the results of this processing. For example, it may become possible to evaluate each condominium, such as identifying a high number of suspicious individuals or salespeople, and to take measures against malicious visitors.

[0047] Furthermore, the display control unit 103 may output information such as evaluations and comments regarding the VR visitor experience to terminals of, for example, the apartment building manager or visiting salespeople. This allows for feedback to be provided to the salespeople regarding apartment management and visiting sales.

[0048] Furthermore, the following configurations also fall within the technical scope of this disclosure. (Item 1) The processor, Analyzing the captured images obtained by photographing the visitor subject, Based on the analysis results of the captured image, the visitor corresponding to the subject is identified, and information about the identified visitor is obtained. The information of the aforementioned visitor includes information on the evaluation of the aforementioned visitor, The captured image and the evaluation information for the visitor are displayed together on the display unit. A method that includes this. (Item 2) The portion of the captured image corresponding to the subject's face is analyzed, The method according to item 1, which includes obtaining information about the visitor based on the results of an analysis of the portion of the subject corresponding to the face. (Item 3) The portion of the captured image corresponding to the subject's clothing is analyzed, Based on the analysis results of the portion of the subject corresponding to the clothing, information regarding the attributes corresponding to the clothing is obtained. The captured image and the evaluation information for the attributes corresponding to the clothing are displayed together. The method described in item 1 or 2. (Item 4) The aforementioned visitor information is stored in the visitor database. If the visitor database does not contain any information that satisfies the predetermined conditions, Based on the analysis results of the portion corresponding to the subject's clothing, only information regarding the attributes corresponding to the clothing is obtained. The method described in item 3. (Item 5) The method according to any one of items 1 to 4, further outputting information for inputting evaluation information regarding the visitor, who is the subject of the captured image displayed on the display unit. (Item 6) The visitor information is information entered by each of the multiple households included in a single residential area, as described in any one of items 1 to 5. (Item 7) The information of the aforementioned visitors includes information provided by the administrator, security company, or public authority of a single residential area, as described in any one of items 1 to 6. (Item 8) An analysis unit analyzes the captured images obtained by photographing the subject, who is a visitor. A visitor information acquisition unit identifies the visitor corresponding to the subject based on the analysis results of the captured image and acquires information about the identified visitor. The information of the aforementioned visitor includes information on the evaluation of the aforementioned visitor, A display control unit that displays the captured image and the evaluation information for the visitor together on a display unit, An information processing system equipped with the following features. (Item 9) Computers, An analysis unit analyzes the captured images obtained by photographing the subject, who is a visitor. A visitor information acquisition unit identifies the visitor corresponding to the subject based on the analysis results of the captured image and acquires information about the identified visitor. The information of the aforementioned visitor includes information on the evaluation of the aforementioned visitor, A display control unit that displays the captured image and the evaluation information for the visitor together on a display unit, A program that makes it function as such. [Explanation of Symbols]

[0049] 1 System 10 servers 20 Display device 21 User terminals 30 Monitoring equipment 101 Analysis Department 102 Visitor information acquisition department 103 Display Control Unit 104 Evaluation Information Input Section 105 Visitor DB

Claims

1. The processor, Analyzing the captured images obtained by photographing the subject, who is a visitor, Based on the analysis results of the captured image, the visitor corresponding to the subject is identified, and information about the identified visitor is obtained. The information of the aforementioned visitor includes information on the evaluation of the aforementioned visitor, The captured image and the evaluation information for the visitor are displayed together on the display unit. Includes, The analysis includes analyzing the portion of the captured image corresponding to the subject's clothing, Based on the analysis results of the portion of the subject corresponding to the clothing, information regarding the attributes corresponding to the clothing is obtained. The visitor information further includes information provided by at least one of the administrators of a residential area, a security company, or a public institution. The information provided is linked to the information regarding the attribute, The display unit displays the provided information together with the captured image and the evaluation information. method.

2. The portion of the captured image corresponding to the subject's face is analyzed, The method according to claim 1, further comprising obtaining information about the visitor based on the results of an analysis of the portion of the subject corresponding to the face.

3. The aforementioned visitor information is stored in the visitor database. If the visitor database does not contain any information that satisfies the predetermined conditions, Based on the analysis results of the portion corresponding to the subject's clothing, only information regarding the attributes corresponding to the clothing is obtained. The method according to claim 2.

4. The display unit further outputs information for inputting evaluation information regarding the visitor, who is the subject of the captured image displayed on the display unit. The method according to any one of claims 1 to 3.

5. The method according to any one of claims 1 to 4, wherein the visitor information is information entered by each of the multiple households included in a single collective residential area.

6. An analysis unit analyzes the captured images obtained by photographing the subject, who is a visitor. A visitor information acquisition unit identifies the visitor corresponding to the subject based on the analysis results of the captured image and acquires information about the identified visitor. The information of the aforementioned visitor includes information on the evaluation of the aforementioned visitor, A display control unit that displays the captured image and the evaluation information for the visitor together on a display unit, Equipped with, The analysis unit analyzes the portion of the captured image corresponding to the subject's clothing, The visitor information acquisition unit acquires information regarding attributes corresponding to the clothing based on the analysis results of the portion corresponding to the subject's clothing. The visitor information further includes information provided by at least one of the administrators of a residential area, a security company, or a public institution. The information provided is linked to the information regarding the attribute, The display control unit displays the provided information together with the captured image and the evaluation information on the display unit. Information processing system.

7. Computers, An analysis unit analyzes the captured images obtained by photographing the subject, who is a visitor. A visitor information acquisition unit identifies the visitor corresponding to the subject based on the analysis results of the captured image and acquires information about the identified visitor. The information of the aforementioned visitor includes information on the evaluation of the aforementioned visitor, A display control unit that displays the captured image and the evaluation information for the visitor together on a display unit, To make it function as, The analysis unit analyzes the portion of the captured image corresponding to the subject's clothing, The visitor information acquisition unit acquires information regarding attributes corresponding to the clothing based on the analysis results of the portion corresponding to the subject's clothing. The visitor information further includes information provided by at least one of the administrators of a residential area, a security company, or a public institution. The information provided is linked to the information regarding the attribute, The display control unit displays the provided information together with the captured image and the evaluation information on the display unit. program.

Images