System, system control method, and program

Abstract

To solve the problem in which it has been difficult to reduce a burden on a user who uses a system that determines a subject captured in a video.SOLUTION: A system comprises: photographing means that photographs a visitor; telephone conversation means with which the visitor and a resident exchange voice to each other; notification means that notifies the visit of the visitor; response means that selects whether the resident responds to the visit of the visitor notified by the notification means by using the telephone conversation means; locking and unlocking means with which the resident operates unlocking a door; inference means that makes an inference about the visitor by using video information acquired from the photographing means and voice information acquired from the telephone conversation means as input data; and teacher data generation means that generates teacher data. The teacher data generation means generates the teacher data from a control state of the response means and the unlocking means after the notification means notifies the visit of the visitor.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a system for determining a subject appearing in a video image. [Background technology]

[0002] In recent years, technologies have become known that use AI to infer objects based on video and audio information. Furthermore, training data is necessary for AI learning, and the following techniques for generating training data are also known. For example, Patent Document 1 discloses a technology that uses AI to automatically select target sounds from audio information acquired by a sound collection device. In the disclosed method, training data used for AI learning is generated by a user inputting the type of audio for the input data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-046018 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in Patent Document 1, to generate training data, a person must perform annotation work by inputting the voice type for each input data. Because a large amount of data is required for AI training, performing annotation work manually has the inconvenience of placing a heavy burden on users who use the system. The present invention has been made in consideration of the above-mentioned problems, and aims to reduce the burden on users. [Means for solving the problem]

[0005] In order to achieve the above object, the present invention is an information processing system, comprising: an imaging means for imaging a visitor; a communication means for exchanging voice with the visitor; an acquisition means for acquiring information relating to the visitor's response to the visitor's arrival; a detection means for detecting the unlocking of a door; an image of the visitor taken by the imaging means and voice information acquired by the communication means as input data; information relating to the response acquired by the acquisition means and unlocking information relating to the unlocking of the door acquired by the detection means as training data; visitor and a notification means for notifying the visitor of the arrival of the visitor, wherein, after the notification means notifies the visitor of the arrival of the visitor, if there is no response to the visitor via the communication means and the door is unlocked, the learning data generation means generates learning data indicating that the visitor has a high reliability. [Effects of the Invention]

[0006] According to the present invention, it is possible to reduce the burden on a user who uses a system that determines a subject appearing in a video image. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a system diagram showing an intercom system according to an embodiment. [Figure 2] 1 is a block diagram showing a hardware configuration of an intercom system according to an embodiment. [Figure 3] 2 is a block diagram showing the software configuration of the intercom system according to the embodiment. FIG. [Figure 4] FIG. 1 is a conceptual diagram of a structure using a learning model consisting of input data, a learning model, and output data in an embodiment. [Figure 5] FIG. 2 is a conceptual diagram illustrating the operation of the intercom system according to the present embodiment. [Figure 6] FIG. 10 is a diagram showing the flow of a learning phase in this embodiment. [Figure 7]FIG. 2 is a diagram showing an example of learning data in the present embodiment. [Figure 8] FIG. 10 is a diagram showing the flow of an inference phase in this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0009] <System configuration> 1 is a system diagram showing an example of an intercom system to which the present invention can be applied. The system shown in Fig. 1 is composed of a network 100, an intercom 101, a data collection server 102, an inference server 103, and an intercom 104.

[0010] The intercom 101, data collection server 102, inference server 103, and intercom 104 are connected to a network 100 and are capable of communicating via the network 100. The network 100 may be configured with communication lines of a type appropriate for the purpose and use, regardless of whether it is wired or wireless. The network 100 may also be configured to allow smartphones, home appliances, etc. (not shown) to be connected, allowing users to access the network remotely.

[0011] A specific example of the use of the system described in this embodiment is use in a residence (apartment building). An intercom 101 installed in each residence acquires video information of the visitor and displays the video of the visitor on a notification unit to notify the residents of the visitor's arrival. The residents check the video of the visitor displayed on the notification unit of the intercom 101 and decide whether or not to respond. When responding to the visitor, the residents operate the intercom 101 to communicate by voice, and if necessary, operate the intercom 101 to select whether to open or close the door.

[0012] The visitor's video information, audio information, response information to the visitor, and door lock / unlock information acquired by the intercom 101 are stored in the data collection server 102 via the network 100 and input to the inference server 103. The inference server 103 performs inference processing using the visitor's video information and audio information received from the intercom 101 as input data, and transmits the inference results to the intercom 101 via the network 100. Details of the inference processing will be described later. The inference server 103 also generates training data from the response information to the visitor and door lock / unlock information received from the intercom 101. Details of training data generation will be described later.

[0013] It is also possible to share the inference results of the inference server 103 and the training data generated by the inference server 103 with intercoms 104 in other residences.

[0014] By using the system of this embodiment, the video information and audio information of the visitor acquired by the intercom 101 as described above is used as input data and inference processing is performed by the inference server 103. The intercom 101 receives the inference result of the inference server 103, which indicates the identity of the visitor, and can provide support to the resident in deciding whether to respond to the visitor or unlock the door.

[0015] <Hardware configuration> FIG. 2 is a diagram illustrating an example of hardware resources of each device constituting the system of FIG.

[0016] The data collection server 102 and the inference server 103 may have the same configuration, and therefore will be collectively described here as an information processing device. The information processing device includes a system bus 201, a CPU 202, a ROM 203, a RAM 204, a HDD 205, and a NIC (Network Interface Card) 206. It also includes an input unit 207, a notification unit 208, and a GPU (Graphical Processing Unit) 209.

[0017] The system bus 201 is connected to a CPU 202, a ROM 203, a RAM 204, a HDD 205, a NIC 206, an input unit 207, a notification unit 208, and a GPU 209, and each unit can exchange data with each other via the system bus 201.

[0018] The CPU 202 is connected to the ROM 203, RAM 204, HDD 205, NIC 206, input unit 207, notification unit 208, and GPU 209 via a system bus 201, and controls all of these. In the following description, unless otherwise specified, the main hardware that executes a program is the CPU 202, and the main software is a program stored in the ROM 203 or HDD 205.

[0019] The ROM 203 stores programs executed by the CPU 202 to control each unit of the information processing apparatus, and information such as parameters relating to the processing of each unit.

[0020] The RAM 204 is a rewritable memory, and is used as a work area when the CPU 202 executes a program stored in the ROM 203 .

[0021] The HDD 205 is a recording medium for recording programs executed by the CPU 202 to control each part of the information processing device, as well as video information, audio information, visitor response information, and door lock / unlock information transmitted from the intercom 101 via the network. The HDD 205 also records learning data used in the learning process described below, and a learning model that is the result of the learning process.

[0022] The NIC 206 connects the information processing device to the network 100. The network connection may be configured as either a wired LAN or a wireless LAN. The CPU 202 controls the NIC 206 to execute communication control processing with other devices connected via the network 100.

[0023] The input unit 207 receives various input operations of the information processing device from the user. The user can perform various settings of the information processing device via the input unit 207. The input unit 207 is an input device for receiving operations from the user, including, for example, a character information input device such as a keyboard, a pointing device such as a mouse or a touch panel, a button, a dial, a joystick, a touch sensor, a touch pad, etc.

[0024] The notification unit 208 is a display or the like that displays icons for user operations, etc. The notification unit 208 displays a GUI (Graphical User Interface) based on an instruction from the CPU 202.

[0025] The GPU 209 is a multi-core computing device that can perform calculations based on instructions from the CPU 202 or cooperate with the CPU 202 to perform calculations.

[0026] The intercom 101 is composed of a system bus 210, a CPU 211, a ROM 212, a RAM 213, a NIC 214, a photographing unit 215, a communication unit 216, a recording unit 217, an input unit 218, and a notification unit 219.

[0027] The system bus 210 is connected to a CPU 211, a ROM 212, a RAM 213, a NIC 214, an imaging unit 215, a call unit 216, a recording unit 217, an input unit 218, and a notification unit 219. Each unit can exchange data with each other via the system bus 210.

[0028] The CPU 211 is connected to and controls all of the ROM 212, RAM 213, NIC 214, imaging unit 215, call unit 216, recording unit 217, input unit 218, and notification unit 219 via a system bus 210. In the following description, unless otherwise specified, the main hardware that executes a program is the CPU 211, and the main software is a program stored in the ROM 212 or recording unit 217.

[0029] The ROM 212 stores a program for controlling the interphone 101 executed by the CPU 211, and information such as parameters relating to the processing of each unit.

[0030] The RAM 213 is a rewritable memory, and is used as a work area when the CPU 211 executes a program stored in the ROM 212 .

[0031] The NIC 214 connects the intercom 101 to the network 100. The network connection may be configured as either a wired LAN or a wireless LAN. The CPU 211 controls the NIC 214 to execute communication control processing with other devices connected via the network 100.

[0032] The photographing unit 215 is composed of an optical lens, an image pickup element, an image processing unit, etc. Video information photographed by the photographing unit 215 is temporarily stored in the RAM 213 and recorded in the recording unit 217 in response to instructions from the CPU 211. The photographing unit 215 also controls zoom, focus, aperture setting, etc. in response to instructions from the CPU 211. The optical lens is a group of lenses including a zoom lens and a focus lens. The image pickup element is composed of a CCD or CMOS element that converts an optical image into an electrical signal. The image processing unit is capable of performing various image processes on the video information and still image information stored in the RAM 213 and the recording unit 217 under the control of the CPU 211. The image processing unit is capable of determining facial information, body type information, and clothing information of the subject, and the CPU 211 stores the extracted facial information, body type information, and clothing information of the subject in the recording unit 217. The image processing performed by the image processing unit includes A / D conversion processing, D / A conversion processing, encoding processing of moving image information and still image information, compression processing, decoding processing, resizing processing, noise reduction processing, color conversion processing, etc. The image processing unit may be configured with a circuit block dedicated to performing specific image processing. Depending on the type of image processing, the CPU 211 may perform image processing according to a program without using the image processing unit. Note that the video information acquired by the imaging unit 215 may also include audio information.

[0033] The communication unit 216 is composed of a microphone, a speaker, an audio processing unit, etc. Audio information acquired by the communication unit 216 is temporarily stored in the RAM 213 and is recorded in the recording unit 217 in response to an instruction from the CPU 211. The audio processing unit can distinguish the subject's voice, and the CPU 211 stores the subject's voice information in the recording unit 217 together with video information of the subject captured by the imaging unit 215.

[0034] Recording unit 217 is equipped with a recording medium, and records the program executed by CPU 211 for controlling intercom 101, the video information captured by imaging unit 215, and the audio information acquired by communication unit 216 on the recording medium based on instructions from CPU 211. Information about the subject obtained by imaging can also be stored in recording unit 217. The recording medium may be an internal recording medium such as an internal flash memory or internal HDD, or an external recording medium such as a removable memory card.

[0035] The input unit 218 receives various input operations for the intercom 101 from the user. Specifically, it includes an operation unit for announcing a visitor, a response unit for selecting whether to respond to the visitor, and a lock / unlock unit for selecting whether to open or close the door. The user can also perform various settings for the intercom 101 via the input unit 218. The input unit 218 is an input device, such as a button or a touch panel, for receiving input from the user. When the visitor operates the input unit 218 to notify the resident of the visitor's arrival, the CPU 211 can control the communication unit 216 to acquire the visitor's voice. In this case, the CPU 211 may immediately transmit the acquired voice to the resident via the notification unit 219, or may control the voice not to be transmitted to the resident until the resident operates the input unit 218 to respond to the visitor.

[0036] The notification unit 219 is a display that displays icons for user operation and video information captured by the image capture unit 215. The display displays a GUI (Graphical User Interface) based on instructions from the CPU 211. Furthermore, the notification unit 219 can output confirmation sounds and warning sounds for user operation, and sounds that notify the arrival of a visitor, from a speaker.

[0037] The intercom 104 has substantially the same hardware configuration as the intercom 101, and therefore a description thereof will be omitted here.

[0038] <Software configuration> FIG. 3 is a diagram showing an example of a software configuration realized by using the hardware resources and programs shown in the hardware configuration diagram of FIG.

[0039] First, the functional blocks of the intercom 101 will be described.

[0040] The CPU 211 uses the RAM 213 as a working area to execute programs stored in the ROM 212 or the recording unit 217, thereby realizing the functions of the following units: a video data unit 301, an audio data unit 302, a response data unit 303, a lock / unlock data unit 304, a data transmission unit 305, and a data reception unit 306.

[0041] The video data unit 301 transmits the captured video information to the data collection server 102 and the inference server 103 via the data transmission unit 305 .

[0042] The voice data unit 302 transmits the acquired voice information to the data collection server 102 and the inference server 103 via the data transmission unit 305 .

[0043] The response data unit 303 transmits the acquired response information indicating the response status to the visitor to the data collection server 102 and the inference server 103 via the data transmission unit 305 .

[0044] The lock / lock data unit 304 transmits the acquired lock / lock information indicating the open / close status of the door to the data collection server 102 and the inference server 103 via the data transmission unit 305 .

[0045] The data receiving unit 306 receives the inference results sent from the inference server 103 .

[0046] Next, the functional blocks of the data collection server 102 will be described.

[0047] The CPU 202 executes a program stored in the ROM 203 or the HDD 205 using the RAM 204 as a work area, thereby realizing the functions of a data storage unit 311 and a data collection / provision unit 312 .

[0048] The data storage unit 311 stores the video information, audio information, response information, and lock / unlock information received by the data collection / provision unit 312 and transmitted from the intercom 101 in the HDD 205 .

[0049] The data collection / provision unit 312 receives the video information, audio information, response information, and lock / unlock information transmitted from the intercom 101. The data collection / provision unit 312 also transmits the received video information and temperature information to the inference server 103.

[0050] Next, the functional blocks of the inference server 103 will be described.

[0051] The CPU 202 executes a program stored in the ROM 203 or the HDD 205 using the RAM 204 as a working area, thereby realizing the functions of the following units: a data storage unit 321, a learning data generation unit 322, a learning unit 323, an inference unit 324, a data transmission unit 325, and a data transmission unit 326.

[0052] The data storage unit 321 stores the video information, audio information, response information, and lock / unlock information transmitted from the intercom 101, and the learning data generated by the learning data generation unit 322, in the HDD 205. The learning data is assumed to be composed of input data and training data.

[0053] The learning data generation unit 322 generates input data from the video information and audio information transmitted from the data collection server 102. The learning data generation unit 322 also generates training data from the response information and lock / unlock information transmitted from the data collection server 102. A specific generation flow will be described later.

[0054] The learning unit 323 learns a learning model using the learning data generated by the learning data generation unit 322. During learning, the GPU 209 can perform efficient calculations by processing a larger amount of data in parallel. Therefore, when learning is performed multiple times using a learning model such as deep learning, it is effective to use the GPU 209 for processing. Therefore, in this embodiment, the GPU 209 is used in addition to the CPU 202 for processing by the learning unit 323. Specifically, when a learning program including a learning model is executed, the CPU 202 and the GPU 209 cooperate to perform calculations to perform learning. Note that the processing by the learning unit 323 may be performed by only the CPU 202 or the GPU 209. Similarly to the learning unit 323, the inference unit 324 may also use the GPU 209.

[0055] The inference unit 324 is specifically an AI, and is a learning model resulting from learning by the learning unit 323. The inference unit 324 performs inference processing using the video information and audio information transmitted from the intercom 101 as input data. Specifically, the inference unit 324 infers whether the visitor is trustworthy or not using the visitor's facial information, body type information, clothing information, voice information, etc. Furthermore, the inference unit 324 stores the results of the inference processing in the data storage unit 321, and transmits them to the intercom 101 via the data transmission unit 326.

[0056] <Input and output of learning model> FIG. 4 is a conceptual diagram showing the input / output structure using the learning model of this embodiment.

[0057] The learning model 404 is a learning model trained using machine learning. Specific examples of machine learning algorithms include nearest neighbor algorithms, naive Bayes algorithms, decision trees, and support vector machines. Also available is deep learning, which uses a neural network to generate features and connection weighting coefficients for learning. Any of the above algorithms can be used as appropriate and applied to this embodiment.

[0058] The input data used for the learning model 404 are two pieces of input data: input data X1 (401), which is video information transmitted from the video data unit 301, and input data X2 (402), which is audio information transmitted from the audio data unit 302.

[0059] As output data from the learning model 404, output data Y (403) indicating the reliability of the subject is output.

[0060] <System Operation> FIG. 5 is a diagram for explaining the operation of a system to which the present invention can be applied, which uses the structure of the learning model shown in FIG.

[0061] 1. The intercom 101 transmits the video information of the visitor obtained by photographing and the audio information obtained from the conversation with the visitor to the inference server 103 via the network. 2. The intercom 101 transmits the response information and lock / unlock information obtained by the user's operation on the input unit to the inference server 103 via the network. 3. Generate input data for learning data based on the received video and audio information. 4. Generate training data for learning based on the received response information and lock / unlock information. 5. AI is trained using input data and training data.

[0062] By performing the operations shown in Figure 5, it is possible to obtain the effect of automatically generating training data without any operation by the user.

[0063] <Learning Phase> FIG. 6 is a flowchart showing the detailed flow of learning in the learning phase.

[0064] First, the processing flow of the intercom 101 will be described with reference to Fig. 6(a). In the learning phase of this embodiment, the video information and audio information acquired by the intercom 101 are used as input data for learning, and training data for learning is generated based on the response information and lock / unlock information.

[0065] In S601, if the CPU 211 determines that a visitor has operated the input unit 218 and notified the resident of the visit (S601, Yes), it controls the notification unit 219 to notify the resident of the visitor's arrival. If the operation of the input unit 218 by the visitor has not been accepted (S601, No), the CPU 211 continues to wait for the operation of the input unit 218 by the visitor.

[0066] In S602, the CPU 211 controls the image capturing unit 215 to capture an image of the visitor, and controls the communication unit 216 to transmit the visitor's voice to the resident. The CPU 211 records the acquired video information and audio information in the recording unit 217.

[0067] In S603, the CPU 211 transmits the video information and audio information recorded in the recording unit 217 in S602 to the data collection server 102 via the NIC 214.

[0068] In S604, if the CPU 211 determines that an operation to respond to the visitor's arrival has been performed via the input unit 218 (S604, Yes), it transmits response information indicating that the resident has responded to the visitor's arrival to the data collection server 102 via the NIC 214 (S605). If the CPU 211 has not received a request to respond to the visitor's arrival via the input unit 218, it determines that the resident has not responded to the visitor's arrival (S604, No). In this case, it transmits response information indicating that the resident has not responded to the visitor's arrival to the data collection server 102 via the NIC 214 (S606).

[0069] In S607, when the CPU 211 determines via the input unit 218 that the resident has performed an operation to unlock the door, it transmits lock / unlock information, which is control status information indicating that the door has been unlocked for visitors, to the data collection server 102 via the NIC 214 (S608). If the CPU 211 has not received a request to unlock the door via the input unit 218, it determines that the resident has not unlocked the door for visitors (S607, No). In this case, it transmits lock / unlock information, which is control status information indicating that the door has not been unlocked for visitors, to the data collection server 102 via the NIC 214 (S609).

[0070] As described above, the intercom 101 transmits the video and audio information acquired in response to a visitor's arrival, response information indicating the resident's response to the visitor, and lock / unlock information to the data collection server 102 via the NIC 214 each time a visitor arrives.

[0071] 6(b), the processing flow of the data collection server 102 will be described. The data collection server 102 records in the HDD 205 a large amount of input data transmitted from the intercom 101 to be used for learning.

[0072] In S610, when the CPU 202 receives video information, audio information, response information, and lock / unlock information from the intercom 101 via the NIC 206 (S610, Yes), it records the received video information, audio information, response information, and lock / unlock information in the HDD 205 (S611). If no data has been received from the intercom 101 (S610, No), the CPU 202 continues to wait for data to be received.

[0073] In S612, the CPU 202 reads the data recorded in the HDD 205 in S611 and transmits it to the inference server 103 via the NIC 206. Note that the recording process of recording the data received from the intercom 101 in the HDD 205 and the data transmission process to the inference server 103 may be performed simultaneously.

[0074] Next, the processing flow of the inference server 103 will be explained using FIG. 6(c).

[0075] In S613, when the CPU 202 receives the video information and audio information transmitted from the data collection server 102 via the NIC 206 (S613, Yes), it generates input data for learning from the received video information and audio information (S614). The CPU 202 analyzes the video information received from the data collection server 102 and extracts subject information. For example, this information may include facial information, body type information, and clothing information of the subject visitor. The CPU 202 can also extract voice information of each subject by analyzing the audio information received from the data collection server 102 together with the video information. The CPU 202 records the input data generated from the analysis results of the video information and audio information in the HDD 205. Specific examples of input data will be described later with reference to the drawings. If the CPU 202 has not received data from the data collection server 102 (S613, No), it continues to wait for data to be received.

[0076] In S615, when the CPU 202 receives the response information and lock / unlock information transmitted from the data collection server 102 via the NIC 206 (S615, Yes), it generates training data for learning from the received response information and lock / unlock information (S616). A specific flow of training data generation will be described later. Furthermore, if the CPU 202 has not received data from the data collection server 102 (S615, No), it continues to wait for data to be received. Note that the process of generating input data and the process of generating learning data may be performed simultaneously.

[0077] In S617, the CPU 202 trains a learning model using the input data generated in S614 and the training data generated in S616 as training data. Details regarding the learning are described in Figure 4 and are therefore omitted here. The CPU 202 can also update the trained model by overwriting the trained model trained in this step with the trained model recorded in the HDD 205 at a predetermined timing.

[0078] Next, a specific flow of generating training data in S616 will be described with reference to FIG. 6(d).

[0079] In S618, if the CPU 202 determines from the response information received from the data collection server 102 that the resident has not responded to the visitor's arrival (S618, No), the process proceeds to S619 in this flowchart.

[0080] In S619, if the lock / unlock information received from the data collection server 102 indicates that the resident unlocked the door for the visitor (Yes in S619), the CPU 202 generates training data indicating that the visitor is highly trustworthy (S620). In this case, for example, the resident confirms the visitor displayed on the intercom 101 and unlocks the door without any voice communication, so it is determined that the visitor is likely to be an old acquaintance of the resident, such as a family member or friend.

[0081] Furthermore, if the CPU 202 determines from the lock / unlock information received from the data collection server 102 that the resident did not unlock the door for the visitor (S619, No), it generates training data indicating that the visitor's trustworthiness is very low (S621). In this case, for example, the resident checks the visitor displayed on the intercom 101, but since there is no voice communication or unlocking of the door, it is highly likely that the resident is playing hooky and is annoyed by the visitor's arrival. In this case, it is necessary to confirm that the resident is inside the house to distinguish this from a state in which the resident is absent from home. Although a detailed explanation is omitted, this can be achieved, for example, by using information from a motion sensor installed in the house or information from a camera or sensor installed in each home appliance.

[0082] In S618, if the CPU 202 determines from the response information received from the data collection server 102 that the resident has responded to the visitor's arrival (S618, Yes), the process proceeds to S622 in this flowchart.

[0083] In S622, if the lock / unlock information received from the data collection server 102 indicates that the resident unlocked the door for the visitor (Yes in S622), the CPU 202 generates training data indicating that the visitor is highly trustworthy (S623). This case is, for example, when the resident unlocks the door after confirming the visitor displayed on the intercom 101 and communicating with them by voice. From this, it is determined that the visitor is likely to be an old acquaintance such as a family member or friend, or a delivery person or other person who is not a problem when it comes to unlocking doors.

[0084] Furthermore, if the resident has not unlocked the door for the visitor based on the lock / unlock information received from the data collection server 102 (S622, No), the CPU 202 generates training data indicating that the visitor is not trustworthy (S624). In this case, for example, the resident has confirmed the visitor displayed on the intercom 101, exchanged voice messages with the resident, and then has not unlocked the door, so it is determined that the visitor is likely to be an undesirable person such as a salesperson who the resident is not interested in.

[0085] In this way, it is possible to automatically generate training data from how residents respond to visitors.

[0086] The training data may also be generated by adding information about the time of the visitor to the response information and lock / unlock information. For example, if a resident exchanges voice messages with a visitor and unlocks the door at the time when family members usually return home, training data can be generated indicating that the visitor is likely to be a family member.

[0087] Similarly, training data may be generated by adding information about the visitor's schedule to the response information and lock / unlock information. For example, if a resident exchanges voice messages with a visitor and unlocks the door within the designated time for the scheduled delivery, training data may be generated indicating that the visitor is likely to be a delivery person. Schedule information may be managed by an intercom or an information processing device. Furthermore, this information can also be obtained by linking with the resident's smartphone, for example.

[0088] By taking time information and schedule information into account in this way, it is possible to generate more accurate training data.

[0089] It is also possible to change the weighting of lock / unlock information depending on location information indicating the location of the door to be opened or closed. For example, in the case of an apartment building, there may be a door at the entrance as well as at each dwelling unit. In such a case, even if a resident accidentally unlocks the entrance door, it is possible that the door to the dwelling unit will not be unlocked. Therefore, if a resident unlocks the dwelling unit door, training data can be generated that gives the visitor a higher reliability than if only the entrance door was unlocked, making it possible to generate more accurate training data.

[0090] It is also possible to generate training data using response information and lock / unlock information in response to visitor arrivals, as well as training data generated by intercom systems in other residences. This allows the response of residents of other residences to be taken into account, making it possible to generate more accurate training data.

[0091] <Learning data> FIG. 7 is a diagram showing an example of learning data made up of input data and training data.

[0092] First, the input data will be described. The input data items include "face," "body type," "clothing," "voice," and "time." Note that "face" and "voice" are not shown in the figure.

[0093] “Face” is the facial information of the visitor detected from the video information acquired by the image capturing unit 215 of the intercom 101 . “Body type” is body type information of the visitor detected from the image information acquired by the image capturing unit 215 of the intercom 101. “Clothing” is clothing information of the visitor detected from the video information acquired by the image capturing unit 215 of the intercom 101. “Voice” is information about the visitor's voice detected from the audio information acquired by the communication unit 216 of the intercom 101 . "Time" is information about the time when the input data was generated.

[0094] Next, the training data will be described. The training data items include "response" and "unlock."

[0095] "Response" is information indicating whether or not the resident responded to the visitor by operating the input unit 218 after the notification unit 219 of the intercom 101 notified the visitor of the visitor's arrival. "Unlock" is information indicating whether or not the resident unlocked the door by operating the input unit 218 after the notification unit 219 of the intercom 101 notified them of the arrival of a visitor.

[0096] Next, learning data IDs 1 to 4 will be explained in order.

[0097] The learning data ID1 is input data in which the response information is "none" and the unlocking information is "yes" for a person with a specific face, a "petite" build, wearing "casual" clothing, and a specific voice. From this input data, the inference unit 324 learns that the person in question is a person with a very high degree of reliability. As the learning progresses, the inference unit 324 becomes able to infer that the person in question is an old acquaintance of the resident, such as a family member or friend.

[0098] The learning data ID2 is input data in which the response information is "none" and the unlocking information is "none" for a person with a specific face, a "medium build," wearing "casual" clothing, and a specific voice. From this input data, the inference unit 324 learns that the person in question is a person with a very low level of reliability. As the learning progresses, the inference unit 324 infers that the person in question is not trustworthy, and when the person in question visits, it becomes possible to output a warning message to the resident via the notification unit 219 of the intercom 101, or not notify the resident of the visit at all.

[0099] The learning data ID3 is input data in which the response information is "yes" and the unlocking information is "yes" for a person with a specific face, a "large build," wearing a "uniform," and a specific voice. From this input data, the inference unit 324 learns that the person in question is trustworthy. As the learning progresses, the inference unit 324 becomes able to infer that the person in question is a delivery person or the like from the design, logo, etc. of the "uniform" worn by the person in question.

[0100] The learning data ID4 is input data in which the response information is "yes" and the unlocking information is "no" for a person with a specific face, a "tall and slim" build, a "suit," and a specific voice. From this input data, the system learns that the person in question is not trustworthy.

[0101] <Inference phase> FIG. 8 is a flowchart showing the detailed flow of inference in the inference phase.

[0102] In S801, when the CPU 202 receives video information and audio information transmitted from the data collection server 102 via the network 100 (S801, Yes), it creates input data to be used for inference processing from the received video information and audio information (S802). The CPU 202 analyzes the video information and audio information received from the data collection server 102 and extracts subject information of the visitor. The CPU 202 records the input data generated from the analysis results of the video information and audio information in the HDD 205. Furthermore, if the CPU 202 has not received data from the data collection server 102 (S801, No), it continues to wait for data to be received.

[0103] In S803, the CPU 202 inputs the input data generated in S802 into the trained model.

[0104] In S804, the trained model uses the video and audio information as input data to infer attribute information indicating who the visitor is. The CPU 202 records the inference results in the HDD 205 and transmits them to the intercom 101 via the network 100 (S805).

[0105] The CPU 202 can also transmit the inference results to an intercom 104 in another residence via the network 100, and can also transmit the results to a terminal such as a smartphone owned by a resident (not shown) or to home appliances in the residence.

[0106] As described above, according to this embodiment, it is possible to automatically generate training data to be used for learning.

[0107] (Other embodiments) The present invention can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having the computer of the system or device read and execute the program. The computer has one or more processors or circuits, and may include multiple separate computers or a network of multiple separate processors or circuits to read and execute computer-executable instructions.

[0108] The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gateway (FPGA), a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

Claims

1. a photographing means for photographing visitors; A communication means for exchanging voice with the visitor; an acquisition means for acquiring information regarding a response to the visit of the visitor; A detection means for detecting unlocking of the door; a learning data generation means for generating learning data for a learning model for inferring attributes of the visitor, using as input data the image of the visitor captured by the capturing means and the voice information acquired by the communication means, and using as training data information about the response acquired by the acquisition means and unlocking information about the unlocking of the door acquired by the detection means; a notification means for notifying the visitor of his / her arrival, An information processing system characterized in that, after the notification means notifies the visitor of his / her arrival, if the visitor is not responded to via the communication means and the door is unlocked, the learning data generation means generates learning data indicating that the visitor is highly trustworthy.

2. The information processing system according to claim 1, characterized in that after the notification means notifies the visitor of his / her arrival, the notification means responds to the visitor via the communication means, and if the door is unlocked, learning data is generated indicating that the visitor is highly trustworthy.

3. The information processing system described in claim 1, characterized in that the learning data generated when the notification means notifies the visitor of the visitor's arrival, but the visitor is not responded to via the communication means and the door is unlocked, has a higher reliability of the visitor than the learning data generated when the notification means notifies the visitor of the visitor's arrival, but the visitor is responded to via the communication means and the door is opened.

4. The information processing system according to claim 1, characterized in that after the notification means notifies the visitor of his / her arrival, the notification means responds to the visitor via the communication means, and if the door is not unlocked, learning data is generated indicating that the visitor has a low trustworthiness level.

5. The information processing system described in claim 1, characterized in that if the notification means notifies the visitor of his / her arrival and the visitor is not responded to via the communication means and the door is not unlocked, learning data is generated indicating that the visitor has a low trustworthiness.

6. The information processing system described in claim 1, characterized in that the learning data generated when the notification means notifies the visitor of the visitor's arrival, the visitor is responded to via the communication means, and the door is not unlocked, has a higher degree of reliability of the visitor than the learning data generated when the notification means notifies the visitor of the visitor's arrival, but the visitor is not responded to via the communication means, and the door is not unlocked.

7. The information processing device further includes means for acquiring the visitor's arrival time, 2. The information processing system according to claim 1, wherein the learning data generating means also uses the visitor's arrival time in generating the learning data.

8. further comprising a schedule management means for managing the visit schedule of the visitor; 2. The information processing system according to claim 1, wherein the learning data generating means also uses schedule information held by the schedule management means to generate the learning data.

9. The locking mechanism further includes a determination means for determining a position of a door to be unlocked in response to the visitor, 2. The information processing system according to claim 1, wherein the learning data generating means also uses the door position information determined by the determining means to generate the learning data.

10. Further comprising a communication means for communicating with another intercom system, 2. The information processing system according to claim 1, wherein said learning data generating means also uses learning data of other intercom systems received by said communication means to generate said learning data.

11. The photography process of photographing visitors, a call process for exchanging voice messages with the caller; an acquisition step of acquiring information regarding a response to the visitor's visit; a detection step of detecting unlocking of the door; a learning data generation process for generating learning data for a learning model for inferring attributes of the visitor, using the image of the visitor captured in the capturing process and the voice information acquired in the calling process as input data, and information about the response acquired in the acquiring process and unlocking information about the door acquired in the detecting process as training data; a notification step of notifying the visitor of the arrival of the visitor, A control method for an information processing system, characterized in that if, after the notification process has notified the visitor of their arrival, there is no response to the visitor via the communication means and the door is unlocked, the learning data generation process generates learning data indicating that the visitor has a high level of trustworthiness.

12. A computer-readable program for causing a computer to operate as the system according to any one of claims 1 to 10.

Images