Information notification system, information notification program, and information notification method

Abstract

Provided are a system for providing information on a subject, an information notification program, and an information notification method that accurately recognize the face of a subject while adding and updating the subject's information. [Solution] An information provision system including a server and a terminal capable of communicating with the server includes a server communication unit capable of communicating with the terminal, a server memory unit capable of storing information about the subject, a learning unit that adds and updates the information about the subject and learns, a face detection unit that detects facial information about the subject based on images or videos of the subject captured by the imaging unit and information from the learning unit that are sent from the terminal, a matching unit that compares and collates the facial information of the subject stored in the server memory unit with the facial information of the subject detected by the face detection unit based on information from the learning unit, and a notification unit that notifies the terminal of the results of the matching by the matching unit and the information about the subject from the matching results.

Description

[Technical Field]

[0001] The present invention relates to an information notification system, an information notification program, and an information notification method. [Background technology]

[0002] 2. Description of the Related Art Conventionally, attribute information such as names, company names, and titles contained in business cards has been compiled into a database and managed using a dedicated application such as a business card management application.

[0003] For example, Patent Document 1 discloses a technology in which attribute information of a business card is read using a portable information terminal device the same size as the business card, the read attribute information is compiled into a database and stored in advance in a memory unit provided in the portable information terminal device, and when a user inputs a company name or the like, the attribute information of the business card is called up from the memory unit and displayed on the screen.

[0004] Furthermore, Patent Document 2 discloses that the eyeglass-type terminal 1 is a terminal with an augmented reality function, is worn by a user 2, and when the user 2 exchanges business cards with a target person 3, the eyeglass-type terminal 1 captures an image including a facial image of the target person 3 and a business card BC showing attribute information of the target person 3, and associates and registers the facial image of the target person 3 with the attribute information; when the user 2 meets the target person 3 again, the eyeglass-type terminal 1 captures an image of the target person 3's face, compares the facial image with the registered facial images, and if a corresponding facial image is registered, the attribute information corresponding to the facial image is displayed on the display 1a of the eyeglass-type terminal 1 together with the target person 3 included in the actual field of view. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-44174 [Patent Document 2] JP 2019-40401 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the technology of the invention of Patent Document 1, which manages attribute information contained in business cards by storing it in a database, it is not possible to search for attribute information using the database unless part of the attribute information, such as the name of the person with whom the business card was exchanged or the name of the company, is entered.

[0007] Therefore, if you meet someone again after exchanging business cards and cannot remember their name or company name, it is difficult to present attribute information based on the attribute information stored in a database.

[0008] Furthermore, the invention of Patent Document 2 requires that the glasses-type terminal be worn at all times, and although the facial image of the person presenting the business card is stored in association with the attribute information of the person shown on the business card, if the information of the person has not been updated, there is a problem in that the identity of the person cannot be determined even if the person actually meets the person.

[0009] Furthermore, although it is stated that the matching unit matches the captured face image with a registered face image, it is unclear how the matching is performed specifically.

[0010] An object of the present invention is to provide a subject information providing system that accurately recognizes the subject's face while adding and updating the subject's information.

[0011] Another object of the present invention is to provide a program for providing information on a subject that accurately recognizes the face of a subject while adding and updating the subject's information.

[0012] Another object of the present invention is to provide a method for providing information on a subject that accurately recognizes the face of the subject while adding and updating the subject's information. [Means for solving the problem]

[0013] An information provision system according to a first aspect of the present invention is an information provision system in which a terminal having an imaging unit capable of imaging a target person and a server can communicate with each other, The server a server communication unit capable of communicating with the terminal; a server storage unit capable of storing information of the subject; a learning unit that adds and updates information about the subject and learns; a face detection unit that detects facial information of the subject based on the image or video of the subject captured by the imaging unit and transmitted from the terminal and information from the learning unit; a matching unit that compares and matches the face information of the subject stored in the server storage unit with the face information of the subject detected by the face detection unit based on information from the learning unit; The information provision system includes a result of the comparison performed by the comparison unit, and a notification unit that notifies the terminal of the information about the subject if the information about the subject is stored in the server memory unit based on the comparison result.

[0014] The learning unit adds and updates information about the subject captured on the device, increasing the amount of facial information data for the subject, making it easier to recognize the subject's face and improving the accuracy of facial recognition. The "learning" of adding and updating information in the learning unit is what is known as AI (Artificial Intelligence).

[0015] The terminal user can also obtain information about the target person whom he or she has met in the past, and can obtain personal information such as the target person's name, occupation, age, etc.

[0016] Therefore, even if the user of the terminal has forgotten personal information such as the target person's name or company name, the user can immediately obtain information such as the target person's name, company name, and when they met previously, through a notification from the notification unit.

[0017] An information providing system according to a second aspect of the present invention is the information providing system according to the first aspect, The learning unit is an information providing system that learns by adding and updating information on the detection result of face information detected by the face detection unit and the matching result of matching performed by the matching unit.

[0018] Learning in the learning section makes it easier to recognize the target person's face, improving the accuracy of detecting and matching the target person's face.

[0019] An information providing system according to a third aspect of the present invention is the information providing system according to the first aspect, the face detection unit detects facial information of the subject from a video of the subject captured by the imaging unit of the terminal; The matching unit is an information providing system that compares and matches the facial information of the subject detected by the face detection unit with facial information of people in the video stored in the server storage unit.

[0020] Videos contain more information than still images, improving the accuracy of detecting and matching the subject's face.

[0021] Furthermore, the process of selecting an image of the subject to be compared from the video is not required.

[0022] An information providing system according to a fourth aspect of the present invention is the information providing system according to the first aspect, the face detection unit detects facial information of the subject from a part of a video of the subject captured by the imaging unit of the terminal; The matching unit is an information provision system that compares and matches a partial image of a video of a subject captured by the imaging unit of the terminal with a matching image of a person stored in the server memory unit based on features.

[0023] Using a partial image of the video as the comparison image has the advantage of making it easier to match.

[0024] An information provision system according to a fifth aspect of the present invention is the information provision system according to the first aspect, The face matching unit is an information providing system that compares and matches a matching image of a subject detected by the face detection unit with a registered image of the person stored in the server storage unit based on features.

[0025] The advantage of this method is that the comparison and matching based on the feature amount improves the matching speed.

[0026] An information provision system according to a sixth aspect of the present invention is an information provision system in which a terminal having an imaging unit capable of imaging a target person and a server can communicate with each other, The server a server communication unit capable of communicating with the terminal; a server storage unit capable of storing information of the subject; a learning unit that adds and updates information about the subject and learns; a face detection unit capable of detecting a human face and detecting facial information of the subject based on an image or video of the subject captured by the imaging unit and transmitted from the terminal and information from the learning unit; a matching unit that accesses a criminal database and compares and matches the facial information of criminals stored in the criminal database with the facial information of the target person detected by the face detection unit based on information from the learning unit; This information provision system includes a result of the matching performed by the matching unit, and a notification unit that notifies the terminal of the information of the target person if the matching result indicates that the target person's information is stored in the criminal database.

[0027] This allows the user of the terminal to be warned of the presence of a criminal.

[0028] An information provision system according to a seventh aspect of the present invention is the information provision system according to the sixth aspect, the server includes a weapon detection unit that detects a weapon; When the weapon detection unit detects a weapon possessed by the target person, the notification unit notifies the terminal of that fact or issues a warning.

[0029] This allows you to immediately see whether a criminal photographed by the device's camera is carrying a weapon.

[0030] An information provision program according to an eighth aspect of the present invention is an information provision program capable of communicating with a terminal having an imaging unit capable of imaging a target person, a server communication process capable of communicating with the terminal; a server storage process capable of storing information of the subject; A learning process for adding and updating information about the subject and learning; a face detection process capable of detecting a human face, which detects facial information of the subject based on the image or video of the subject captured by the imaging unit and transmitted from the terminal and information in the learning process; a matching process for comparing and matching the face information of the subject stored in the server storage process with the face information of the subject detected by the face detection process based on information from the learning process; This is an information provision program that executes the results of the matching process and a notification process that notifies the terminal of the subject's information if the subject's information has been stored in the server storage process based on the matching results.

[0031] Such a program will have the same effect as the first aspect.

[0032] An information providing method according to a ninth aspect of the present invention is an information providing method capable of communicating with a terminal having an imaging unit capable of imaging a target person, The server: a server communication step capable of communicating with the terminal; a server storage step capable of storing information of the subject; a learning process of adding and updating information about the subject and learning; a face detection process capable of detecting a human face, which detects facial information of the subject based on an image or video of the subject captured by the imaging unit and transmitted from the terminal, and information in the learning process; a matching step of comparing and matching the face information of the subject stored in the server storage step with the face information of the subject detected in the face detection step based on the information in the learning step; This information provision method includes a result of the comparison in the comparison step, and a notification step of notifying the terminal of the information of the subject if the information of the subject has been stored in the server storage step based on the comparison result.

[0033] Such a program provides the same effects as the first and sixth aspects.

[0034] An information provision program according to a tenth aspect of the present invention is an information provision program that enables communication between a terminal having an imaging unit that can image a target person and a server, a server communication process capable of communicating with the terminal; a server storage process capable of storing information of the subject; A learning process for adding and updating information about the subject and learning; A face detection process capable of detecting a human face, which detects facial information of a subject based on an image or video of the subject captured in the imaging process transmitted from the terminal and information in the learning process; a matching process in which the face information of criminals stored in the criminal database is compared and matched with the face information of the target person detected in the face detection process based on the information in the learning process; This is an information provision program that executes the results of the matching process, and a notification process that notifies the terminal of the target person's information if the matching result indicates that the target person's information is stored in the criminal database.

[0035] Such a program provides the same effects as the information providing system according to the sixth aspect.

[0036] An information providing method according to an eleventh aspect of the present invention is an information providing method in which a terminal having an imaging unit capable of imaging a target person and a server can communicate with each other, The server a server communication step capable of communicating with the terminal; a server storage step capable of storing information of the subject; a learning process of adding and updating information about the subject and learning; a face detection process capable of detecting a human face, which detects facial information of the subject based on an image or video of the subject captured by the imaging unit and transmitted from the terminal and information in the learning process; a matching process for accessing a criminal database and comparing and matching the facial information of criminals stored in the criminal database with the facial information of the target person detected in the face detection process based on the information in the learning process; This is an information provision method that includes the results of the matching process, and a notification process of notifying the terminal of the information of the target person if the matching result indicates that the target person's information is stored in the criminal database.

[0037] Such a method provides the same effects as the information provision system according to the seventh aspect and the information provision program according to the tenth aspect. [Brief explanation of the drawings]

[0038] [Figure 1] 1 is a conceptual diagram of an information providing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a conceptual diagram of an information providing system according to the embodiment. [Figure 3] FIG. 2 is a conceptual diagram of an information providing system according to the embodiment. [Figure 4] FIG. 2 is a conceptual diagram of an information providing system according to the embodiment. [Figure 5] FIG. 2 is a conceptual diagram of an information providing system according to the embodiment. [Figure 6] FIG. 2 is a conceptual diagram of an information providing system according to the embodiment. [Figure 7] 10 is a flowchart of the information providing system according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0039] An information providing system 10 according to this embodiment will be described with reference to the drawings.

[0040] The information providing system 10 according to this embodiment includes a server 100, The server 100 includes a terminal 200 that can communicate with the server 100.

[0041] The server 100 includes a server communication unit 110 that communicates with the terminal, a server storage unit 120 capable of storing information of the subject; a learning unit 130 that learns information about the subject; a face detection unit 140 that detects facial information of a target person based on information from the learning unit 130; a matching unit (150) that compares and matches the face information of the subject stored in the server storage unit (120) with the face information of the subject detected by the face detection unit (140) based on information from the learning unit (130); and a notification unit 160 that notifies the terminal of information.

[0042] The server communication unit 110 is capable of communicating with the terminal 200 wirelessly or via a wired connection.

[0043] In this embodiment, the terminal 200 is a smartphone, and therefore communication is performed wirelessly. Wi-Fi or the like may also be used.

[0044] The server storage unit 120 stores data, such as facial information and personal information of the subject.

[0045] If the target person is not stored in the server storage unit 120, the target person is stored as a new person together with personal information.

[0046] In this case, when and where the new person was encountered is also stored. The content of the conversation with the new person may also be stored.

[0047] The server storage unit 120 stores the detection result of the face information by the face detection unit 140 and the matching result by the matching unit 150 .

[0048] The learning unit 130 then adds and updates this information and learns it.

[0049] The learning unit 130 identifies patterns and rules from the large amount of information input to the server 100 and predicts unknown information.

[0050] Specifically, in this embodiment, the learning unit 130 uses a neural network (convolutional neural network).

[0051] A neural network is a machine learning program or model that makes decisions in a manner similar to the human brain, using processes that mimic the way biological neurons work together to identify phenomena, consider options, and reach conclusions.

[0052] All neural networks consist of layers of nodes, or artificial neurons, including an input layer, one or more hidden layers, and an output layer.

[0053] Each node connects to other nodes and has associated weights and thresholds. If the output of any individual node exceeds the set threshold, that node is activated and sends data to the next layer of the network. If the threshold is not exceeded, the data is not passed to the next layer of the network.

[0054] Specifically, as shown in FIGS. 2 to 4, this embodiment uses a convolutional neural network (CNN).

[0055] Methods used to classify types and categories include R-CNN (Regions with Convolutional Neural Network), Fast R-CNN, Faster R-CNN, SSD (Single Shot MultiBox Detector), YOLO (You Only Look Once), and HOG (Histogram of Oriented Gradients).

[0056] In this embodiment, SSD is used. SSD looks at an image only once and detects objects in that image. The algorithm looks at the image only once, detects objects in the image, and surrounds their location with a rectangle. It is also a method of correcting and deep learning the deviation in the size of the bounding box in the target image using predicted information on the class (type of object). Unlike R-CNN, it is a method called One-Stage that directly detects the position of objects.

[0057] SSDs use convolutional neural networks (CNNs), which are used for image recognition and object detection tasks, and are capable of processing large amounts of image data at high speeds, enabling real-time processing.

[0058] The advantages of SSD are its fast processing speed, ability to detect objects instantly, and ability to detect objects even when there are many objects in the image.

[0059] A convolutional neural network (CNN) is a neural network that contains convolutional and pooling layers.

[0060] The overall picture is that the entire input image is first filtered in the convolution layer. The processed image is sent to the pooling layer, which outputs a single value from a certain region. There are two methods: max pooling and average pooling.

[0061] Finally, several fully connected layers, in which all nodes are connected, are stacked to perform image recognition.

[0062] In other words, the convolutional layer detects the shading patterns in the image (feature extraction such as edge extraction), and the pooling layer considers the object to be the same even if its position changes (allowing for positional deviation), and by combining these layers, features are extracted from the image.

[0063] Furthermore, since image identification cannot be achieved by simply extracting features, "classification based on features" is required for identification, and the role of this classification is played by the fully connected layer and output layer.

[0064] In the fully connected layer, image data whose features have been extracted through the convolutional layer and pooling layer is connected to one node, and a value (feature variable) transformed by an activation function is output. As the number of nodes increases, the number of divisions of the feature space increases, and the number of feature variables that characterize each region also increases.

[0065] In the output layer, the output (feature variables) from the fully connected layer is converted into probabilities using a softmax function, and classification is performed by maximizing the probability of correct classification into each region.

[0066] A convolutional layer is a layer that uses finely divided filters to extract specific features.

[0067] Convolution is the operation of extracting features from an image (or one image of a video) (creating a feature map).

[0068] A filter performs some kind of feature extraction process, such as detecting tilt, a gradient in a certain direction, or detecting a concave or convex part in the center. Multiple filters are applied to all pixels of the original image in the same way, but the resulting image will be completely different from the original image. This output image is called a channel.

[0069] If the input image is a color image, that is, if there are three R, G, and B channels, the filter will be three-dimensional. For example, if the window size is 3x3, a 3x3x3 filter including RGB will be prepared. Applying this filter will create one element of the feature map.

[0070] The image is moved one pixel at a time using a filter, and the filter is checked to see if there is a matching pattern and output. All input data is scanned, and the results of whether there is any similarity to the filter are collected and are called a "feature map (features)."

[0071] A pooling layer is a layer that reduces the size of an image, a process called "downsampling." For example, when using max pooling, the size of the image is reduced by comparing the pixels in each region and extracting the maximum value as the feature of that region. This reduces the number of parameters the network needs to learn.

[0072] In the pooling layer, the maximum (or average) value (single pixel value) is selected from a range (pixel set) of the image generated in the convolution layer, and the resolution is reduced.

[0073] In CNN, input images and corresponding correct data are given as training data, and learning is performed by optimizing parameters (such as convolutional filters and connection weights of fully connected layers).

[0074] As shown in Figure 4, examples of efficient parameter optimization include applying an activation function (ReLU: Rectified Linear Unit) to the image data after filtering, and inserting a dropout layer to avoid overfitting.

[0075] An activation function (ReLU) is a function that sets all output values ​​below 0 to 0, and sends only the part above a certain threshold to the next layer as meaningful information. It is placed after the convolution filter or fully connected layer and has the function of further emphasizing the extracted features.

[0076] The dropout layer prevents overfitting. Overfitting occurs when excessive optimization is performed on features that only the training data has, resulting in a lower accuracy rate for unknown data. The dropout layer prevents overfitting by randomly cutting some of the connections between the nodes in the fully connected layer and the output layer.

[0077] Although only one set of convolutional filter layer and pooling layer is shown in Figures 2 to 4, by stacking these to form multiple layers, newer features can be extracted and recognition accuracy can be improved.

[0078] Depending on the type of input image (input video), various variations are possible, such as adding a layer that performs image preprocessing called a "normalization layer," adding multiple fully connected layers, or omitting the dropout layer.

[0079] The learning unit 130 learns through supervised learning based on a large amount of "human face data" read by the computer (server storage unit 120). By learning "human faces," the learning unit 130 can distinguish and detect a human face when unknown data (face information of a target person) is input using the features learned by the learning unit 130.

[0080] If the learning unit 130 learns that the features of a human face are "eyes," "nose," and "mouth," it can learn to recognize only human faces with the same features of eyes, nose, and mouth among images and videos of cars and animals, and the face detection unit 140 can detect only human faces.

[0081] A face has parts such as eyes, ears, nose, and mouth, and since there are individual differences in size, shape, position, etc., faces have a high degree of uniqueness. By having the learning unit 130 learn these features, the face detection unit 140 can not only identify individuals but also detect their age, gender, etc.

[0082] On the other hand, even for the same person, the facial appearance is not fixed, but changes due to factors such as aging, glasses, makeup, etc. Because of these characteristics, the learning unit 130 predicts changes in the same person from a large amount of data, and the face detection unit 140 detects the face based on the information from the learning unit 130, taking into account the facial characteristics.

[0083] The process of face recognition can be divided into three steps: face detection, feature extraction, and face matching.

[0084] The face detection unit 140 performs face detection. In face detection, after removing areas unnecessary for face authentication from a pre-registered image or a match image, the face is detected and the position of the face is identified.

[0085] In this embodiment, the face detection unit 140 extracts a part of the moving image of the subject captured by the terminal 200 as an image, and detects the face information of the subject.

[0086] The face detection unit 140 may detect the face information of the subject from a video of the subject captured by the terminal 200.

[0087] 5, once the face position is identified, the face detection unit 140 displays a rectangle called a bounding box A to indicate the area of ​​the face. This process makes it possible to obtain information about the position and size of the face.

[0088] The face detection unit 140 extracts features. In feature extraction, parts such as the eyes, nose, and mouth are detected based on the information of the bounding box A obtained by face detection, and facial features are obtained. This process makes it possible to grasp the facial feature points and position information.

[0089] The face detection unit 140 can correct the tilt of the face based on the position information of both eyes. The face detection unit 140 can also estimate the posture and detect changes in facial expression based on the position information of the nose. The face detection unit 140 can also grasp changes in facial expression based on the position information of the mouth.

[0090] In this embodiment, YOLO (You Only Look Once) is used when detecting the face of a target person from a video in the face detection unit 140. It is also possible to combine YOLO and SSD.

[0091] YOLO detects objects present in an image and classifies them by enclosing them in bounding boxes.

[0092] YOLO performs object detection by loading the image only once, making it more efficient at image processing than other algorithms.

[0093] In addition to automatically detecting objects in an image, the system can also categorize them, allowing it to automatically determine the type of object in an image, enabling tasks such as automatically counting and identifying objects in an image to be accomplished.

[0094] The matching unit 150 performs face matching by comparing a registered image registered in the server storage unit 120 with a matching image detected by the face detection unit 140 based on the feature amount, thereby identifying a person.

[0095] The matching unit 150 performs one-to-many authentication and searches for an image having similar features to the match image detected by the face detection unit 140 from among a plurality of registered images stored in the server storage unit 120.

[0096] When the matching unit 150 identifies a person who is the target, the notification unit 160 notifies the terminal 200 of personal information stored in the server storage unit 120, including the target person's name, company name, job title, age, and hobbies, as well as information such as the date and time of the last meeting, location (location information), and name of the meeting.

[0097] If the matching unit 150 is unable to identify the target person, the notification unit 160 notifies the terminal 200 of this fact.

[0098] If the matching unit 150 determines that the target person is the same person as the person stored in the server storage unit 120 (at this time, the matching unit 150 may confirm with the user of the terminal 200 whether or not they are the same person), the data of the person stored in the server storage unit 120 is updated to include the current information.

[0099] Therefore, the information on the subject is updated every time the matching unit 150 identifies a person.

[0100] As the facial information of the subject is updated and the amount of information on multiple different images and videos and information that has been matched in the past increases, the matching accuracy of the matching unit 150 improves (learns).

[0101] Furthermore, the matching unit 150 can utilize information that has been matched in the past, thereby reducing the time required for matching.

[0102] At this time, the server storage unit 120 adds not only the facial information of the target person but also information such as when and where the person met.

[0103] The matching unit 150 may compare and match the moving image of the subject captured by the terminal 200 with the moving image of the subject stored in the server storage unit 120.

[0104] FIG. 6 shows an example in which a match image G detected by the face detection unit 140 is compared with a plurality of registered images stored in the server storage unit 120 based on feature amounts.

[0105] The collation result in FIG. 6 is that the possibility that the subject is A is 0.1 and the possibility that the subject is B is 0.9, and the notification unit 160 notifies the terminal 200 that the possibility that the subject is B is high.

[0106] Furthermore, the matching unit 150 may perform face matching with criminal data stored in the server storage unit 120 or in a database (criminal database) of the Tokyo Metropolitan Police Department.

[0107] If the matching unit 150 identifies the target person as a criminal based on the matching result, the notification unit 160 notifies the terminal 200 of this fact or issues a warning.

[0108] The weapon detection unit 170 detects a weapon as an object in the same manner as the face detection unit 140.

[0109] Weapons are guns, knives, kitchen knives, and other weapons that can kill or injure people, and the range of weapons can be determined by the settings.

[0110] When the weapon detection unit 170 detects a weapon, the notification unit 160 notifies the terminal 200 of this or issues a warning.

[0111] The notification by the notification unit 160 may be a voice notification, or may be displayed on a screen that is a display unit of the terminal 200.

[0112] Furthermore, if the weapon detection unit 170 detects a weapon, the notification unit 160 notifies the police.

[0113] <Terminal 200> The terminal 200 includes a terminal communication unit 210 that can communicate with the server 100, a terminal storage unit 220 capable of storing data; a terminal control unit 230 that controls the terminal 200; a display unit 240 that displays data stored in the terminal storage unit 220 and data accessed from the server 100; an imaging unit 250 that images a subject; a voice transmission unit 260 that transmits the notification content from the notification unit 160 by voice; and an application 270 for operating the information providing system 10.

[0114] An example of the terminal 200 is a so-called smartphone (mobile phone).

[0115] The terminal communication unit 210 is a communication unit that can access the Internet and can communicate with the server 100 .

[0116] The terminal storage unit 220 is a storage unit that stores data, such as access history, IDs, and passwords.

[0117] The terminal storage unit 220 can also store personal information, images, and videos of the subject.

[0118] The terminal control unit 230 controls the terminal 200 .

[0119] The display unit 240 is capable of displaying information on websites and the like and information stored in the server 100.

[0120] For example, the notification content from notification unit 160 may be displayed on display unit 240 as text.

[0121] The imaging unit 250 has a camera function and captures moving images and still images (pictures) of the subject.

[0122] The image or video of the subject captured by the imaging unit 250 is subjected to face detection by the face detection unit 140 of the server 100 .

[0123] The image capturing unit 250 can read the business card of the target person, thereby obtaining personal information such as the target person's name, company name, job title, address, and telephone number.

[0124] The image, video, audio, and business card information of the subject captured by the imaging unit 250 are stored in the server storage unit 120. Note that they may also be stored in the terminal storage unit 220.

[0125] The images, videos, and business card information of the subject captured by the imaging unit 250 may be directly subjected to face detection by the face detection unit 140 of the server 100, or may be temporarily stored in the server memory unit 120 and then compared with past data of the subject stored in the server memory unit 120.

[0126] Specifically, by starting the application 270 and starting the imaging unit 250 via the application 270, the video, still images, and noun information captured by the imaging unit 250 are sent to the server 100, and the face detection unit 140 detects the face of the subject.

[0127] The voice transmitting unit 260 transmits voice through a speaker built into the terminal 200 .

[0128] The voice transmitting unit 260 transmits the result of the verification of the subject by the verifying unit 150 by voice.

[0129] When the matching unit 150 identifies a person who is a target, the voice transmitting unit 260 transmits the content of the notification unit 160 to the terminal 200 by voice.

[0130] The content transmitted by the voice transmitting unit 260 is notified to the user of the terminal 200 through the earphone 300.

[0131] In this embodiment, the earphone 300 connected to the terminal 200 is wireless, but may be wired.

[0132] This allows the user of the terminal 200 to know the information of the target person when they meet again and information about previous encounters with the target person.

[0133] <Information provision system 10 flowchart> FIG. 7 shows a flowchart of the information providing system 10 according to this embodiment.

[0134] First, the learning unit 130 learns "human faces" based on a large amount of "human face data" read by the server storage unit 120 (step S11, learning step).

[0135] The user of the terminal 200 selects the application 270 displayed on the display unit 240 .

[0136] The user of the terminal 200 captures an image of the target person using a camera, which is the imaging unit 250. The image or video of the target person captured by the imaging unit 250 becomes a target for face detection by the face detection unit 140 of the server 100.

[0137] The face detection unit 140 uses the features learned by the learning unit 130 to distinguish and detect a human face when unknown data (face information of a target person) is input (step S12, face detection step).

[0138] In face detection by the face detection unit 140, areas unnecessary for face authentication are removed from images and matching images previously registered in the server storage unit 120, and then a process is performed to detect a face and identify its position (step S13, face detection process).

[0139] Specifically, when the face detection unit 140 identifies the position of the face, it displays a rectangle called a bounding box A to indicate the area of ​​the face (step S14, face detection step).

[0140] The feature extraction by the face detection unit 140 detects parts such as the eyes, nose, and mouth based on the information in the bounding box A obtained by face detection, and obtains the feature amounts of the face (step S15, face detection step).

[0141] The face matching by the matching unit 150 identifies a person by comparing a registered image registered in the server storage unit 120 with a matching image detected by the face detection unit 140 based on the feature amount (step S16, matching step).

[0142] If the matching unit 150 is unable to identify the target person, the notification unit 160 notifies the terminal 200 of this fact (step S17).

[0143] When the matching unit 150 identifies the target person, the notification unit 160 notifies the terminal 200 of personal information stored in the server storage unit 120, including the target person's name, company name, job title, age, and hobbies, as well as information such as the date and time of the last meeting, location, and name of the meeting (step S18, notification process).

[0144] When the terminal 200 receives a notification from the notification unit 160, the voice transmission unit 260 transmits the content of the notification (step S19).

[0145] In this embodiment, the voice transmitting unit 260 transmits voice to the user of the terminal 200 through an earphone.

[0146] The present invention can be implemented in various forms with various improvements, modifications, or variations added thereto without departing from the spirit of the invention. [Explanation of symbols]

[0147] 10 Information provision system 100 servers 120 Server storage unit 121 Personal Information Registration Department 130 Learning Department 140 Face detection unit 150 Collation Unit 160 Notification Department 200 devices 250 Imaging unit 260 Voice Transmission Unit 270 apps (applications) 300 earphones

Claims

1. An information providing system in which a terminal having an imaging unit capable of imaging a target person and a server can communicate with each other, The server a server communication unit capable of communicating with the terminal; a server storage unit capable of storing information of the subject; a learning unit that adds and updates information about the target person based on the detection result and the matching result, and learns the information; a face detection unit that detects facial information of the subject based on the image or video of the subject captured by the imaging unit and transmitted from the terminal and information from the learning unit; a matching unit that compares and matches the face information of the subject stored in the server storage unit with the face information of the subject detected by the face detection unit based on information from the learning unit; a notification unit that notifies the terminal of the result of the collation by the collation unit and, if the result of the collation indicates that information about the subject is stored in the server storage unit, notifies the terminal of the information about the subject; The matching unit performs one-to-N authentication, searches for an image having similar features to the matching image detected by the face detection unit from a plurality of registered images stored in the server storage unit, and when the matching unit identifies the target person, the notification unit notifies the terminal of personal information including the target person's name, company name, job title, age, and hobbies stored in the server storage unit, as well as information including the date and time, location, and name of the last meeting, and when the matching unit cannot identify the target person, the notification unit notifies the terminal that the target person could not be identified, The terminal is an information providing system that transmits the notification content from the notification unit by voice.

2. The imaging unit of the terminal reads the business card of the target person, The server storage unit stores the read business card information, 2. The information providing system according to claim 1, wherein the learning unit learns by adding and updating information on the detection results of face information detected by the face detection unit and the matching results matched by the matching unit for each subject.

3. the face detection unit detects facial information of the subject from a video of the subject captured by the imaging unit of the terminal; 2. The information providing system according to claim 1, wherein the matching unit compares and matches the facial information of the target person detected by the face detection unit with facial information of people in the video stored in the server storage unit.

4. the face detection unit detects facial information of the subject from a part of a video of the subject captured by the imaging unit of the terminal; The information provision system according to claim 1, wherein the matching unit compares and matches a portion of an image of a video of a subject captured by the imaging unit of the terminal with a matching image of the person stored in the server storage unit based on features, and calculates a ratio.

5. The face detection unit removes areas unnecessary for face authentication from an image of a subject registered in advance, detects a face, and performs a process of identifying the position of the face. When the position of the face is identified, a bounding box is displayed on the terminal to indicate the area of ​​the face, and the position and information of the face are obtained, and parts including the eyes, nose, and mouth are detected to obtain facial feature amounts.

2. The information providing system according to claim 1, wherein the matching unit compares and matches the matching image of the subject detected by the face detection unit with the registered image of the person stored in the server memory unit based on the features.

6. An information providing program that enables a terminal having an imaging unit capable of imaging a target person to communicate with a server, The server a server communication process capable of communicating with the terminal; a server storage process capable of storing information of the subject; a learning process for adding and updating information about the subject based on the detection results and the matching results; a face detection process for detecting facial information of the subject based on the image or video of the subject captured by the imaging unit and transmitted from the terminal and information in the learning process; a matching process for comparing and matching the face information of the subject stored in the server storage process with the face information of the subject detected by the face detection process based on information from the learning process; a result of the collation in the collation process, and if the information of the subject person has been stored in the server storage process based on the collation result, a notification process is executed to notify the terminal of the information of the subject person; The matching process performs one-to-N authentication, searches for an image having similar features to the matching image detected in the face detection process from among a plurality of registered images stored in the server storage process, and if the matching process identifies a target person, the notification process notifies the terminal of personal information including the target person's name, company name, job title, age, and hobbies stored in the server storage process, as well as information including the date and time, location, and name of the last meeting, and if the matching process fails to identify the target person, the notification process notifies the terminal of the fact that the target person could not be identified. The terminal includes an information providing program that outputs the notification content in the notification process by voice.

7. An information providing method in which a terminal having an imaging unit capable of imaging a target person can communicate with a server, The server a server communication step capable of communicating with the terminal; a server storage step capable of storing information of the subject; a learning process of adding and updating information about the target person based on the detection results and the matching results; a face detection process for detecting facial information of the subject based on the image or video of the subject captured by the imaging unit and transmitted from the terminal and information in the learning process; a matching step of comparing and matching the face information of the subject stored in the server storage step with the face information of the subject detected in the face detection step based on the information in the learning step; a notification step of notifying the terminal of the result of the comparison in the comparison step, and if the server storage step determines that the information of the subject person has been stored based on the comparison result, The matching step performs one-to-N authentication, searches for an image having similar features to the match image detected in the face detection step from a plurality of registered images stored in the server storage step, and if the target person is identified in the matching step, the notification step notifies the terminal of personal information including the target person's name, company name, job title, age, and hobbies stored in the server storage step, as well as information including the date and time, location, and name of the last meeting, and if the target person cannot be identified in the matching step, the notification step notifies the terminal of the fact that they could not be identified. The information providing method includes the step of transmitting the notification content by voice from the terminal.

Images