Information notification system, information notification program, and information notification method

Abstract

This invention provides a system for providing information about individuals, an information notification program, and an information notification method that accurately perform facial recognition of individuals while adding and updating their information. [Solution] An information provision system including a server and a terminal capable of communicating with the server includes a server communication unit capable of communicating with the terminal, a server storage unit capable of storing information about a subject, a learning unit that adds and updates information about the subject and learns it, a face detection unit that detects the face information of a subject based on images or videos of the subject captured by the imaging unit transmitted from the terminal and information from the learning unit, a matching unit that compares and matches the face information of the subject stored in the server storage unit with the face information of the subject detected by the face detection unit based on information from the learning unit, and a notification unit that notifies the terminal of the results of the matching performed by the matching unit and the information of the subject based on the matching results.

Description

Technical Field

[0001] The present invention relates to an information notification system, an information notification program, and an information notification method.

Background Art

[0002] Conventionally, using a dedicated application such as a business card management application, attribute information such as names, company names, and job titles included in business cards has been database-managed.

[0003] For example, in Patent Document 1, an attribute information of a business card is read by a portable information terminal device having the same size as a business card, and the read attribute information is database-created and stored in advance in a storage unit provided in the portable information terminal device. When a user inputs a company name or the like, a technology is disclosed in which the attribute information of the business card is called from the storage unit and displayed on a screen.

[0004] Further, in Patent Document 2, the glasses-type terminal 1 is a terminal having an augmented reality function, which is worn by the user 2. When the user 2 exchanges business cards with the target person 3, the glasses-type terminal 1 captures an image including a face image of the target person 3 and a business card BC showing the attribute information of the target person 3, and discloses registering the face image of the target person 3 in association with the attribute information. When the user 2 meets the target person 3 again, the glasses-type terminal 1 captures an image of the face image of the target person 3, collates the face image with the registered face image, and if the corresponding face image is registered, the attribute information corresponding to the face image is displayed on the display 1a of the glasses-type terminal 1 together with the target person 3 included in the real field of view.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

[0006] However, in the technology described in Patent Document 1, which databases and manages attribute information contained in business cards, it is not possible to search for attribute information using the database unless some of the attribute information, such as the name and company name of the person with whom the business card was exchanged, is entered.

[0007] Therefore, if you meet someone again after exchanging business cards and cannot remember their name or company name, it becomes difficult to present attribute information based on the database.

[0008] Furthermore, in the invention described in Patent Document 2, the user must always wear a glasses-type terminal, and although the facial image of the person to whom the business card is presented and the attribute information of the person shown on the business card are stored in association, there is a problem that if the person's information is not updated, it is not possible to determine the identity of the person even when meeting them in person.

[0009] Furthermore, although it is stated that the matching unit compares the captured facial image with the registered facial image, it is unclear exactly how the comparison is performed.

[0010] The objective of the present invention is to provide a subject information provision system that accurately performs facial recognition of subjects while adding and updating subject information.

[0011] Another object of the present invention is to provide a subject information provision program that accurately performs facial recognition of subjects while adding and updating subject information.

[0012] Another object of the present invention is to provide a method for providing information about a subject that accurately performs facial recognition of the subject while adding and updating the subject's information. [Means for solving the problem]

[0013] An information provision system according to the first aspect of the present invention is an information provision system in which a terminal having an imaging unit capable of capturing images of a subject and a server can communicate with each other, The aforementioned server, A server communication unit capable of communicating with the aforementioned terminal, A server storage unit capable of storing information about the subject, The learning section adds and updates information about the target individuals for further learning, A face detection unit detects the face information of a subject based on an image or video of the subject captured by the imaging unit transmitted from the terminal, and information from the learning unit. A matching unit compares and matches the subject's face information stored in the server storage unit with the subject's face information detected by the face detection unit, based on information from the learning unit. The information provision system includes the following: the results of the matching by the matching unit, and a notification unit that notifies the terminal of the information of the subject if the information of the subject is stored in the server storage unit based on the matching results.

[0014] The learning unit adds and updates information about subjects captured by the terminal, resulting in a larger dataset of facial data, making it easier to recognize faces and improving the accuracy of facial recognition. This "learning" process, which involves adding and updating information in the learning unit, is what is known as AI (Artificial Intelligence).

[0015] Furthermore, the user of the device can obtain information about individuals they have met in the past, including their names, occupations, ages, and other personal information.

[0016] Therefore, even if the device user has forgotten personal information such as the name or company name of the person in question, the notification unit will immediately provide the device user with information such as the person's name, company name, and when they last met.

[0017] The information providing system according to the second aspect of the present invention is the information providing system according to the first aspect, wherein the learning unit is an information providing system that learns by adding and updating the detection result of the face information detected by the face detection unit and the information of the collation result collated by the collation unit.

[0018] By learning in the learning unit, it becomes easier to recognize the face of the target person, and the face detection accuracy and collation accuracy of the target person are improved.

[0019] The information providing system according to the third aspect of the present invention is the information providing system according to the first aspect, wherein the face detection unit detects the face information of the target person from the video of the target person captured by the imaging unit of the terminal, the collation unit is an information providing system that compares and collates the face information of the target person detected by the face detection unit with the face information of the person in the video stored in the server storage unit.

[0020] Since the amount of information in a video is larger than that in a still image, the face detection accuracy and collation accuracy of the target person are improved.

[0021] In addition, the step of selecting an image of the target person to be compared from the video becomes unnecessary.

[0022] The information providing system according to the fourth aspect of the present invention is the information providing system according to the first aspect, wherein the face detection unit detects the face information of the target person from a part of the video of the target person captured by the imaging unit of the terminal, the collation unit is an information providing system that compares and collates an image of a part of the video of the target person captured by the imaging unit of the terminal with a collation image of a person stored in the server storage unit based on feature amounts.

[0023] There is an advantage that it is easier to collate by using an image of a part of the video as a comparison image.

[0024] The information providing system according to the fifth aspect of the present invention is the information providing system according to the first aspect, wherein The face matching unit is an information provision system that compares and matches the matching image of the subject detected by the face detection unit with the registered image of the person stored in the server storage unit based on feature quantities.

[0025] This method has the advantage of improving matching speed by comparing and matching based on feature data.

[0026] An information provision system according to the sixth aspect of the present invention is an information provision system in which a terminal having an imaging unit capable of capturing images of a subject and a server can communicate with each other, The aforementioned server, A server communication unit capable of communicating with the aforementioned terminal, A server storage unit capable of storing information about the subject, The learning section adds and updates information about the target individuals for further learning, A face detection unit capable of detecting a person's face, which detects the face information of a subject based on an image or video of the subject captured by the imaging unit transmitted from the terminal, and information from the learning unit. A matching unit accesses a criminal database and compares and matches the facial information of criminals stored in the criminal database with the facial information of a target person detected by the facial detection unit, based on information from the learning unit. The information provision system includes the following: the results of the matching by the matching unit, and, if the information of the target person is stored in the criminal database based on the matching results, a notification unit that notifies the terminal of the information of the target person.

[0027] This allows the device user to be warned that a criminal is present.

[0028] The information provision system relating to the seventh aspect of the present invention is an information provision system relating to the sixth aspect, The server includes a weapon detection unit that detects weapons, The system provides information by having the notification unit notify the terminal of the weapon detection unit, or issue a warning, when the weapon detection unit detects a weapon possessed by the subject.

[0029] This allows you to immediately determine whether a criminal photographed by the device's camera is armed.

[0030] An information provision program according to the eighth aspect of the present invention is an information provision program that can communicate with a terminal having an imaging unit capable of capturing images of a subject, A server communication process that can communicate with the aforementioned terminal, A server memory processing system that can store information about the subject, A learning process that adds and updates information about the target individuals, The system is capable of detecting human faces, and performs a face detection process that detects the face information of a subject based on an image or video of the subject captured by the imaging unit transmitted from the terminal, and information from the learning process. A matching process that compares and matches the subject's face information stored in the server storage process with the subject's face information detected by the face detection process, based on the information from the learning process. This is an information provision program that performs the following: the results of the matching process, and, if the server storage process has stored the information of the subject based on the matching results, a notification process that notifies the terminal of the information of the subject.

[0031] Such a program would produce the same effect as the first phase.

[0032] An information provision method relating to the ninth aspect of the present invention is an information provision method that can communicate with a terminal having an imaging unit capable of capturing images of a subject, The aforementioned server, A server communication process that can communicate with the aforementioned terminal, A server storage process that can store information about the subject, A learning process that involves adding and updating information about the target individuals, The system is capable of detecting a person's face, and includes a face detection step that detects the face information of a subject based on an image or video of the subject captured by the imaging unit transmitted from the terminal, and information from the learning step. A matching step which compares and matches the facial information of the subject stored in the server storage step with the facial information of the subject detected by the facial detection step, based on the information from the learning step. The information provision method includes the results of the matching process described above, and a notification process that notifies the terminal of the information of the subject if the information of the subject was stored in the server storage process based on the matching results.

[0033] Such a program would produce the same effect as the first and sixth phases.

[0034] The tenth information provision program relating to a curved surface of the present invention is an information provision program that enables communication between a terminal having an imaging unit capable of imaging a subject and a server, A server communication process that can communicate with the aforementioned terminal, A server memory processing system that can store information about the subject, A learning process that adds and updates information about the target individuals, A face detection process is available that can detect a person's face and detects the face information of the subject based on the image or video of the subject captured by the imaging process transmitted from the terminal and the information from the learning process. A matching process that accesses a criminal database and compares and matches the facial information of criminals stored in the criminal database with the facial information of the target person detected by the facial detection process, based on the information from the learning process. This is an information provision program that performs the following: matching as a result of the matching process, and, if the information of the target person is stored in the criminal database based on the matching result, notifies the terminal of the information of the target person.

[0035] Such a program would have the same effect as an information provision system related to the sixth phase.

[0036] An information provision method relating to the eleventh aspect of the present invention is an information provision method in which a terminal having an imaging unit capable of capturing images of a subject and a server can communicate with each other, The aforementioned server, A server communication process that can communicate with the aforementioned terminal, A server storage process that can store information about the subject, A learning process that involves adding and updating information about the target individuals, The system is capable of detecting a person's face, and includes a face detection step that detects the face information of a subject based on an image or video of the subject captured by the imaging unit transmitted from the terminal, and information from the learning step. A matching step involves accessing a criminal database and comparing and matching the facial information of criminals stored in the criminal database with the facial information of the target person detected by the facial detection step, based on the information from the learning step. The information provision method includes the following steps: a verification step, and if the verification results indicate that the information of the subject is stored in the criminal database, a notification step of notifying the terminal of the information of the subject.

[0037] This method would have the same effect as the information provision system for the seventh phase and the information provision program for the tenth phase. [Brief explanation of the drawing]

[0038] [Figure 1] Conceptual diagram of an information provision system relating to one embodiment of the present invention. [Figure 2] A conceptual diagram of the information provision system according to the same embodiment. [Figure 3] A conceptual diagram of the information provision system according to the same embodiment. [Figure 4] A conceptual diagram of the information provision system according to the same embodiment. [Figure 5] A conceptual diagram of the information provision system according to the same embodiment. [Figure 6] A conceptual diagram of the information provision system according to the same embodiment. [Figure 7] A flowchart of the information provision system according to the same embodiment. [Modes for carrying out the invention]

[0039] The information provision system 10 according to this embodiment will be described with reference to the drawings.

[0040] The information provision system 10 according to this embodiment includes a server 100 and Includes a server 100 and a terminal 200 capable of communicating with it.

[0041] Server 100 includes a server communication unit 110 that communicates with the terminal, A server storage unit 120 that can store information about the subject, A learning unit 130 that learns information about the target person, Based on information from the learning unit 130, the face detection unit 140 detects the face information of the subject, A matching unit 150 compares and matches the subject's face information stored in the server storage unit 120 with the subject's face information detected by the face detection unit 140 based on information from the learning unit 130. It includes a notification unit 160 that notifies the terminal of information.

[0042] The server communication unit 110 can communicate with the terminal 200 wirelessly or via a wired connection.

[0043] In this embodiment, terminal 200 is a smartphone and therefore communicates wirelessly. Wi-Fi or similar technologies may be used.

[0044] The server memory unit 120 stores data, including facial information and personal information of the subject.

[0045] If the target person is not stored in the server storage unit 120, they will be stored as a new person along with their personal information.

[0046] In this case, the time and place of meeting the new person are also remembered. The content of the conversation with the new person may also be remembered.

[0047] The server storage unit 120 stores the face information detection results from the face detection unit 140 and the matching results from the matching unit 150.

[0048] Then, in the learning section 130, this information is added and updated for learning.

[0049] The learning unit 130 identifies patterns and rules from the large amount of information input to the server 100 and predicts unknown information.

[0050] Specifically, in this embodiment, the learning unit 130 uses a neural network (convolutional neural network).

[0051] A neural network is a machine learning program or model that makes decisions in a way similar to the human brain, using a process that mimics how biological neurons work together to identify phenomena, consider options, and draw conclusions.

[0052] All neural networks consist of layers of nodes, or artificial neurons, each containing one input layer, one or more hidden layers, and one output layer.

[0053] Each node connects to other nodes, and associated weights and thresholds are set. When the output of any individual node exceeds a set threshold, that node is activated and sends data to the next layer of the network. If the threshold is not exceeded, the data is not passed to the next layer of the network.

[0054] Specifically, as shown in Figures 2 to 4, this embodiment uses a convolutional neural network (CNN).

[0055] Methods for classifying types and categories include R-CNN (Regions with Convolutional Neural Network), Fast R-CNN, Faster R-CNN, SSD (Single Shot MultiBox Detector), YOLO (You Only Look Once), and HOG (Histogram of Oriented Gradients).

[0056] In this embodiment, an SSD is used. The SSD looks at an image only once and detects objects within that image. The algorithm looks at an image only once, detects objects within that image, and encloses their positions with rectangles. Furthermore, it is a method that corrects and uses deep learning to correct the size discrepancies of the bounding boxes in the target image using prediction information of the class (type of object). Unlike R-CNN, it is a one-stage method that directly detects the position of objects.

[0057] SSDs utilize convolutional neural networks (CNNs), making them suitable for tasks such as image recognition and object detection. Furthermore, SSDs can process large amounts of image data at high speed, enabling real-time processing.

[0058] The advantages of SSDs are their fast processing speed, the ability to detect objects instantly, and their ability to detect objects even when there are many objects in the image.

[0059] A convolutional neural network (CNN) is a neural network that includes convolutional layers and pooling layers.

[0060] The overall process involves first filtering the entire input image in a convolutional layer. The processed image is then sent to a pooling layer, where a single value is output from a specific region. Two methods for this are maximum pooling and average pooling.

[0061] Finally, several fully-connected layers, each containing all node connections, are stacked on top of each other to perform image recognition.

[0062] In other words, the convolutional layer detects the grayscale pattern of the image (feature extraction such as edge detection), and the pooling layer considers an object to be the same even if its position changes (allowing for positional shifts). By combining these layers, features are extracted from the image.

[0063] Furthermore, simply extracting features is not enough to identify an image; therefore, "feature-based classification" is necessary for identification, and the fully connected layer and output layer are responsible for this classification.

[0064] In a fully connected layer, image data from which feature portions have been extracted through convolutional and pooling layers are combined into a single node, and values ​​(feature variables) transformed by an activation function are output. As the number of nodes increases, the number of divisions in the feature space increases, and the number of feature variables characterizing each region increases.

[0065] In the output layer, the output (feature variables) from the fully connected layer is converted into probabilities using the softmax function, and classification is performed by maximizing the probability of correctly classifying each region.

[0066] A convolutional layer is a layer that uses finely divided filters to extract specific features.

[0067] Convolution is the process of extracting features from an image (or a single frame from a video) (creating a feature map).

[0068] A filter performs some kind of feature extraction process, and can take many forms, such as detecting tilt, gradients in a certain direction, or central concavity or convexity. Multiple filters are applied similarly to all pixels of the original image, but the resulting image will be completely different from the original. This output image is called a channel.

[0069] If the input image is a color image, meaning it has three R, G, and B channels, the filter will be three-dimensional. For example, if the window size is 3x3, a 3x3x3 filter including RGB will be prepared. Applying this filter creates one element of the feature map.

[0070] The image is moved pixel by pixel using a filter, and the results of checking for patterns that fit the filter are output. The results of scanning all input data and determining whether there were any similarities with the filter are called a "feature map (features)".

[0071] A pooling layer is a layer that reduces the size of an image, a process called "downsampling." For example, when using maximum pooling (decimation), the pixels of each region are compared, and the maximum value among them is extracted as the feature of that region, thereby reducing the image size. This reduces the number of parameters that the network needs to learn.

[0072] In the pooling layer, the maximum value (or average value) (1 pixel value) is selected from a certain range (set of pixels) of the image generated by the convolutional layer, and the resolution is reduced.

[0073] In a CNN, input images and their corresponding ground truth data are provided as training data, and learning is performed by optimizing parameters (such as the connection weights of convolutional filters and fully connected layers).

[0074] As shown in Figure 4, examples of how to efficiently optimize this parameter include applying an activation function (ReLU: Rectified Linear Unit) to the filtered image data and adding a dropout layer to avoid overfitting.

[0075] An activation function (ReLU) is a function that sets all output values ​​less than 0 to 0, sending only the portion above a certain threshold as meaningful information to the next layer. It is placed after convolutional filters and fully connected layers and has the function of further emphasizing the extracted features.

[0076] The dropout layer prevents overfitting. Overfitting occurs when the model over-optimizes features that are only present in the training data, leading to a decrease in accuracy on unfamiliar data. The dropout layer prevents overfitting by randomly disconnecting some of the connections between the fully connected layer's nodes and the output layer.

[0077] Although only one pair of convolutional filter layers and pooling layers are shown in Figures 2 to 4, stacking these to create a multi-layered structure allows for the extraction of even more new features and improves recognition accuracy.

[0078] Depending on the type of input image (or video), various variations are possible, such as adding a layer called a "normalization layer" for image preprocessing, using multiple fully connected layers, or omitting the dropout layer.

[0079] The learning unit 130 learns using supervised learning based on a large amount of "human face data" read by the computer (server memory unit 120). By learning "human faces," the learning unit 130 can identify and detect a human face when unknown data (face information of a target person) is input, based on the features learned by the learning unit 130.

[0080] If the learning unit 130 learns "eyes," "nose," and "mouth" as features of a human face, it can learn to recognize only human faces with the same eye, nose, and mouth features in a collection of images and videos of cars and animals, and the face detection unit 140 can then detect only human faces.

[0081] Faces have various features such as eyes, ears, nose, and mouth, and their size, shape, and position vary from person to person, giving each face a high degree of individuality. By learning these features, the learning unit 130 can not only identify individuals but also detect age, gender, and other characteristics.

[0082] On the other hand, even for the same person, facial appearance is not fixed but changes due to factors such as age, glasses, and makeup. Given these characteristics, the learning unit 130 predicts changes in the same person from a large amount of data, and the face detection unit 140 detects faces based on the information from the learning unit 130, taking into account the characteristics of the face.

[0083] The facial recognition process can be divided into three steps: face detection, feature extraction, and face matching.

[0084] The face detection unit 140 performs face detection. In face detection, it removes areas unnecessary for face recognition from pre-registered images or matching images, then detects faces and determines their positions.

[0085] In this embodiment, the face detection unit 140 extracts a portion of the video of the subject captured by the terminal 200 as an image and detects the subject's face information.

[0086] The face detection unit 140 may also detect the subject's face information from the video of the subject captured by the terminal 200.

[0087] As shown in Figure 5, when the face detection unit 140 can identify the position of a face, it displays a rectangle called a bounding box A to indicate the area of ​​the face. This process allows the system to obtain information about the position and size of the face.

[0088] The face detection unit 140 performs feature extraction. In feature extraction, based on the bounding box A information obtained from face detection, it detects parts such as the eyes, nose, and mouth, and calculates the facial features. This process makes it possible to understand the feature points and positional information of the face.

[0089] The face detection unit 140 can correct the tilt of the face using the position information of both eyes. Furthermore, the face detection unit 140 can estimate posture and detect changes in facial expression from the position information of the nose. Additionally, the face detection unit 140 can understand changes in facial expression from the position information of the mouth.

[0090] In this embodiment, when the face detection unit 140 detects a subject's face from a video, YOLO (You Only Look Once) is used. A combination of YOLO and SSD is also possible.

[0091] YOLO detects objects within an image and classifies them by enclosing them in bounding boxes.

[0092] YOLO performs object detection by reading an image only once, making it more efficient in image processing than other algorithms.

[0093] Furthermore, it can not only automatically detect objects in an image but also categorize them, allowing for automatic identification of the types of objects within an image. This enables tasks such as automatically counting objects in an image and identifying objects within an image.

[0094] The matching unit 150 performs face matching. In face matching, the unit identifies a person by comparing the registered image stored in the server storage unit 120 with the feature quantities of the matching image detected by the face detection unit 140.

[0095] The matching unit 150 performs one-to-many authentication and searches for images with similar feature quantities to the matching image detected by the face detection unit 140 from among multiple registered images stored in the server storage unit 120.

[0096] If the matching unit 150 identifies the person in question, the notification unit 160 notifies the terminal 200 of the person's personal information, including their name, company name, job title, age, and hobbies, as well as information such as the date and time of the last meeting, the location (location information), and the name of the meeting, which is stored in the server storage unit 120.

[0097] If the matching unit 150 is unable to identify the person in question, the notification unit 160 notifies the terminal 200 of this fact.

[0098] If the matching unit 150 determines that the person in question is the same person as the person stored in the server storage unit 120 (at this time, the matching unit 150 may also confirm with the user of the terminal 200 whether or not they are the same person), the data of the person stored in the server storage unit 120 is updated to include the current information.

[0099] Therefore, the subject's information is updated each time the matching unit 150 identifies a person.

[0100] As the facial information of the subject is updated, and the amount of information from multiple different images and videos, as well as previously matched information, increases, the matching accuracy of the matching unit 150 improves (it learns).

[0101] Furthermore, the matching unit 150 can utilize previously matched information, thereby shortening the matching time.

[0102] Furthermore, at this time, the server memory unit 120 adds not only the subject's facial information but also information such as when and where they met.

[0103] The matching unit 150 may also compare and verify the video of the subject captured by the terminal 200 with the video of the subject stored in the server storage unit 120.

[0104] Figure 6 shows an example of comparing and matching the matching image G detected by the face detection unit 140 with multiple registered images stored in the server storage unit 120 based on their features.

[0105] The matching results in Figure 6 show that the probability of the subject being A is 0.1 and the probability of being B is 0.9. The notification unit 160 notifies the terminal 200 that there is a high probability that the subject is B.

[0106] The matching unit 150 may also perform facial matching with criminal data stored in the server storage unit 120 or the Metropolitan Police Department's database (criminal database).

[0107] If the matching unit 150 identifies the subject as a criminal based on the matching results, the notification unit 160 notifies the terminal 200 of this fact or issues a warning.

[0108] The weapon detection unit 170 detects weapons, which are physical objects, in the same manner as the face detection unit 140.

[0109] Weapons include handguns, knives, kitchen knives, and other items capable of killing or injuring people, and the range of these weapons can be determined by the settings.

[0110] If the weapon detection unit 170 detects a weapon, the notification unit 160 notifies the terminal 200 of this fact or issues a warning.

[0111] The notification from the notification unit 160 may be an audio notification, or it may be displayed on the screen, which is the display unit of the terminal 200.

[0112] Furthermore, if the weapon detection unit 170 detects a weapon, the notification unit 160 will notify the police.

[0113] <Terminal 200> Terminal 200 includes a terminal communication unit 210 that can communicate with server 100, A terminal storage unit 220 capable of storing data, A terminal control unit 230 controls terminal 200, A display unit 240 that displays data stored in the terminal storage unit 220 and data accessed from the server 100, An imaging unit 250 for imaging the subject, A voice transmission unit 260 that transmits the notification content from the notification unit 160 by voice, This includes an application 270 for operating the information provision system 10.

[0114] An example of terminal 200 would be a so-called smartphone (mobile phone).

[0115] The terminal communication unit 210 is a communication unit that can access the internet and communicate with the server 100.

[0116] The terminal storage unit 220 is a storage unit that stores data. For example, it stores access history, IDs, and passwords.

[0117] Furthermore, the terminal storage unit 220 can store the subject's personal information, images, and videos.

[0118] The terminal control unit 230 controls the terminal 200.

[0119] The display unit 240 can display information from websites or within the server 100.

[0120] For example, the notification content from the notification unit 160 may be displayed as text on the display unit 240.

[0121] The imaging unit 250 has camera functionality and captures video and still images (pictures) of the subject.

[0122] Images and videos of the subject captured by the imaging unit 250 are subject to face detection by the face detection unit 140 of the server 100.

[0123] The imaging unit 250 can read the subject's business card. This allows the system to obtain personal information such as the subject's name, company name, job title, address, and telephone number.

[0124] Images, videos, audio, and business card information of the subject captured by the imaging unit 250 are stored in the server storage unit 120. Alternatively, they may also be stored in the terminal storage unit 220.

[0125] The images, videos, and business card information of the subject captured by the imaging unit 250 may be directly used for face detection by the face detection unit 140 of the server 100, or they may be temporarily stored in the server storage unit 120 and then compared with past data of the subject stored in the server storage unit 120.

[0126] Specifically, by launching the application 270 and activating the imaging unit 250 via the application 270, videos, still images, and noun information captured by the imaging unit 250 are sent to the server 100, and the face detection unit 140 detects the face of the subject.

[0127] The voice transmission unit 260 transmits sound through the speaker built into the terminal 200.

[0128] The voice transmission unit 260 transmits the results of the matching of the subject by the matching unit 150 in voice.

[0129] When the matching unit 150 identifies the target person, the voice transmission unit 260 transmits a voice message containing the content that the notification unit 160 will notify the terminal 200 of.

[0130] The content transmitted by the voice transmission unit 260 is notified to the user of the terminal 200 through the earphone 300.

[0131] In this embodiment, the earphones 300 connected to the terminal 200 are wireless, but they may also be wired.

[0132] This allows the user of terminal 200 to access information about the target person and information about their previous encounter when they meet the target person again.

[0133] <Information Provision System 10 Flowchart> Figure 7 shows a flowchart of the information provision system 10 according to this embodiment.

[0134] First, the learning unit 130 learns "human faces" based on the large amount of "human face data" read by the server storage unit 120 (Step S11, learning process).

[0135] The user of terminal 200 selects the application 270 displayed on the display unit 240.

[0136] The user of terminal 200 takes an image of the subject using the camera, which is the imaging unit 250. The image or video of the subject captured by the imaging unit 250 is then used for face detection by the face detection unit 140 of server 100.

[0137] The face detection unit 140 identifies and detects a human face when unknown data (face information of a subject) is input, using the features learned by the learning unit 130 (step S12, face detection process).

[0138] In the face detection process of the face detection unit 140, areas unnecessary for face authentication are removed from images and matching images that have been previously registered in the server storage unit 120, and then a face is detected and its position is determined (step S13, face detection process).

[0139] Specifically, when the face detection unit 140 can identify the position of a face, it displays a rectangle called a bounding box A to indicate the area of ​​the face (step S14, face detection step).

[0140] The feature extraction of the face detection unit 140 is performed by detecting parts such as the eyes, nose, and mouth based on the information within the bounding box A obtained from face detection, and determining the facial features (step S15, face detection process).

[0141] The face matching unit 150 identifies a person by comparing the registered image stored in the server storage unit 120 with the feature quantities of the matching image detected by the face detection unit 140 (step S16, matching process).

[0142] If the matching unit 150 is unable to identify the person in question, the notification unit 160 notifies the terminal 200 of this fact (step S17).

[0143] If the matching unit 150 identifies the person in question, the notification unit 160 notifies the terminal 200 of the personal information of the person, including their name, company name, job title, age, and hobbies, as well as information such as the date, time, place, and name of the last meeting, which is stored in the server storage unit 120 (step S18, notification process).

[0144] When terminal 200 receives a notification from notification unit 160, voice transmission unit 260 transmits the content of that notification (step S19).

[0145] In this embodiment, the voice transmission unit 260 transmits audio to the user of the terminal 200 through the earphones.

[0146] The present invention can also be implemented in various improved, modified, or altered forms without departing from its spirit. [Explanation of Symbols]

[0147] 10. Information Provision System 100 servers 120 Server Storage Unit 121 Personal Information Registration Department 130 Learning Department 140 Face detection unit 150 Verification Unit 160 Notification Department 200 terminals 250 Imaging Unit 260 Voice Transmitter 270 apps (applications) 300 earphones

Claims

1. An information provision system that enables communication between a terminal having an imaging unit capable of capturing images of a subject and a server, The aforementioned server, A server communication unit capable of communicating with the aforementioned terminal, A server storage unit capable of storing information about the subject, The learning section adds and updates information about the target individuals for further learning, A face detection unit capable of detecting a person's face, which detects the face information of a subject based on an image or video of the subject captured by the imaging unit transmitted from the terminal, and information from the learning unit. A matching unit compares and matches the subject's face information stored in the server storage unit with the subject's face information detected by the face detection unit, based on information from the learning unit. An information provision system including the results of the matching by the matching unit, and a notification unit that notifies the terminal of the information of the subject if the information of the subject is stored in the server storage unit based on the matching results.

2. The information provision system according to claim 1, wherein the learning unit learns by adding and updating information such as the detection results of face information detected by the face detection unit and the matching results of the matching unit.

3. The face detection unit detects the subject's face information from the video of the subject captured by the imaging unit of the terminal, The information provision system according to claim 1, wherein the matching unit compares and matches the face information of the subject detected by the face detection unit with the face information of the person in the video stored in the server storage unit.

4. The face detection unit detects the subject's face information from a portion of the subject's video captured by the imaging unit of the terminal, The information provision system according to claim 1, wherein the matching unit compares and matches a portion of the video of the subject captured by the imaging unit of the terminal with a matching image of the person stored in the server storage unit based on feature quantities.

5. The information provision system according to claim 1, wherein the face matching unit compares and matches the matching image of the subject detected by the face detection unit with the registered image of the person stored in the server storage unit based on feature quantities.

6. An information provision system that enables communication between a terminal having an imaging unit capable of capturing images of a subject and a server, The aforementioned server, A server communication unit capable of communicating with the aforementioned terminal, A server storage unit capable of storing information about the subject, The learning section adds and updates information about the target individuals for further learning, A face detection unit capable of detecting a person's face, which detects the face information of a subject based on an image or video of the subject captured by the imaging unit transmitted from the terminal, and information from the learning unit. A matching unit accesses a criminal database and compares and matches the facial information of criminals stored in the criminal database with the facial information of a target person detected by the facial detection unit, based on information from the learning unit. An information provision system including the results of the matching by the matching unit, and a notification unit that notifies the terminal of the information of the target person if the information of the target person is stored in the criminal database based on the matching results.

7. The server includes a weapon detection unit that detects weapons, The information provision system according to claim 6, wherein if the weapon detection unit detects a weapon owned by the subject, the notification unit notifies the terminal of that fact or a warning.

Images