User identity identification method, electronic device and medium

Abstract

The present invention provides a user identity recognition method, electronic device and medium, the method comprising: receiving a video stream sent by a shooting device, and performing human body recognition on images in the video stream; when it is determined that a human body is recognized, extracting human body characteristic data, and comparing it with the human body characteristic data in a registration database, and when it is determined that a corresponding registered user is recognized, initiating a response corresponding to the detection of the registered user; when it is determined that the corresponding registered user is not recognized, tracking the human body in the image according to the received video stream, and collecting other identity recognition information corresponding to the human body; when it is determined that the corresponding registered user is recognized according to other identity recognition information, storing the human body characteristic data in the registration database. By updating the registration database using this method, various situations of the human body can be covered, the accuracy of human body recognition can be increased, the effect of human body recognition can be optimized, the recognition can be made more and more accurate, and the problem of difficulty in human body recognition in various scenarios can be solved.

Description

Technical Field

[0001] The present invention relates to the field of cross-media fusion identification, and in particular to a user identity identification method, electronic equipment and medium. Background Art

[0002] At present, the scope of application of smart home in various fields is expanding, and various manufacturers are also developing towards the goal of realizing whole-house intelligence. The premise of realizing whole-house intelligence is the real-time identification and tracking of home users in the house. Only by knowing the identity, location, and living habits of home users in the house can we better understand the user's behavioral habits and preferences, and thus provide users with more accurate services.

[0003] At present, the user's identity is generally determined by using a camera device in the house to first identify the face and then the body. In daily home scenes, it is difficult for the camera device to directly capture the user's facial information in many cases. When the facial information cannot be recognized, the user's identity is generally determined by recognizing the body. During the initial registration, usually only a basic human body image is recorded by the camera device for registration. The registered human body image is difficult to cover the human body postures and features in various scenes. Therefore, the features extracted by the recognition device in the initial registered image are not rich. This leads to that in complex home scenes, even if the human body image is captured, it is difficult to match the captured human body image with the initially recorded human body image due to the complex indoor lighting, the non-fixed installation position of the camera device, the changeable position and posture of the person, and the occlusion. This makes it difficult for the human body to accurately judge the identity information of the household personnel in complex daily home life. Simply using the initially registered face or body image to identify the identity of the person has limitations, and the recognition rate of the person identity is low and the false detection rate is high. Summary of the invention

[0004] As described in the background technology, the features extracted by the recognition device in the initial registered image are not rich. This results in that in complex home scenes, even if a human image is captured, it is difficult to match the captured human image with the initially recorded human image due to the complex lighting conditions in the room, the non-fixed installation position of the shooting device, the variable positions and postures of the people, and occlusion. This makes it difficult for the human body to accurately judge the identity information of household occupants in complex daily home life. Simply using the initially registered face or body image to identify a person has limitations, and has a low recognition rate and a high false detection rate. The present invention solves the problem of difficulty in human recognition in various scenarios.

[0005] A first aspect of the present invention provides a method for identifying a user identity, the method comprising:

[0006] Receiving a video stream sent by a camera, and performing human body recognition on images in the video stream;

[0007] When it is determined that a human body is recognized, human body feature data is extracted and compared with the human body feature data in the registration database. When it is determined that a corresponding registered user is recognized, a response corresponding to the detection of the registered user is initiated;

[0008] When it is determined that the corresponding registered user is not identified, tracking the human body in the image according to the received video stream, and collecting other identity recognition information corresponding to the human body;

[0009] When it is determined that the corresponding registered user is identified based on the other identity recognition information, the human body feature data is stored in a registration database.

[0010] In some embodiments of the present application, collecting the other identity recognition information corresponding to the human body includes at least one of the following steps:

[0011] Performing face recognition corresponding to the human body on the image in the video stream, and extracting face data features when it is determined that a face is recognized;

[0012] An audio stream sent by an audio acquisition device is received, and voiceprint recognition is performed on speech frames in the audio stream to obtain voiceprint data features.

[0013] In some embodiments of the present application, determining to identify a corresponding registered user according to the other identity identification information includes at least one of the following steps:

[0014] When it is determined that a face is recognized, facial feature data is extracted, and compared with the facial feature data in the registration database to determine the facial similarity. If the facial similarity is greater than the face threshold, it is determined that the first registered user corresponding to the facial feature data is recognized;

[0015] When it is determined that an audio stream is received, voice frames in the audio stream are extracted and compared with the voiceprint feature data in the registration database to determine the voiceprint similarity. If the voiceprint similarity is greater than the voiceprint threshold, it is determined that a second registered user corresponding to the voiceprint feature data is identified.

[0016] In some embodiments of the present application, the method further comprises:

[0017] If it is determined that the first registered user is identified and the second registered user is identified, and the first registered user is inconsistent with the second registered user, the registered user corresponding to the higher similarity between the face similarity and the voiceprint similarity is determined to be the identified first registered user.

[0018] In some embodiments of the present application, tracking a human body in the image according to the received video stream and collecting other identity recognition information corresponding to the human body include:

[0019] Determining a first angle of the tracked human body relative to the camera device;

[0020] If face recognition is performed on the image in the video stream, a second angle of the face in the image relative to the camera is determined, and when the second angle is consistent with the second angle, facial feature data of the face is extracted;

[0021] When receiving an audio stream sent by an audio collection device, determine a third angle of the sound direction corresponding to the audio stream relative to the collection direction of the audio collection device, and when the third angle is consistent with the first angle, extract the voice frame of the audio stream and perform voiceprint recognition.

[0022] In some embodiments of the present application, determining a first angle of a tracked human body relative to a camera device includes:

[0023] calculate

[0024] Among them, α is the horizontal field of view angle of the shooting device, w is the horizontal number of pixels in the image, and x is the horizontal coordinate value of the human body in the image.

[0025] In some embodiments of the present application, human body recognition is performed on an image in a video stream, including:

[0026] Every n frames of images, the images in the video stream are input into the deep learning model YOLO to obtain images with marked human body areas. The deep learning model YOLO takes images with marked human body areas as input and trains the network model with the goal of outputting the human body areas marked in the images, where n is a positive integer.

[0027] In some embodiments of the present application, extracting human feature data and comparing it with human feature data in a registration database to determine and identify a corresponding registered user includes:

[0028] Extract human feature data, compare it with human feature data in the registration database, and determine human similarity;

[0029] If the human body similarity is greater than the human body threshold, it is determined that a third registered user corresponding to the human body feature data is identified.

[0030] According to one aspect of an exemplary embodiment, an electronic device is provided, the electronic device comprising at least a processor and a memory, the processor being configured to implement the following steps when executing a computer program stored in the memory:

[0031] Receiving a video stream sent by a camera, and performing human body recognition on images in the video stream;

[0032] When it is determined that a human body is recognized, human body feature data is extracted and compared with the human body feature data in the registration database. When it is determined that a corresponding registered user is recognized, a response corresponding to the detection of the registered user is initiated;

[0033] When it is determined that the corresponding registered user is not identified, tracking the human body in the image according to the received video stream, and collecting other identity recognition information corresponding to the human body;

[0034] When it is determined that the corresponding registered user is identified based on the other identity recognition information, the human body feature data is stored in a registration database.

[0035] According to one aspect of an exemplary implementation, a computer-readable storage medium is provided, wherein computer program instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer is caused to execute the steps executed in any of the above embodiments.

[0036] In the above-mentioned embodiments of the present application, by continuously updating the human feature data in the registration database, since the added human features are images that cannot be recognized by human body recognition, various situations of the human body can be covered, and the accuracy of human body recognition can be increased. Through the dynamic update of the registration database, the effect of human body recognition can be optimized to make it more and more accurate. In addition, since the installation positions of various shooting devices in family scenes vary greatly, it is difficult to see the faces of family members in most cases, and voiceprints are not easy to extract. Therefore, after optimizing human body recognition, the recognition accuracy can be effectively improved. After a period of time, accurate identity confirmation can be performed even through human body recognition alone.

[0037] On the basis of conforming to the common sense in the art, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a structural diagram of a user identity recognition system;

[0039] Figure 2 A flowchart of the steps of a user identification method;

[0040] Figure 3 A complete flow chart of steps for a user identification method;

[0041] Figure 4 The present invention is a schematic diagram of the structure of an electronic device. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present invention will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present invention, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present invention, "multiple" means two or more than two.

[0043] In the following, the terms "first" and "second" are used for descriptive purposes only and shall not be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present invention, unless otherwise specified, "plurality" means two or more.

[0044] In actual application, smart home hopes to achieve whole-house intelligence, that is, to directly identify the user's identity through the camera, so as to realize real-time identification and tracking of home users in the house. Only by knowing the identity, location, and living habits of home users in the house can we better understand the user's behavioral habits and preferences, so as to provide users with more accurate services.

[0045] The following are explanations of some of the terms that appear in this article:

[0046] The term "human body detection algorithm" in the embodiment of the present invention is to draw the border of the object in the image through the human body detection model, identify the physical category of the object in the image and output the position parameters of the object, wherein the position parameters can frame a rectangular box representing the location of the object.

[0047] Figure 1 The structure of the user identity recognition system provided in the embodiment of the present application is exemplified.

[0048] As shown in the figure, various types of intelligent monitoring devices (such as the audio collection device 101 and the shooting device 102 shown in the figure) are connected to the user identification device 103, and the user identification device 103 is connected to the server 104 through a network (not shown in the figure). The server 104 can also communicate with the user's mobile terminal 106 through a mobile communication network 105. Here, the server 104 can be allowed to communicate with the mobile terminal 106 through a local area network (LAN), a wireless local area network (WLAN) and a mobile terminal 106. In some application scenarios, the intelligent monitoring device and the intelligent speaker can be connected to the user identification device through a local area network, and the user identification device can be connected to the server through the Internet.

[0049] Among them, the mobile terminal can be a personal computer, mobile phone, tablet computer, notebook or other computer device with certain computing power and communication function; the smart display device can be a smart TV, smart monitor or other device; the smart home management device can be a management device such as a smart housekeeper that can control electronic home devices.

[0050] The mobile terminal involved in the embodiment of the present application may be a device that provides voice and / or data connectivity to a user, a handheld device with a wireless connection function, or other processing devices connected to a wireless modem. In different systems, the name of the mobile terminal may also be different. For example, in a 5G system, the terminal device may be called UE (User Equipment). The wireless terminal device can communicate with one or more core networks via RAN (Radio Access Network). The wireless terminal device may be a mobile terminal device, such as a mobile phone (or "cellular" phone) and a computer with a mobile terminal device. For example, it may be a portable, pocket-sized, handheld, computer-built-in or vehicle-mounted mobile device that exchanges language and / or data with a wireless access network. For example, PCS (Personal Communication Service) phone, cordless phone, SIP (Session Initiated Protocol) phone, WLL (Wireless Local Loop) station, PDA (Personal Digital Assistant) and other devices. The wireless terminal device may also be referred to as a system, a subscriber unit (Subscriber Unit), a subscriber station (Subscriber Station), a mobile station (Mobile Station), a mobile station (Mobile), a remote station (Remote Station), an access point (Access Point), a remote terminal device (Remote Terminal), an access terminal device (Access Terminal), a user terminal device (User Terminal), a user agent (User Agent), and a user device (User device), but is not limited in the embodiments of the present application.

[0051] The device connected to the user identification device 103 that has a monitoring function (such as an audio and video data collection function) can be called an intelligent monitoring device (such as an audio collection device 101, a shooting device 102). In some embodiments, the monitoring function and the sound broadcasting function may also be integrated into one user identification device. For example, a smart speaker can have both a monitoring function (such as an audio collection device provided in the smart speaker) and a sound playback function. It should be noted that in the embodiments of the present application, the classification and naming of the devices are only examples and do not constitute a limitation to the present application.

[0052] The type of intelligent monitoring equipment can be one or multiple (two or more), for example, the intelligent monitoring equipment can include an audio acquisition device with communication function, a camera, etc. The number of intelligent monitoring equipment can be one or more. Figure 1 Only the audio collection device 101 and the camera device 102 are shown as examples. The audio collection device and the camera device can be fixedly installed in the area to be monitored, such as the living room, bedroom, etc. in the house.

[0053] The intelligent monitoring device has data collection and communication functions, and can send the collected monitoring data to the user identification device. In some examples, the intelligent monitoring device includes a data collector and a communicator connected thereto, the data collector can be used to collect monitoring data, and the communicator can send the monitoring data to the user identification device. In other examples, the intelligent monitoring device includes a data collector, a communicator, and a processor connected to the data collector and the communicator, wherein the data collector is used to collect monitoring data, the processor processes the collected data, and the communicator sends the monitoring data collected by the data collector and / or the information processed by the processor to the user identification device 103.

[0054] Among them, the audio collection device in the intelligent monitoring equipment can be a single microphone or a microphone array, such as a microphone array with a surround structure formed by multiple microphones. The microphone array has the characteristics of beamforming, noise suppression and speech enhancement in a specific beam direction, which can meet the needs of sound data collection at a long distance, high quality and in a specific direction, ensuring a higher success rate and accuracy of speech recognition. At the same time, based on the beamforming characteristics of the microphone array, the position of the sound source can be located according to the difference in sound data collected by each microphone in the microphone array.

[0055] The shooting device in the intelligent monitoring equipment can be a color camera, a depth camera, or a combination of the two. The color camera can be an ordinary color camera or a wide-angle color camera. The depth camera can be a binocular camera, a structured light camera, or a camera based on time of flight (TOF).

[0056] In some embodiments, the shooting device can be arranged on a gimbal to control the shooting angle of the device. The gimbal is a supporting component of the shooting device, and the gimbal can be driven by a control motor to rotate in different dimensions or angles, thereby driving the shooting device to adjust the shooting angle, so as to achieve tracking of the target for shooting.

[0057] In some embodiments, the intelligent monitoring device may also include a device capable of acquiring audio data and video data, that is, an audio acquisition device and a camera may be integrated. In other embodiments, the smart speaker may also have an audio data acquisition function, such as an audio acquisition device integrated in the smart speaker, and the smart speaker with an audio acquisition function may also be used as an intelligent monitoring device. The embodiments of the present application do not limit the appearance of the intelligent monitoring device.

[0058] The user identification device 103 may be an independent device, or may be integrated with other devices, or may be implemented by adding the functions provided by the embodiments of the present application to other devices (such as a smart home gateway or a set-top box).

[0059] The user identification device 103 may have functions such as sound data processing, image and video data processing, and data monitoring and control. The sound data processing function is mainly used to process and analyze the sound data detected by the audio acquisition device 101, such as audio noise reduction, echo cancellation, and voiceprint recognition. It can also perform voice recognition and specific positioning of the sound source. Among them, the voice recognition and sound source location positioning functions can be completed independently by the user identification device 103, or they can be requested to be completed by the server. The image and video data processing function is mainly used to process and analyze the video stream data collected by the shooting device 102 to determine the behavior of the object in the image. The image and video data processing function can be completed independently by the user identification device 103, or it can be requested to be completed by the server 104. The data monitoring and control is mainly used to send control instructions to the shooting device to wake up the shooting device for image and video data processing and control the shooting angle of the shooting device. The monitoring and control function can also be used to send control instructions to the smart speaker to trigger the smart speaker.

[0060] The user identification device 103 can send the monitoring data collected by the intelligent monitoring device (such as the video stream data collected by the camera) to the server, so that the server can send it to the mobile terminal 106 used by the user associated with the monitored object through the mobile communication network. The user identification device 103 can also provide a query function. For example, the user can send a query request to the user identification device 103 through the application program (APP) on the terminal (such as a mobile phone) used by the user, and the user identification device 103 can respond to the request and send the audio and video of the target monitored object requested to be queried to the terminal, so that the user can view the status of the target monitored object through the APP.

[0061] The server 104 may be an independently deployed server, a distributed server, or a server cluster. The server 104 may use cloud technology to provide powerful processing capabilities.

[0062] Based on the above architecture, in an actual application scenario, some smart monitoring devices with monitoring functions in a family (residence) can be connected to the user identification device 103 and connected to the server.

[0063] After the user identification device 103 determines the user's identity, the determined user can control the smart home devices in the home through voice information, such as turning on the air conditioner in the bedroom, querying information, etc., and the voice information to be recognized input by the user can be collected through smart devices such as smart speakers and smart TVs. The smart device that collects the voice information to be recognized can directly perform intent recognition on the voice information to be recognized locally, such as performing a user intent template matching method, or can send the voice information to be recognized, or the text information of the voice information to be recognized to the user identification device 103. The user identification device 103 performs intent recognition on the text information of the voice information to be recognized. After determining the target intent corresponding to the voice information to be recognized, some subsequent processing is performed to achieve control of smart home devices according to the user's expectations.

[0064] The embodiment of the present invention provides a user identity recognition method, which is applied to the user recognition device 103, such as Figure 2 As shown, the following steps are included:

[0065] Step S201, receiving a video stream sent by a shooting device, and performing human body recognition on images in the video stream;

[0066] The home scene is photographed by the photographing device 102, and the user identification device 103 receives the video stream obtained by the photographing, and performs human body recognition on the image in the video stream;

[0067] In some embodiments, an image in a video stream is input into a deep learning model YOLO to obtain an image with human body regions marked. The deep learning model YOLO takes the image with human body regions marked as input and performs network model training with the goal of outputting the human body regions marked in the image.

[0068] Among them, the deep learning model YOLO converts the target detection problem into a regression problem (Regreesion problem), and directly predicts the boudning box regression and related category information. The deep learning model YOLO is a single network that can be trained end-to-end. It does not require a separate search for Region Proposals or a separate Classifier. Therefore, its detection speed is very fast and the effect of detecting human bodies is very good. It can detect human body areas that are blocked, poorly lit, and only contain part of the body in the image.

[0069] In this embodiment, in order to reduce the computational complexity of real-time human body detection, human body detection is not performed in every frame. Instead, the images in the video stream are input into the deep learning model YOLO every n frames to obtain images that identify the human body area. In this embodiment, n is taken as 5. Technical personnel in this field can determine the number of interval frames for detection based on the computational complexity of the equipment, and there is no limitation here.

[0070] Step S202, when it is determined that a human body is recognized, human body feature data is extracted and compared with the human body feature data in the registration database, and when it is determined that a corresponding registered user is recognized, a response corresponding to the registered user is initiated;

[0071] The user identification device 103 obtains the image of the identified human body area detected in step S201, extracts the partial image of the human body area in the image, and inputs the partial image of the human body area into the human body feature extraction network, wherein the human body feature extraction network is used to determine the key point information of each skeleton of the human body in the human body area. The human body feature extraction network uses the image frames sent within the historical time period and the human body key points in the image frames as training samples, uses the image frame data in the training samples as input features, compares the output human body key points with the annotation results of the key point positions, adjusts the parameters of the human body key point detection model according to the comparison results, and obtains the human body key point detection model. In this embodiment, the training samples include images with human body key points in a certain frame of the video or images without human body key points in a certain frame of the video. The specific model training process can adopt the existing method, which will not be further described here.

[0072] Among them, in the human feature extraction network, there are at least two cascaded convolutional neural networks. The specific number of the convolutional neural networks is determined by the complexity of the target of calculating the key points of the human body and the computing power of the current device, and no excessive restrictions are made here.

[0073] First, the key point information of each key point is used to determine the positioning key points from the image that identifies the human body area, and the skeleton information of the human body is determined based on the key points of each point. For example, the OpenPose algorithm can be used to estimate human body motion posture using each key point. The OpenPose network uses the VGG-19 model to extract features from the image and compares them with the human body features in the registration database to obtain the similarity of the human body. The above-mentioned algorithm for extracting human body features is not limited to the algorithm provided in the above-mentioned embodiment. Any algorithm that can be used to extract human body features can be used in the above-mentioned embodiment, and there is no limitation here.

[0074] In some embodiments, the key point information can be obtained using the skeleton point acquisition model of the commonly used MSCOCO key point data set. The model can identify 18 key skeleton points of the human body, including: 0 (nose), 1 (neck), 2 (right shoulder), 3 (right elbow), 4 (right wrist), 5 (left shoulder), 6 (left elbow), 7 (left wrist), 8 (abdomen), 9 (right hip), 10 (right knee), 11 (right ankle), 12 (left hip), 13 (left knee), 14 (left ankle), 15 (right eye), 16 (left eye), 17 (right ear), 18 (left ear).

[0075] In some embodiments, extracting human feature data and comparing it with human feature data in a registration database to determine that a corresponding registered user has been identified includes:

[0076] The human body feature data is extracted and compared with the human body feature data in the registration database to determine the human body similarity. If the human body similarity is greater than the human body threshold, it is determined that a third registered user corresponding to the human body feature data is identified. When it is determined that the corresponding registered user is identified, a response corresponding to the registered user is detected.

[0077] In some embodiments, in order to ensure a high recognition accuracy in a smart home environment, the human threshold is adjusted to a larger value. Although this increases the number of cases where a registered user is not recognized, it can be subsequently optimized based on the update of the human feature data in the registration database, thereby ensuring the accuracy of recognition based on the human feature data.

[0078] In some embodiments, when it is determined that a corresponding registered user is identified, the human body feature data is stored in a registration database. Specifically, when a corresponding registered user is identified in a certain frame of the video stream of the shooting device, the human body feature data collected when the user appears in the shooting device is stored in the registration database to improve the accuracy of human body recognition.

[0079] Step S203, when it is determined that the corresponding registered user is not identified, the human body in the image is tracked according to the received video stream, and other identity recognition information corresponding to the human body is collected;

[0080] When the human body similarity between the human body feature data in the registration database and the human body feature data extracted from a certain frame of the video stream is lower than the human body threshold, the user identification device tracks the human body in the image identified by the model and collects other identity identification information corresponding to the human body;

[0081] Collecting the other identity recognition information corresponding to the human body includes at least one of the following steps:

[0082] Performing face recognition corresponding to the human body on the image in the video stream, and extracting face data features when it is determined that a face is recognized;

[0083] An audio stream sent by an audio acquisition device is received, and voiceprint recognition is performed on speech frames in the audio stream to obtain voiceprint data features.

[0084] In some embodiments, when other identity recognition information is facial features, facial recognition corresponding to the human body is performed on the image in the video stream. Specifically, when it is determined that a face is recognized, facial feature data is extracted and compared with the facial feature data in the registration database to determine the facial similarity. If the facial similarity is greater than the face threshold, it is determined that the first registered user corresponding to the facial feature data is recognized.

[0085] Among them, the face threshold is determined by collecting a large amount of home scene data in advance, detecting faces on them, and adjusting the face threshold from small to large until there is no misidentification of the user identity determined by the above face threshold, that is, the detected user identity is a certain correct result, and the face threshold is determined as the face threshold to be used.

[0086] In other embodiments, since the computational complexity of extracting facial features in the human body area is relatively small, a method of performing feature extraction on each frame can be adopted. If no face is detected, the next frame is detected. If a face is detected, facial features are extracted and the extracted facial feature data is compared with the registration database to determine the user's identity.

[0087] In some embodiments, when the other identity recognition information is a voiceprint feature, an audio stream sent by an audio acquisition device is received, and when it is determined that the audio stream is received, the voice frames in the audio stream are extracted and compared with the voiceprint feature data in the registration database to determine the voiceprint similarity. If the voiceprint similarity is greater than the voiceprint threshold, it is determined that the second registered user corresponding to the voiceprint feature data is identified.

[0088] Among them, the voiceprint threshold is determined by collecting a large amount of home scene data in advance, detecting the human voice, and adjusting the voiceprint threshold from small to large until there is no misidentification of the user identity determined by the above voiceprint threshold, that is, the detected user identity is a certain correct result, and the voiceprint threshold is determined as the voiceprint threshold used.

[0089] In some embodiments, tracking a human body in the image according to the received video stream and collecting other identity recognition information corresponding to the human body includes:

[0090] Specifically, it is necessary to determine the angle of the human body relative to the facing direction of the camera device in the image captured by the camera device, and determine the angle of the human body relative to the facing direction of the intelligent monitoring device when collecting other identity information. When the angle of the tracked human body relative to the camera device is consistent with the angle of other identity recognition information, the step of using other identity recognition information to determine the user's identity can be performed.

[0091] Firstly, determining a first angle of the tracked human body relative to the camera device;

[0092] If face recognition is performed on the image in the video stream, a second angle of the face in the image relative to the camera is determined, and when the second angle is consistent with the first angle, facial feature data of the face is extracted;

[0093] Since the human body in the human body area is detected, the facial feature data of the human face can also be extracted from the human body after determining the first angle of the human body relative to the shooting device;

[0094] When receiving an audio stream sent by an audio collection device, determine a third angle of the sound direction corresponding to the audio stream relative to the collection direction of the audio collection device, and when the third angle is consistent with the first angle, extract the voice frame of the audio stream and perform voiceprint recognition.

[0095] The sound source is located using the microphone array in the audio acquisition device. If the angle determined by the sound direction corresponding to the audio stream is consistent with the angle of the tracked human body relative to the shooting device, the voice frame of the audio stream is extracted and voiceprint recognition is performed.

[0096] In some embodiments, determining a first angle of the tracked human body relative to the camera includes:

[0097] calculate

[0098] Among them, α is the horizontal field of view angle of the shooting device, w is the horizontal number of pixels in the image, and x is the horizontal coordinate value of the human body in the image.

[0099] For example, when the horizontal direction of the camera is 0°, the angle β between the human body and the camera is:

[0100]

[0101] In some embodiments, when other identity recognition information is a fingerprint feature, after the tracked human body passes the fingerprint verification on a smart device in a smart home environment, the user identity is determined using the identity verified by the fingerprint.

[0102] The above-mentioned other identity recognition information is not limited to the above-mentioned category characteristics. Specific identity identification identifiers such as RFID or Bluetooth iBeacon indoor positioning modules carried by users that can identify personal identity can be used as identity recognition information. More accurate identity recognition information is used to determine unidentified human feature data. There is no limitation here and personnel in this field can freely set it.

[0103] Step S204: When it is determined that a corresponding registered user is identified based on the other identity recognition information, the human body feature data is stored in a registration database.

[0104] After determining the user identity information according to the other identity recognition information, the human body feature data extracted from the human body area image captured by the shooting device is stored under the human body feature data in the registration database under the corresponding registered user.

[0105] In some embodiments, if it is determined that the first registered user is identified and the second registered user is identified, and the first registered user is inconsistent with the second registered user, the registered user corresponding to the higher similarity between the face similarity and the voiceprint similarity is determined to be the identified first registered user.

[0106] In some embodiments, before using the above-mentioned face recognition and voiceprint recognition, the face recognition method and the voiceprint recognition method are first tested respectively, and the method with higher recognition accuracy among face recognition and voiceprint recognition is determined by using a large amount of real data in the collection scenarios. When the registered user is identified using the above-mentioned face recognition and voiceprint recognition, the method with higher recognition accuracy is selected to be returned as the user identity information.

[0107] This embodiment provides a complete flow chart of a complete user identification method, such as Figure 3 As shown, the following steps are included:

[0108] Step S301, registering the body, face, and voiceprint feature information of each user in the smart home environment in a registration database;

[0109] Step S302, receiving a video stream sent by a shooting device, and performing human body recognition on images in the video stream;

[0110] Step S303, when it is determined that a human body is recognized, human body feature data is extracted and compared with the human body feature data in the registration database. If the comparison is greater than the human body threshold, step S304 is executed, otherwise step S305 is executed;

[0111] Step S304, determining that a corresponding registered user is identified, and initiating a response corresponding to detection of the registered user;

[0112] Step S305, if it is determined that the corresponding registered user is not identified, the unidentified human image is tracked, and if it is determined, step S306 is executed, and if it is determined that the audio stream is received, step S307 is executed;

[0113] Step S306, extracting facial feature data, comparing it with facial feature data in a registration database, determining facial similarity, and if the facial similarity is greater than a facial threshold, and if it is determined that a second angle of the face in the image relative to the camera is consistent with a first angle of the tracked human body relative to the camera, executing step S308;

[0114] Step S307, extracting the speech frames in the audio stream, comparing them with the voiceprint feature data in the registration database, and determining the voiceprint similarity. If the voiceprint similarity is greater than the voiceprint threshold, it is determined that the second registered user corresponding to the voiceprint feature data is identified, and if the third angle of the collection direction of the audio collection device is consistent with the first angle of the tracked human body relative to the camera, step S308 is executed;

[0115] Step S308, determining a registered user corresponding to a higher similarity between the face similarity and the voiceprint similarity;

[0116] Step S309, storing the human body feature data in a registration database corresponding to the registered user.

[0117] The method provided by the present application can solve the problem that the captured human image is difficult to match with the initially recorded human image due to the complex light in the room, the non-fixed installation position of the shooting device, and the changeable position and posture of the person, and the problem that it is difficult to accurately judge the identity information of the household personnel in the complex daily household life with only the initially recorded human body. When the initially recorded human feature data cannot identify the human body, the human body is tracked. Since the recognition rate of the face and voiceprint is very high, the user's identity can be determined, and the unrecognizable human feature data is stored in the registration database. By continuously updating the human feature data in the registration database, since the added human features are images that cannot be recognized by human recognition, various situations of the human body can be covered, and the accuracy of human recognition can be increased. Through the dynamic update of the registration database, the effect of human recognition can be optimized to make it more and more accurate. In addition, since the installation positions of various shooting devices in the home scene are quite different, it is difficult to see the faces of family members in most cases, and the voiceprint is not easy to extract. Therefore, after optimizing human recognition, the recognition accuracy can be effectively improved. After a period of time, accurate identity confirmation can be performed even through human recognition alone, solving the problem of difficulty in human recognition in various scenes.

[0118] Figure 4 A schematic diagram of the structure of an electronic device provided in some embodiments of the present application, the electronic device may be the user identification device 103 provided in the above embodiment, or may be another type of electronic device, the electronic device may have relatively large differences due to different configurations or performances, the electronic device includes: a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404;

[0119] The memory 403 stores a computer program. When the program is executed by the processor 401, the processor 401 completes the steps of the electronic device executing the corresponding functions in the above method.

[0120] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0121] The communication interface 402 is used for communication between the electronic device and other devices.

[0122] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0123] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0124] When the processor is used to execute the computer program stored in the memory, the following is achieved:

[0125] Receiving a video stream sent by a camera, and performing human body recognition on images in the video stream;

[0126] When it is determined that a human body is recognized, human body feature data is extracted and compared with the human body feature data in the registration database. When it is determined that a corresponding registered user is recognized, a response corresponding to the detection of the registered user is initiated;

[0127] When it is determined that the corresponding registered user is not identified, tracking the human body in the image according to the received video stream, and collecting other identity recognition information corresponding to the human body;

[0128] When it is determined that the corresponding registered user is identified based on the other identity recognition information, the human body feature data is stored in a registration database.

[0129] In some embodiments, the processor is used to collect other identity recognition information corresponding to the human body, including at least one of the following steps:

[0130] Performing face recognition corresponding to the human body on the image in the video stream, and extracting face data features when it is determined that a face is recognized;

[0131] An audio stream sent by an audio acquisition device is received, and voiceprint recognition is performed on speech frames in the audio stream to obtain voiceprint data features.

[0132] In some embodiments, the processor is configured to determine, based on the other identity identification information, that a corresponding registered user is identified, comprising at least one of the following steps:

[0133] When it is determined that a face is recognized, facial feature data is extracted, and compared with the facial feature data in the registration database to determine the facial similarity. If the facial similarity is greater than the face threshold, it is determined that the first registered user corresponding to the facial feature data is recognized;

[0134] When it is determined that an audio stream is received, voice frames in the audio stream are extracted and compared with the voiceprint feature data in the registration database to determine the voiceprint similarity. If the voiceprint similarity is greater than the voiceprint threshold, it is determined that a second registered user corresponding to the voiceprint feature data is identified.

[0135] In some embodiments, the processor is further configured to:

[0136] If it is determined that the first registered user is identified and the second registered user is identified, and the first registered user is inconsistent with the second registered user, the registered user corresponding to the higher similarity between the face similarity and the voiceprint similarity is determined to be the identified first registered user.

[0137] In some embodiments, the processor is used to track the human body in the image according to the received video stream and collect other identity recognition information corresponding to the human body, including:

[0138] Determining a first angle of the tracked human body relative to the camera device;

[0139] If face recognition is performed on the image in the video stream, a second angle of the face in the image relative to the camera is determined, and when the second angle is consistent with the second angle, facial feature data of the face is extracted;

[0140] When receiving an audio stream sent by an audio collection device, determine a third angle of the sound direction corresponding to the audio stream relative to the collection direction of the audio collection device, and when the third angle is consistent with the first angle, extract the voice frame of the audio stream and perform voiceprint recognition.

[0141] In some embodiments, the processor is configured to determine a first angle of the tracked human body relative to the camera, including:

[0142] calculate

[0143] Among them, α is the horizontal field of view angle of the shooting device, w is the horizontal number of pixels in the image, and x is the horizontal coordinate value of the human body in the image.

[0144] In some embodiments, the processor is configured to perform human body recognition on an image in a video stream, including:

[0145] Every n frames of images, the images in the video stream are input into the deep learning model YOLO to obtain images that identify the human body area. The deep learning model YOLO takes the images that identify the human body area as input and trains the network model with the goal of outputting images that identify the human body area. The n is a positive integer.

[0146] In some embodiments, the processor is used to extract human feature data, compare it with the human feature data in the registration database, and determine that the corresponding registered user is identified, including:

[0147] The human body feature data is extracted and compared with the human body feature data in the registration database to determine the human body similarity. If the human body similarity is greater than the human body threshold, it is determined that a third registered user corresponding to the human body feature data is identified.

[0148] Based on the above embodiments, the present application provides a computer-readable storage medium, in which a computer program executable by an electronic device is stored, and the computer-executable instructions are used to enable a computer to execute a user identity recognition method.

[0149] The above-mentioned computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor in the electronic device, including but not limited to magnetic storage such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc., optical storage such as CD, DVD, BD, HVD, etc., and semiconductor storage such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD), etc.

[0150] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0151] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0152] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0153] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0154] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A user identification method, characterized in that: The method comprises: Receiving a video stream sent by a camera, and performing human body recognition on images in the video stream; When it is determined that a human body is recognized, human body feature data is extracted and compared with the human body feature data in the registration database. When it is determined that a corresponding registered user is recognized, a response corresponding to the registered user is initiated; wherein the human body feature data is information on key points of human bones in the human body area; When it is determined that the corresponding registered user is not identified, the human body in the image is tracked according to the received video stream, and other identity recognition information corresponding to the human body is collected; wherein the other identity recognition information includes facial data features or voiceprint data features; When it is determined that the corresponding registered user is identified based on the other identity recognition information, the human body feature data is stored in a registration database.

2. The method according to claim 1, characterized in that Collecting the other identity recognition information corresponding to the human body includes at least one of the following steps: Performing face recognition corresponding to the human body on the image in the video stream, and extracting face data features when it is determined that a face is recognized; An audio stream sent by an audio acquisition device is received, and voiceprint recognition is performed on speech frames in the audio stream to obtain voiceprint data features.

3. The method according to claim 2, characterized in that Determining that a corresponding registered user is identified according to the other identity identification information includes at least one of the following steps: When it is determined that a face is recognized, facial feature data is extracted, and compared with the facial feature data in the registration database to determine the facial similarity. If the facial similarity is greater than the face threshold, it is determined that the first registered user corresponding to the facial feature data is recognized; When it is determined that an audio stream is received, voice frames in the audio stream are extracted and compared with the voiceprint feature data in the registration database to determine the voiceprint similarity. If the voiceprint similarity is greater than the voiceprint threshold, it is determined that a second registered user corresponding to the voiceprint feature data is identified.

4. The method according to claim 3, characterized in that The method further comprises: If it is determined that the first registered user is identified and the second registered user is identified, and the first registered user is inconsistent with the second registered user, the registered user corresponding to the higher similarity between the face similarity and the voiceprint similarity is determined to be the identified first registered user.

5. The method according to claim 2, characterized in that: Tracking a human body in the image according to the received video stream and collecting other identity recognition information corresponding to the human body, including: Determining a first angle of the tracked human body relative to the camera device; If face recognition is performed on the image in the video stream, a second angle of the face in the image relative to the camera is determined, and when the second angle is consistent with the second angle, facial feature data of the face is extracted; When receiving an audio stream sent by an audio collection device, determine a third angle of the sound direction corresponding to the audio stream relative to the collection direction of the audio collection device, and when the third angle is consistent with the first angle, extract the voice frame of the audio stream and perform voiceprint recognition.

6. The method according to claim 5, characterized in that Determining a first angle of a tracked human body relative to a camera device includes: calculate Among them, α is the horizontal field of view angle of the shooting device, w is the horizontal number of pixels in the image, and x is the horizontal coordinate value of the human body in the image.

7. The method according to claim 1, characterized in that Perform human body recognition on images in video streams, including: Every n frames of images, the images in the video stream are input into the deep learning model YOLO to obtain images with marked human body areas. The deep learning model YOLO takes images with marked human body areas as input and trains the network model with the goal of outputting the human body areas marked in the images, where n is a positive integer.

8. The method according to claim 1, characterized in that Extracting human feature data and comparing it with the human feature data in the registration database to identify the corresponding registered user, including: Extract human feature data, compare it with human feature data in the registration database, and determine human similarity; If the human body similarity is greater than the human body threshold, it is determined that a third registered user corresponding to the human body feature data is identified.

9. An electronic device, characterized in that: The electronic device comprises at least a processor and a memory, and the processor is used to implement the steps of the user identification method as claimed in any one of claims 1 to 8 when executing the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the instructions are executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 8.

Images