Information identification method and device, electronic equipment and storage medium

Abstract

The invention relates to an information identification method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring sound information, and identifying whether the sound information comprises the sound of a first object; acquiring image information in response to the fact that the sound information does not include the sound of the first object; identifying whether the image information comprises a predetermined object; wherein the predetermined object at least comprises the first object. Through the method, the power consumption of the electronic equipment can be reduced, and the information identification accuracy is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of identification technology, and in particular to an information identification method and device, an electronic device, and a storage medium. Background Art

[0002] At present, anti-theft systems based on face recognition often need to be turned on for 24 hours, and the image recognition system turned on for a long time will generate high power consumption. In some situations where it is impossible to plug in and can only rely on batteries, the image recognition system turned on for a long time often has higher requirements for battery life. Summary of the invention

[0003] The present invention provides an information identification method and device, an electronic device, and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an information identification method, comprising:

[0005] Acquiring sound information, and identifying whether the sound information is a sound including a first object;

[0006] acquiring image information in response to the sound information not including the sound of the first object;

[0007] Identify whether the image information includes a predetermined object; wherein the predetermined object at least includes the first object.

[0008] In some embodiments, the method further comprises:

[0009] Identifying a sound type of the sound information;

[0010] In response to the sound information not including the sound of the first object, acquiring the image information comprises:

[0011] In response to the sound information not including the sound of the first object and the sound type of the sound information including a predetermined type, the image information is acquired.

[0012] In some embodiments, the identifying whether the sound information includes the sound of the first object includes:

[0013] determining whether the sound type includes a sound of the type to which the first object belongs;

[0014] In response to the sound type including the type to which the first object belongs, identifying whether the sound information includes the sound of the first object;

[0015] In response to the sound type not including the type to which the first object belongs, it is determined that the sound information does not include the sound of the first object.

[0016] In some embodiments, the identifying whether the sound information includes the sound of the first object includes:

[0017] Extracting voiceprint features of the sound information;

[0018] Comparing the voiceprint feature of the sound information with the pre-stored voiceprint feature of the first object to obtain a first comparison result;

[0019] It is determined whether the sound information includes the sound of the first object according to the first comparison result.

[0020] In some embodiments, the identifying whether the image information includes a predetermined object includes:

[0021] Detecting objects in the image information and extracting features of each object in the image information;

[0022] Comparing the features of each object in the image information with the pre-stored image features of the predetermined object to obtain a second comparison result;

[0023] Determine whether the image information includes the predetermined object according to the second comparison result.

[0024] In some embodiments, the method further comprises:

[0025] If the image information does not include the predetermined object, an alarm message is output.

[0026] In some embodiments, in response to the sound information not including the sound of the first object, acquiring the image information includes:

[0027] In response to the sound information not including the sound of the first object, acquiring at least two pieces of image information;

[0028] The identifying whether the image information includes a predetermined object includes:

[0029] Perform image recognition on each piece of image information to obtain a recognition result of each piece of image information;

[0030] Based on the recognition result of each piece of image information, it is determined whether the at least two pieces of image information include the predetermined object.

[0031] In some embodiments, the identifying whether the sound information includes the sound of the first object includes:

[0032] Performing speech separation on the sound information;

[0033] For each separated speech, identifying whether the speech includes the sound of the first object; in response to any separated speech including the sound of the first object, determining that the sound information includes the sound of the first object;

[0034] In response to the fact that none of the separated voices include the voice of the first object, it is determined that the voice information does not include the voice of the first object. According to a second aspect of an embodiment of the present disclosure, there is provided an information identification device, comprising:

[0035] a first recognition module configured to obtain sound information and identify whether the sound information includes a sound of a first object;

[0036] An acquisition module configured to acquire image information in response to the sound information not including the sound of the first object;

[0037] The second recognition module is configured to recognize whether the image information includes a predetermined object; wherein the predetermined object at least includes the first object.

[0038] In some embodiments, the apparatus further comprises:

[0039] a third recognition module, configured to recognize a sound type of the sound information;

[0040] The acquisition module is further configured to acquire the image information in response to the sound information not including the sound of the first object and the sound type of the sound information including a predetermined type.

[0041] In some embodiments, the first recognition module is further configured to determine whether the sound type includes the sound of the type to which the first object belongs; in response to the sound type including the type to which the first object belongs, identify whether the sound information includes the sound of the first object; in response to the sound type not including the type to which the first object belongs, determine that the sound information does not include the sound of the first object.

[0042] In some embodiments, the first recognition module is further configured to extract voiceprint features of the sound information; compare the voiceprint features of the sound information with pre-stored voiceprint features of the first object to obtain a first comparison result; and determine whether the sound information includes the sound of the first object based on the first comparison result.

[0043] In some embodiments, the second recognition module is further configured to detect objects in the image information and extract features of each object in the image information; compare the features of each object in the image information with pre-stored image features of the predetermined object to obtain a second comparison result; and determine whether the image information includes the predetermined object based on the second comparison result.

[0044] In some embodiments, the apparatus further comprises:

[0045] The output module is configured to output alarm information if the image information does not include the predetermined object.

[0046] In some embodiments, the acquisition module is further configured to acquire at least two pieces of image information in response to the sound information not including the sound of the first object;

[0047] The second recognition module is further configured to perform image recognition on each piece of image information to obtain a recognition result for each piece of image information; and based on the recognition result for each piece of image information, determine whether the at least two pieces of image information include the predetermined object.

[0048] In some embodiments, the first recognition module is further configured to perform voice separation on the sound information; for each separated voice, identify whether the voice includes the voice of the first object; in response to any separated voice including the voice of the first object, determine that the sound information includes the voice of the first object; in response to none of the separated voices including the voice of the first object, determine that the sound information does not include the voice of the first object. According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, comprising:

[0049] processor;

[0050] a memory for storing processor-executable instructions;

[0051] Wherein, the processor is configured to execute the information identification method as described in the first aspect above.

[0052] According to a fourth aspect of an embodiment of the present disclosure, there is provided a storage medium, including:

[0053] When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the information identification method as described in the first aspect above.

[0054] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0055] In an embodiment of the present disclosure, the electronic device first determines whether the sound of the first object is included based on the acquired sound information, and only acquires the image information to identify whether the predetermined object is included if the sound information does not include the sound of the first object. On the one hand, considering that the memory occupied by sound information is usually less than that occupied by image information, and the power consumption of recognition based on image information is also greater than that of recognition based on sound information, the embodiment of the present disclosure prioritizes identifying whether the sound information includes the sound of the first object, and only acquires the image information for information recognition if the sound information does not include the sound of the first object, which can reduce the power consumption of the electronic device and thus reduce the battery life requirements of the electronic device. On the other hand, considering that there are certain errors in single sound information recognition or image information recognition, recognition based on two different types of information is performed on the basis of reducing the power consumption of the electronic device, and the method of supporting image recognition of objects other than the first object based on image information can improve the accuracy of information recognition and has high intelligence.

[0056] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0058] Figure 1 The present invention is a flow chart of an information recognition method based on face recognition in related technology.

[0059] Figure 2 It is a flow chart of an information identification method shown in an embodiment of the present disclosure.

[0060] Figure 3 It is a principle block diagram of the information identification method in the embodiment of the present disclosure.

[0061] Figure 4 The schematic diagram of the model training in the embodiment of the present disclosure.

[0062] Figure 5 It corresponds to Figure 3 An example flow chart of the information identification method.

[0063] Figure 6 The figure is a diagram of an information identification device according to an exemplary embodiment.

[0064] Figure 7 It is a block diagram of an electronic device shown in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0065] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0066] Figure 1 is a flow chart of an information recognition method based on face recognition in the related technology, such as Figure 1 As shown, the following steps are included:

[0067] S01, input image;

[0068] S02, image (face) recognition algorithm;

[0069] S03. Determine whether a strange face appears.

[0070] In the above process, it is necessary to continuously obtain facial images and input them into the facial recognition algorithm for image recognition. Since the image information occupies a large amount of memory and the image recognition algorithm consumes a lot of power, the power consumption of the recognition device will be relatively high. In some situations where it is impossible to plug in and can only rely on batteries, there are high requirements for the battery life of the recognition device. At the same time, simply using image information will also have the problem of low accuracy due to the influence of the shooting environment.

[0071] In this regard, the present disclosure provides an information identification method. Figure 2 is a flow chart of an information identification method shown in an embodiment of the present disclosure, such as Figure 2 As shown, the information identification method includes the following steps:

[0072] S11, acquiring sound information, and determining whether the sound information includes the sound of the first object;

[0073] S12, in response to the sound information not including the sound of the first object, acquiring image information;

[0074] S13. Identify whether the image information includes a predetermined object; wherein the predetermined object at least includes the first object.

[0075] The information identification method provided by the embodiments of the present disclosure may be implemented independently by a terminal or a server, or may be implemented jointly by a terminal and a server. For example, the terminal may independently perform the information identification method described below, or the terminal may send collected sound information to the server, and the server may execute the information identification method based on the received sound information, and may receive image information collected by the terminal when the sound information does not include the sound of the first object to further determine whether the image information includes a predetermined object.

[0076] In the embodiments of the present disclosure, the terminal may be a smart door lock, a camera, a mobile phone, a smart speaker or other device, and the sound information and image information may be collected by the same terminal or by different terminals; in addition, the server may be an independent physical server or a server cluster or distributed system composed of multiple physical servers, and the terminal and the server may be connected to each other by wired or wireless communication, which is not limited in the embodiments of the present disclosure. Taking the application of the information identification method of the embodiments of the present disclosure to an electronic device as an example, the electronic device includes the aforementioned terminal or server.

[0077] The information recognition method of the disclosed embodiment can be applied to access control systems, for example, access control can be opened after successful information recognition; it can also be applied to monitoring systems in homes or public areas, for example, a warning can be issued after information recognition fails to inform of malicious intrusion, etc.

[0078] In step S11, the electronic device obtains sound information, which may be sound of a predetermined frame length or sound within a preset time length. The sound information may be human voice or sounds of other animals, door opening, glass breaking, or car horns, etc., which may be collected.

[0079] The electronic device determines whether the sound information includes the sound of the first object. In the embodiment of the present disclosure, the sound information can be directly subjected to sound recognition to determine whether it includes the sound of the first object, or it can be determined first that the sound type of the sound information includes the type to which the first object belongs, and then determine whether the sound information includes the sound of the first object. The first object can be a specific person or a specific object other than a person. In addition, in the embodiment of the present disclosure, the sound information can be pre-processed, such as denoising and volume normalization, and then the electronic device can identify whether it includes the sound of the first object based on the pre-processed sound information.

[0080] It should be noted that, in the embodiments of the present disclosure, the sound information may include the sound of one or more objects. In some embodiments, the step of identifying whether the sound information includes the sound of the first object includes:

[0081] Performing speech separation on the sound information;

[0082] For each separated speech, identifying whether the speech includes the sound of the first object;

[0083] In response to any of the separated voices including the voice of the first object, determining that the voice information includes the voice of the first object;

[0084] In response to the fact that none of the separated voices includes the voice of the first object, it is determined that the voice information does not include the voice of the first object.

[0085] In the embodiments of the present disclosure, the sound information may be first subjected to speech separation, for example, traditional methods such as independent component analysis (ICA) and sparse principal component analysis (Sparse PCA) may be used for speech separation, and deep learning methods may also be used, such as using a TasNet network to perform speech separation on time domain sound signals, or converting the sound signals to the frequency domain or other transform domains, and using deep clustering and other networks to perform speech separation based on the more obvious signal features of the speech signals in the transform domain.

[0086] After performing speech separation, the electronic device will identify whether the separated speech includes the sound of the first object. As mentioned above, sound recognition can be directly performed on each separated speech to determine whether each speech includes the sound of the first object. Alternatively, for each speech, it can be determined that the sound type includes the type to which the first object belongs and then whether the speech includes the sound of the first object.

[0087] It should be noted that in the embodiment of the present disclosure, if the sound signal only includes the voice of one object, then after voice separation, the result of the voice separation may be only one voice. At this time, the electronic device can determine whether the one voice (which may be the original sound information) includes the sound of the first object.

[0088] In the embodiment of the present disclosure, if any speech includes the sound of the first object, it is determined that the sound information includes the sound of the first object. Otherwise, if none of the separated speech includes the sound of the first object, it is determined that the sound information does not include the sound of the first object.

[0089] It can be understood that by determining whether each voice recognition includes the voice of the first object after voice separation, the accuracy of voice recognition of the first object can be improved, thereby increasing the necessity of acquiring image information for image recognition. In addition, the solution of the embodiment of the present disclosure can flexibly cope with various scenarios (for example, supporting scenarios in which one person or multiple people speak), and has good intelligence.

[0090] In step S12, the electronic device obtains image information when the sound information does not include the sound of the first object, for example, when it is determined that the sound information does not include the sound of the first object, a camera application is started to collect image information, or image information is obtained from other devices. Exemplarily, the electronic device may obtain one or more pieces of image information.

[0091] In step S13, the electronic device identifies whether the image information includes a predetermined object. The predetermined object may be the first object, and may also include an object other than the first object.

[0092] In the embodiment of the present disclosure, if the sound information includes the sound of the first object, the information recognition is successful, and there is no need to obtain image information at this time; if the sound information does not include the sound of the first object, the electronic device further obtains image information, and when it is recognized that the image information includes the predetermined object, the information recognition is successful.

[0093] Exemplarily, taking house security as an example, the first object is the owner of the house A. When the sound information does not include the voice of the owner of the house A, the electronic device obtains image information again, and determines whether the image information includes the owner of the house A, or includes both the owner of the house A and the owner of the house B based on the face recognition algorithm. If the image information includes the owner of the house A, or also includes the owner of the house B, the door lock can be opened. In this embodiment, it is only necessary to register the voice of the owner of the house A to determine whether to obtain the image information. When the image information is obtained, the image recognition of the first object and the preset object other than the first object is performed to perform the unlocking operation. Since there are usually multiple owners of the house returning home together, the image information supports the method of image recognition of objects other than the first object, which can also improve the accuracy of information recognition (for example, it can be set to judge that the information recognition is successful only when the owner of the house A and the owner of the house B are both successfully recognized), and the intelligence is relatively high.

[0094] For another example, the first object includes a pet dog raised by the house owner A. If the sound information is not the sound of the dog raised by the owner, the electronic device obtains image information and determines whether the image information includes the dog of the house owner A and the house owner A based on the image recognition algorithm. In this embodiment, by recognizing sound information other than people to determine whether to obtain image information, and in the case of obtaining image information, image recognition of various types of objects is supported to perform, for example, unlocking operations or information security-related processing, which can improve the accuracy of information recognition and has strong flexibility.

[0095] It should be noted that in the disclosed embodiment, when the electronic device recognizes based on image information, if there are multiple images, and the multiple images are taken at the same angle, the electronic device can fuse the multiple images, and then recognize based on the fused image to determine whether the predetermined object is included. In addition, in the disclosed embodiment, the electronic device can also recognize each image to determine whether the predetermined object is included.

[0096] It can be understood that in the embodiment of the present disclosure, the electronic device first determines whether the sound of the first object is included based on the acquired sound information, and only when the sound information includes the sound of the first object does it acquire the image information to identify whether it includes the predetermined object. On the one hand, considering that the memory occupied by sound information is usually less than the memory occupied by image information, and the power consumption of recognition based on image information is also greater than the power consumption of recognition based on sound information, the embodiment of the present disclosure prioritizes identifying whether the sound information includes the sound of the first object, and only when the sound information does not include the sound of the first object does it acquire the image information for information recognition, which can reduce the power consumption of the electronic device and thus reduce the battery life requirements of the electronic device. On the other hand, considering that there are certain errors in single sound information recognition or image information recognition, on the basis of reducing the power consumption of the electronic device, recognition is based on two different types of information, and the method of supporting image recognition of objects other than the first object based on image information can improve the accuracy of information recognition and has high intelligence.

[0097] In some embodiments, the method further comprises:

[0098] Identifying a sound type of the sound information;

[0099] In response to the sound information not including the sound of the first object, acquiring the image information includes:

[0100] In response to the sound information not including the sound of the first object and the sound type of the sound information including a predetermined type, the image information is acquired.

[0101] In the disclosed embodiments, when the electronic device identifies the sound type of the sound information, it may extract the features of the sound information and then use the feature judgment rules corresponding to the sound type to identify the sound type of the current sound information, or it may identify the sound type in the sound information based on a trained neural network model. When the sound information does not include the sound of the first object, the electronic device also obtains image information based on the sound type of the sound information. Among them, the predetermined type of sound can be characterized as a predetermined event. For example, the sound information is not the voice of the owner of the house, but the sound type is a predetermined type such as a human voice, a door opening sound, and a glass breaking sound, indicating that someone may have illegally intruded. At this time, image information can also be further obtained to identify whether the predetermined object is included. For another example, the sound information is not the sound of a pet at home, but the sound type is the sound of other animals, indicating that there may be an animal invasion, etc.

[0102] It can be understood that in the embodiments of the present disclosure, when the sound information does not include the sound of the first object, further determining whether the sound type includes a predetermined type, that is, determining whether a predetermined event has occurred based on the sound information and then acquiring image information, can reduce meaningless image information acquisition and recognition. For example, if the sound information does not include the voice of the owner of the house, but the sound type is wind or rain, etc., then there may be no one passing by or intruding, and there is no need to acquire image information for information recognition. Therefore, in this way, the necessity of image information acquisition and image information recognition can be enhanced, and the power consumption of electronic devices can be reduced.

[0103] In some embodiments, the identifying whether the sound information includes the sound of the first object includes:

[0104] determining whether the sound type includes a sound of the type to which the first object belongs;

[0105] In response to the sound type including the type to which the first object belongs, identifying whether the sound information includes the sound of the first object;

[0106] In response to the sound type not including the type to which the first object belongs, it is determined that the sound information does not include the sound of the first object.

[0107] In the disclosed embodiment, the electronic device first determines that the sound type of the sound information includes the type to which the first object belongs, and then determines whether the sound information is the sound of the first object. When identifying whether the sound information includes the sound of the first object, it can be based on the voiceprint feature to use rule judgment or based on the principle of trained model recognition to identify whether the sound information includes the sound of the first object. The voiceprint feature may include auditory features and acoustic features. Auditory features refer to sound features that can be identified and described by the human ear, such as speaking with a lot of breath and full of breath; acoustic features refer to a set of acoustic description parameters extracted from the sound information, such as the frequency and intensity of the sound.

[0108] It should be noted that if the sound information is directly identified as to whether it includes the sound of the first object, since the sound recognition of the first object focuses more on specific voiceprint features, for example, more detailed features need to be extracted to identify whether the sound information includes the sound of a specific object, it is easily affected by sounds of the same type or different types, and higher accuracy requirements are placed on the recognition algorithm; and when classifying sound types, the differences between different types of sounds will be greater, and thus the accuracy of identifying the sound type first will be higher. On this basis, the embodiment of the present disclosure identifies whether the sound information includes the sound of the first object when the sound type of the current sound information is consistent with the sound of the type to which the first object belongs, so that more detailed feature extraction can be performed for this type, thereby improving the accuracy of identifying whether the sound of the first object is included.

[0109] In addition, in the embodiment of the present disclosure, if the electronic device determines that the sound type does not include the type to which the first object belongs, it directly determines that the sound information does not include the sound of the first object. In this way, the efficiency of identifying whether the sound information includes the sound of the first object can also be improved.

[0110] In some embodiments, the identifying the sound type of the sound information includes:

[0111] The sound type of the sound information is identified based on a preset sound classification model; wherein the sound classification model is obtained by training a neural network model based on sample sounds of the type to which the first object belongs and sample sounds of the predetermined type.

[0112] In the disclosed embodiment, the electronic device can identify the sound type of sound information based on a trained neural network model. Since the model training method can extract richer features based on more samples, it can improve the accuracy of sound type recognition compared to the rule-based judgment method.

[0113] It should be noted that in the embodiments of the present disclosure, if a model is used to identify the sound type, and if a model is used to identify the sound of the first object when the sound type is consistent with the sound of the type to which the first object belongs, the sound type identification and the sound identification of the first object can share a recognition model, which can be obtained by training a multi-task model based on the sample sound type and the sound of the sample object. Since the multi-task model is characterized by being able to share common features between tasks and taking into account the individual features of each task, in the embodiments of the present disclosure, for example, the most basic acoustic features of sound information can be shared, and the sound type identification and the sound identification of the first object have their own individual features, so the learning efficiency and prediction accuracy of the sound type identification and / or the sound identification of the first object can be improved by sharing the task model.

[0114] In some embodiments, the identifying whether the sound information includes the sound of the first object includes:

[0115] Extracting voiceprint features of the sound information;

[0116] Comparing the voiceprint feature of the sound information with the pre-stored voiceprint feature of the first object to obtain a first comparison result;

[0117] It is determined whether the sound information includes the sound of the first object according to the first comparison result.

[0118] In the disclosed embodiment, recognition can be performed based on a trained voiceprint recognition model, for example, voiceprint features in the sound information are extracted based on the voiceprint recognition model, and then the voiceprint features of the voiceprint information are compared with the voiceprint features of the first object to obtain a first comparison result of the similarity value. When the first comparison result is greater than a first comparison threshold, it is determined that the sound information includes the sound of the first object, and when the first comparison result is less than or equal to the first comparison threshold, it is determined that the sound information is not the sound of the first object. The voiceprint recognition model is trained based on a sound sample including the first object, and the voiceprint features of the first object stored in the electronic device can be based on the features extracted and saved by the voiceprint recognition model.

[0119] Of course, the embodiments of the present disclosure are not limited to extracting voiceprint features based on a voiceprint recognition model driven by sample data. For example, traditional methods may be used to analyze sound information based on precise mathematical rules.

[0120] It can be understood that in the embodiments of the present disclosure, the scheme of identifying whether the sound information includes the sound of the first object by extracting voiceprint features for comparison is simple and effective; and because the voiceprint features of the first object are stored in the electronic device in advance, there is no need for frequent extraction before each comparison, thereby improving the efficiency of voice recognition of the first object.

[0121] In some embodiments, the identifying whether the image information includes a predetermined object includes:

[0122] Detecting objects in the image information and extracting features of each object in the image information;

[0123] Comparing the features of each object in the image information with the pre-stored image features of the predetermined object to obtain a second comparison result;

[0124] Determine whether the image information includes the predetermined object according to the second comparison result.

[0125] In the disclosed embodiment, the electronic device may also perform recognition based on a trained object recognition model, for example, detecting an object in image information based on the object recognition model, and then comparing the extracted object features with the features of a predetermined object to obtain a second comparison result of a similarity value, and when the second comparison result is greater than a second comparison threshold, determining that the image information includes the predetermined object, and when the second comparison result is less than or equal to the second comparison threshold, determining that the image information does not include the predetermined object. The object recognition model is trained based on an image sample including the predetermined object, and the features of the predetermined object stored in the electronic device may be based on the features extracted and saved by the object recognition model.

[0126] In the embodiments of the present disclosure, the object recognition model can be a face recognition model or a recognition model of other objects. Taking the face recognition model as an example, the embodiments of the present disclosure are not limited to extracting face features based on the face recognition model driven by face sample data. For example, it can also be a traditional method to analyze face images based on precise mathematical rules to extract features. In addition, in the embodiments of the present disclosure, the detection of objects in image information can be completed based on an independent detection algorithm. For example, after an independent face detection algorithm detects a face, the face features are extracted based on the face recognition model.

[0127] It should be noted that if the predetermined object includes the first object and objects other than the first object, the electronic device may compare the features of each object extracted from the image information with the image features of the first object and the image features of objects other than the first object, that is, compare the individual objects, and determine whether the image information includes the predetermined object based on the second comparison results of each object. For example, when one or more objects of the predetermined object are identified in the image information, it can be determined that the image information includes the predetermined object.

[0128] It can be understood that in the embodiments of the present disclosure, the scheme of identifying whether the image information includes a predetermined object by extracting the features of the object for comparison is simple and effective; and since the image features of the predetermined object are stored in the electronic device in advance, there is no need for frequent extraction before each comparison, thereby improving the efficiency of identifying the predetermined object in the image information.

[0129] In some embodiments, the method further comprises:

[0130] If the image information does not include the predetermined object, an alarm message is output.

[0131] In the embodiments of the present disclosure, if the image information does not include the predetermined object, for example, in the application of home monitoring, it means that a stranger may enter, and an alarm message can be output to remind the family. Exemplarily, in the embodiments of the present disclosure, the electronic device can output the alarm message in the form of sound and / or light, or send a text message to the user terminal, make a phone call, etc. to alarm.

[0132] It is understandable that the electronic device outputs an alarm message when the image information does not include the predetermined object, so that the user can immediately know that the current information recognition has failed, which is highly intelligent.

[0133] In some embodiments, in response to the sound information not including the sound of the first object, acquiring the image information includes:

[0134] In response to the sound information not including the sound of the first object, acquiring at least two pieces of image information;

[0135] The identifying whether the image information includes a predetermined object includes:

[0136] Perform image recognition on each piece of image information to obtain a recognition result of each piece of image information;

[0137] Based on the recognition result of each piece of image information, it is determined whether the at least two pieces of image information include the predetermined object.

[0138] As mentioned above, the electronic device can obtain multiple image information. In the embodiment of the present disclosure, the electronic device can send image acquisition requests to multiple devices with shooting functions to obtain multiple image information. The electronic device can also obtain multiple image information based on multiple cameras included in itself, for example, the multiple image information is images taken from different angles; in addition, the electronic device can also obtain multiple image information collected within a preset time period through a device including a camera (which can be the electronic device itself). The embodiment of the present disclosure does not limit the method of obtaining multiple image information (at least two image information).

[0139] In the embodiment of the present disclosure, the electronic device can perform image recognition on each piece of image information, and determine whether at least two pieces of image information include a predetermined object based on the recognition result of each piece of image information. For example, if the predetermined object is recognized in image information that meets a preset number, it is determined that at least two pieces of image information include the predetermined object.

[0140] It can be understood that in the embodiment of the present disclosure, by obtaining multiple image information when the sound information does not include the sound of the first object, and performing image recognition on each piece of image information to determine whether it includes the predetermined object, since the image quality of the image information may be affected by the acquisition environment, recognition errors may occur when only one piece of image information is used for recognition. The embodiment of the present disclosure can improve the accuracy of recognizing the predetermined object by performing recognition through multiple image information.

[0141] Figure 3 is a principle block diagram of the information identification method in the embodiment of the present disclosure, such as Figure 3 As shown in the figure, in the part marked by L0, after the electronic device receives the audio (i.e. obtains the sound information), it is input into the sound event model marked by L1 to extract features and identify whether it is an illegal break-in-related event marked by L1a, or a human voice marked by L1b, or other events marked by L1c. It should be noted that the sound event model is the sound classification model of the embodiment of the present disclosure. Different events correspond to different sound types. The sound of glass breaking and door opening in L1a belongs to the predetermined type of sound in the embodiment of the present disclosure. Figure 3 As shown, if the sound type is identified by L1a, face recognition identified by L3 is directly performed; if it is human voice identified by L1b, voiceprint recognition identified by L2 is further performed, and face recognition identified by L3 is performed when a stranger voice identified by L2a is recognized. Otherwise, if the voiceprint recognition is determined to be the voice of the owner identified by L2b, the identity recognition is completed and the electronic device does not need to process it. After face recognition identified by L3, if it is a stranger face identified by L3a, it means that someone has invaded. At this time, an alarm can be sent as identified by L4. If the face recognition is the face of the owner identified by L3b, the electronic device does not need to process it.

[0142] Figure 3In the embodiment, when performing voiceprint recognition of the L2 mark, such as the part marked by L21, the electronic device can use the voiceprint recognition model to extract the characteristics of the voice segment (i.e., the audio of the L0 mark), i.e., the unknown voice code Usembd, and then calculate the similarity between the unknown voice code Usembd and the homeowner's voice code TSmbd, and determine it to be the homeowner when the similarity is greater than a threshold, otherwise it is a stranger; when performing face recognition of the L3 mark, such as the part marked by L31, the electronic device can use the face recognition model to extract the characteristics of the face segment (i.e., the image information in the embodiment of the present disclosure), i.e., the unknown face code UFembd, and then calculate the similarity between the homeowner's face code Tfembd and the unknown face code UFembd, and determine it to be the homeowner when the similarity is greater than a threshold, otherwise it is a stranger.

[0143] Figure 3 In the process, the voiceprint recognition model, face recognition model and sound event model involved can be obtained through model training. Figure 4 is a schematic diagram of the model training in the embodiment of the present disclosure, such as Figure 4 As shown, the model training method identified by K0 is used to train the voiceprint recognition model sr identified by K1, the sound event model sed identified by K2, and the face recognition model fr identified by K3. Different models may use different initial networks. For example, the voiceprint recognition model sr identified by K1 can be obtained by training a voice sample including a voice segment of the homeowner using a deep neural network (DNN) model; the sound event model sed identified by K2 can be obtained by training different types of sound samples such as human voice, rain sound, glass breaking sound, and rain sound using an AlexNet network model; the face recognition model fr identified by K3 can be obtained by training an image sample including a photo of the homeowner using a VGG model. Among them, after the voiceprint recognition model sr identified by K1 is trained, it can save the homeowner's voice code Tsembd (that is, the voiceprint features of the first object) to identify whether it is the homeowner's voice. After the face recognition model fr identified by K3 is trained, it can save the homeowner's facial code TFembd (for example, the image features of the first object and / or the image features of objects other than the first object) to identify whether it is the homeowner's face.

[0144] Figure 5 It corresponds to Figure 3 An example flow chart of the information identification method is shown in FIG. Figure 5 As shown, the following steps are included:

[0145] S21, audio reception;

[0146] In the embodiment of the present disclosure, audio reception means that the electronic device obtains sound information.

[0147] S22, feature calculation;

[0148] In the disclosed embodiment, the feature calculation is that the electronic device first determines the homeowner's voice code TSmbd and the homeowner's face code Tfembd.

[0149] S23, network computing;

[0150] In the disclosed embodiment, the network calculation includes the aforementioned input of the received audio into the sound event model, and after the sound event model recognizes that the current audio is human voice, the audio is then input into the voiceprint recognition model. If the voiceprint recognition model recognizes that it is a stranger's voice, image information is obtained and sent to the face recognition model. In addition, if the sound event model recognizes that the sound is an illegal break-in related event such as glass breaking and door opening, image information is also obtained and sent to the face recognition model.

[0151] S24, result judgment;

[0152] In the disclosed embodiment, the result judgment is that the electronic device extracts the features of the image information based on the face recognition model, namely, the unknown face code UFembd, and then calculates the similarity between the pre-stored homeowner face code Tfembd and the unknown face code UFembd. If the similarity is greater than a threshold, it is determined to be the homeowner, otherwise it is a stranger.

[0153] S25, give / not give an alarm;

[0154] In the embodiment of the present disclosure, the electronic device may, based on the result in step S24, give an alarm if the person is determined to be a stranger, and not give an alarm if the person is determined to be the homeowner.

[0155] It should be noted that the audio detection function can be turned on for a long time in the embodiments of the present disclosure. For example, the sound information collected for a period of time is continuously input into the sound event model. When a voiceprint or a specific sound event that does not belong to the owner of the house is identified based on the sound event model, the surveillance camera will be turned on to collect image information for face recognition.

[0156] It can be understood that the embodiment of the present disclosure prioritizes identifying whether the sound information comes from the homeowner, and only obtains image information for face recognition when the sound information does not come from the homeowner. This can reduce the power consumption of electronic devices, thereby reducing the battery life requirements of electronic devices. At the same time, on the basis of reducing the power consumption of electronic devices, the method of identifying based on two different types of information can improve the accuracy of identifying the homeowner.

[0157] Figure 6 FIG. 1 is a diagram showing an information identification device according to an exemplary embodiment. Figure 6 , the device comprises:

[0158] A first recognition module 101 is configured to obtain sound information and identify whether the sound information includes the sound of a first object;

[0159] An acquisition module 102 is configured to acquire image information in response to the sound information not including the sound of the first object;

[0160] The second recognition module 103 is configured to recognize whether the image information includes a predetermined object; wherein the predetermined object at least includes the first object.

[0161] In some embodiments, the apparatus further comprises:

[0162] A third identification module 104 is configured to identify the sound type of the sound information;

[0163] The acquisition module 102 is further configured to acquire the image information in response to the sound information not including the sound of the first object and the sound type of the sound information including a predetermined type.

[0164] In some embodiments, the first recognition module 101 is further configured to determine whether the sound type includes the sound of the type to which the first object belongs; in response to the sound type including the type to which the first object belongs, identify whether the sound information includes the sound of the first object; in response to the sound type not including the type to which the first object belongs, determine that the sound information does not include the sound of the first object.

[0165] In some embodiments, the first recognition module 101 is further configured to extract voiceprint features of the sound information; compare the voiceprint features of the sound information with pre-stored voiceprint features of the first object to obtain a first comparison result; and determine whether the sound information includes the sound of the first object based on the first comparison result.

[0166] In some embodiments, the second recognition module 103 is further configured to detect objects in the image information and extract features of each object in the image information; compare the features of each object in the image information with pre-stored image features of the predetermined object to obtain a second comparison result; and determine whether the image information includes the predetermined object based on the second comparison result.

[0167] In some embodiments, the apparatus further comprises:

[0168] The output module 105 is configured to output alarm information if the image information does not include the predetermined object.

[0169] In some embodiments, the acquisition module 102 is further configured to acquire at least two pieces of image information in response to the sound information not including the sound of the first object;

[0170] The second recognition module 103 is further configured to perform image recognition on each piece of image information to obtain a recognition result for each piece of image information; and determine whether the at least two pieces of image information include the predetermined object based on the recognition result for each piece of image information.

[0171] In some embodiments, the first recognition module 101 is further configured to perform speech separation on the sound information; for each separated speech, identify whether the speech includes the sound of the first object; in response to any separated speech including the sound of the first object, determine that the sound information includes the sound of the first object; in response to none of the separated speech including the sound of the first object, determine that the sound information does not include the sound of the first object.

[0172] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0173] Figure 7 8 is a block diagram of an electronic device according to an exemplary embodiment. The electronic device can be identified as a device 800, which can be a smart speaker, a personal computer, etc.

[0174] Reference Figure 7 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0175] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0176] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0177] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0178] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0179] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0180] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0181] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0182] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as Wi-Fi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0183] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0184] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0185] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the aforementioned information identification method.

[0186] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims above.

[0187] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An information identification method, characterized in that: The method comprises: Acquiring sound information, and identifying whether the sound information includes the sound of the first object; acquiring image information in response to the sound information not including the sound of the first object; Identify whether the image information includes a predetermined object; wherein the predetermined object at least includes the first object.

2. The method according to claim 1, characterized in that The method further comprises: Identifying a sound type of the sound information; In response to the sound information not including the sound of the first object, acquiring the image information includes: In response to the sound information not including the sound of the first object and the sound type of the sound information including a predetermined type, the image information is acquired.

3. The method according to claim 2, characterized in that The identifying whether the sound information includes the sound of the first object includes: determining whether the sound type includes a sound of the type to which the first object belongs; In response to the sound type including the type to which the first object belongs, identifying whether the sound information includes the sound of the first object; In response to the sound type not including the type to which the first object belongs, it is determined that the sound information does not include the sound of the first object.

4. The method according to any one of claims 1 to 3, characterized in that: The identifying whether the sound information includes the sound of the first object includes: Extracting voiceprint features of the sound information; Comparing the voiceprint feature of the sound information with the pre-stored voiceprint feature of the first object to obtain a first comparison result; It is determined whether the sound information includes the sound of the first object according to the first comparison result.

5. The method according to any one of claims 1 to 3, characterized in that: The identifying whether the image information includes a predetermined object includes: Detecting objects in the image information and extracting features of each object in the image information; Comparing the features of each object in the image information with the pre-stored image features of the predetermined object to obtain a second comparison result; Determine whether the image information includes the predetermined object according to the second comparison result.

6. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: If the image information does not include the predetermined object, an alarm message is output.

7. The method according to any one of claims 1 to 3, characterized in that: In response to the sound information not including the sound of the first object, acquiring the image information includes: In response to the sound information not including the sound of the first object, acquiring at least two pieces of image information; The identifying whether the image information includes a predetermined object includes: Perform image recognition on each piece of image information to obtain a recognition result of each piece of image information; Based on the recognition result of each piece of image information, it is determined whether the at least two pieces of image information include the predetermined object.

8. The method according to any one of claims 1 to 3, characterized in that: The identifying whether the sound information includes the sound of the first object includes: Performing speech separation on the sound information; For each separated speech, identifying whether the speech includes the sound of the first object; In response to any of the separated voices including the voice of the first object, determining that the voice information includes the voice of the first object; In response to the fact that none of the separated voices includes the voice of the first object, it is determined that the voice information does not include the voice of the first object.

9. An information recognition device, characterized in that: The device comprises: a first recognition module configured to obtain sound information and identify whether the sound information includes a sound of a first object; An acquisition module configured to acquire image information in response to the sound information not including the sound of the first object; The second recognition module is configured to recognize whether the image information includes a predetermined object; wherein the predetermined object at least includes the first object.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the information identification method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the information identification method as claimed in any one of claims 1 to 8.

Images