Multi-mode-based family member relationship autonomous learning method and device

Abstract

The invention relates to a multi-mode-based family member relationship autonomous learning method and device. The method comprises the following steps: acquiring image data and voice data; the image data comprises a face image, and the voice data comprises voiceprint information and semantic information; acquiring attribute information corresponding to the face image based on the image data, predicting a family identity corresponding to the face image based on the face image in combination with the voiceprint information and the attribute information, and generating a family personnel knowledge graph; identity keywords related to family identities are monitored on the basis of continuously updated semantic information, and a family personnel knowledge graph is iteratively corrected on the basis of the identity keywords and continuously updated face images, attribute information and voiceprint information, so that autonomous learning of family personnel relationships is realized. According to the invention, through combination of various modes such as human faces and voiceprints and combination of a natural language processing technology, completely autonomous learning of family personnel relationships is realized, and naturalness and friendliness of man-machine interaction are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multimodal-based autonomous learning method and device for family relationships. Background Art

[0002] As robot technology matures, safer and smarter home robots are gradually entering people's lives, usually used for cleaning, caring for the elderly and children, providing entertainment or helping to manage household affairs. Home robots provide services to families. Since the service objects are relatively fixed, it is necessary to mark the identities of family members to provide targeted services.

[0003] Existing technologies rely on users to collect and register face, voiceprint and other information of each family member in advance, and require manual annotation of the relationships between family members.

[0004] The above method requires all family members to cooperate with the collection work. The operation is relatively complicated and not user-friendly, and cannot meet the needs of all families. Summary of the invention

[0005] Based on this, it is necessary to provide a multimodal-based autonomous learning method and device for family relationships in response to the above technical problems.

[0006] In a first aspect, the present application provides a multimodal-based autonomous learning method for family relationships.

[0007] The method includes:

[0008] Acquire image data and voice data; wherein the image data includes a face image, and the voice data includes voiceprint information and semantic information;

[0009] Obtain attribute information corresponding to the face image based on the image data, predict the family identity corresponding to the face image based on the face image combined with the voiceprint information and attribute information; obtain a family member knowledge graph based on the predicted multiple family identities;

[0010] Based on the continuously updated semantic information, identity keywords related to family identity are monitored. Based on the identity keywords, as well as the continuously updated facial images, attribute information and voiceprint information, the knowledge graph of family members is iteratively corrected to achieve autonomous learning of family member relationships.

[0011] In one embodiment, predicting the family identity corresponding to the face image based on the face image combined with voiceprint information and attribute information includes:

[0012] Matching the facial image and the voiceprint information according to the timestamp when the facial image was captured and the timestamp when the voiceprint information was captured, as well as the facial orientation corresponding to the facial image and the sound source orientation corresponding to the voiceprint information;

[0013] Based on the attribute information corresponding to the facial image and the voiceprint information successfully matched with the facial image, the family identity corresponding to the facial image is predicted.

[0014] In one embodiment, the method further comprises:

[0015] Cluster the historical face images, and use the historical face image with the highest confidence among the historical face images in the same cluster as the reference image;

[0016] Cluster the historical voiceprint information, and use the historical voiceprint information with the highest confidence in the same cluster as the reference voiceprint;

[0017] Based on the face image combined with voiceprint information and attribute information, the family identity corresponding to the face image is predicted, including:

[0018] Matching the face image with the reference image, and matching the voiceprint information with the reference voiceprint;

[0019] Based on the reference image that has the highest matching degree with the facial image, combined with the reference voiceprint that has the highest matching degree with the voiceprint information and the attribute information corresponding to the reference image, the family identity corresponding to the facial image is predicted.

[0020] In one embodiment, the method further comprises:

[0021] If the number of images in the historical face images of the same cluster is less than the threshold, it is marked as a guest;

[0022] The threshold is the weighted value of the number of corresponding images in each historical face image cluster.

[0023] In one embodiment, based on identity keywords, and continuously updated facial images, attribute information, and voiceprint information, the family member knowledge graph is iteratively modified, including:

[0024] On the same timeline, taking the capture moment of the identity keyword as the base point, searching for facial images, attribute information and voiceprint information acquired within a preset time window before and after the base point, and matching the searched facial images, attribute information and voiceprint information with the family identity represented by the identity keyword respectively;

[0025] If the match is successful, the searched facial image is associated with the family identity represented by the identity keyword, and the family member knowledge graph is modified.

[0026] In one embodiment, the family member knowledge graph includes: the relationship between multiple family identities, the facial images corresponding to each family identity, the voiceprint information corresponding to each facial image, and the attribute information corresponding to each facial image.

[0027] In a second aspect, the present application also provides a multi-modal family relationship autonomous learning device. The device includes:

[0028] A collection module, used to obtain image data and voice data; wherein the image data includes face images, and the voice data includes voiceprint information and semantic information;

[0029] A preliminary prediction module is used to obtain attribute information corresponding to a face image based on image data, predict the family identity corresponding to the face image based on the face image combined with voiceprint information and attribute information; and obtain a family member knowledge graph based on the predicted multiple family identities;

[0030] The iterative update module is used to monitor identity keywords related to family identity based on continuously updated semantic information, and iteratively correct the family member knowledge graph based on identity keywords, as well as continuously updated facial images, attribute information and voiceprint information, to achieve autonomous learning of family member relationships.

[0031] In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the multimodal-based family personnel relationship autonomous learning method when executing the computer program.

[0032] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above-mentioned multimodal-based family personnel relationship autonomous learning method when executed by a processor.

[0033] In a fifth aspect, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned multimodal-based family personnel relationship autonomous learning method are implemented.

[0034] The above-mentioned multimodal-based autonomous learning method and device for family personnel relationships obtains image data and voice data; wherein the image data includes a face image, and the voice data includes voiceprint information and semantic information; based on the image data, the attribute information corresponding to the face image is obtained, and the family identity corresponding to the face image is predicted based on the face image combined with the voiceprint information and the attribute information; the family personnel knowledge graph is obtained based on the predicted multiple family identities; based on the continuously updated semantic information, the identity keywords related to the family identity are monitored, and based on the identity keywords, and the continuously updated face images, attribute information and voiceprint information, the family personnel knowledge graph is iteratively corrected to realize the autonomous learning of family personnel relationships. The present invention realizes completely autonomous learning of family personnel relationships by combining multiple modalities such as face and voiceprint, and combining natural language processing technology, without the need for manual annotation by the user, and improves the naturalness and friendliness of human-computer interaction. At the same time, with continuous data collection and iteration, the recognition results of family personnel relationships will become more and more accurate, which is also in line with the characteristics of robots as a new life, that is, continuous self-learning and growth. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of a process of a multimodal autonomous learning method for family member relationships in one embodiment;

[0036] Figure 2 Schematic diagram of a process of a face clustering algorithm in one embodiment;

[0037] Figure 3 Schematic diagram of the process of a voiceprint clustering algorithm in one embodiment;

[0038] Figure 4 A schematic diagram of the structure flow of a robot interaction system in one embodiment;

[0039] Figure 5 It is a structural block diagram of a multi-modal family personnel relationship autonomous learning device in one embodiment;

[0040] Figure 6 It is a structural block diagram of a multimodal family relationship autonomous learning device in another embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] The present application embodiment provides a multi-modal family relationship autonomous learning method, such as Figure 1 As shown, the following steps are included:

[0043] Step 102, acquiring image data and voice data; wherein the image data includes a face image, and the voice data includes voiceprint information and semantic information.

[0044] Among them, image data can be collected and obtained by using visible light cameras, infrared cameras and other devices configured on the robot, and voice data can be collected and obtained by using microphones and other devices configured on the robot.

[0045] Face images can be obtained by identifying faces and cropping them based on image data through image recognition technology.

[0046] Voiceprint information can be obtained by processing voice technology using voiceprint recognition technology.

[0047] Semantic information can be converted into text using automatic speech recognition technology, and then the semantics in the text can be analyzed using natural language processing technology to extract key information and intent from the text.

[0048] Step 104, obtaining attribute information corresponding to the facial image based on the image data, predicting the family identity corresponding to the facial image based on the facial image combined with the voiceprint information and the attribute information; and obtaining a family member knowledge graph based on the predicted multiple family identities.

[0049] In addition to facial data, image data also includes body data, such as height, gender, body shape, etc. The approximate age range corresponding to the face can also be identified through facial data. The above age, height, gender, body shape and other information are the attribute information corresponding to the facial data.

[0050] Voiceprint information is a unique biological feature of an individual, carrying personal biological information such as gender, age, height and weight. It has the characteristics of universality, specificity, relative stability and difficulty in replicability.

[0051] This embodiment matches the facial image of the same person with the voiceprint information and attribute information, and predicts the data basis of the family identity of the person at the same time. Since family members are relatively fixed, the traditional family composition can be used as a reference to predict the family identity of people who frequently appear in the family. For example, if the robot recognizes that the permanent population in a family includes a woman of about 30 years old, a boy of about 5 years old, and a man of about 60 years old, then based on the woman as the starting point, the robot will predict that the boy and the woman are in a mother-son relationship, and the man and the woman are in a father-daughter relationship, thereby determining the family identity of the three people.

[0052] After predicting and confirming the family identities, a family member knowledge graph is generated based on the relationships between family identities to achieve more systematic management of family member information.

[0053] Step 106, based on the continuously updated semantic information, monitor identity keywords related to family identity, and iteratively correct the family member knowledge graph based on the identity keywords, as well as the continuously updated facial images, attribute information and voiceprint information, to achieve autonomous learning of family member relationships.

[0054] When working at home, the robot will continuously collect image data and voice data, and continuously obtain facial images, attribute information, voiceprint information and semantic information.

[0055] Based on the semantic information, the robot can monitor identity keywords, including names and titles (mom, dad, etc.). In this embodiment, the responses to names and titles can also be monitored as identity keywords, so as to more quickly and accurately modify the knowledge graph of family members. By introducing natural language processing technology, semantic analysis and context understanding are used to improve the accuracy of identity confirmation.

[0056] In addition to identity keywords, the family knowledge graph can be modified. Updated facial images, attribute information, and voiceprint information will also affect the modification results and modification frequency. For example, when a new facial image appears, it is necessary to return to step 102 to perform the prediction step of the family identity corresponding to the facial image, and then update the family knowledge graph.

[0057] Identity keywords and facial images, attribute information, and voiceprint information can also be used as joint information to determine whether to update the identification of a family identity. For example, if the identity keyword "mom" appears during a conversation, and the facial images, attribute information, and voiceprint information collected from both parties in the conversation are determined to be male, then the identity keyword will not trigger the modification of the family member knowledge graph.

[0058] This embodiment combines multiple modalities such as face and voiceprint, as well as natural language processing technology, to achieve completely autonomous learning of family relationships, without the need for manual annotation by the user, thus improving the naturalness and friendliness of human-computer interaction. At the same time, with continuous data collection and iteration, the recognition results of family relationships will become more and more accurate, which is also in line with the characteristics of robots as a new life, that is, continuous self-learning and growth.

[0059] In one embodiment, step 104 predicts the family identity corresponding to the facial image based on the facial image combined with the voiceprint information and the attribute information, including: matching the facial image and the voiceprint information according to the timestamp when the facial image was captured and the timestamp when the voiceprint information was captured, as well as the facial orientation corresponding to the facial image and the sound source orientation corresponding to the voiceprint information; predicting the family identity corresponding to the facial image according to the attribute information corresponding to the facial image and the voiceprint information successfully matched with the facial image.

[0060] The facial image and voiceprint information whose capture timestamps differ by no more than a preset time period (e.g., 5 seconds) and whose direction deviations do not exceed a preset angle (e.g., 45 degrees) are matched. If the match is successful, it is judged that the pair of data is likely to belong to the same person. Based on the facial image and voiceprint information in the pair of data, as well as the attribute information corresponding to the facial image, the family identity corresponding to the facial image is predicted. The prediction method has been described above and will not be repeated here.

[0061] In this embodiment, the matching of facial images and voiceprint information can be further optimized by adding gender, age and other features contained in the facial image to match the gender, age and other features contained in the voiceprint information. If the above matching is successful and the timestamp distance and direction deviation meet the matching criteria, the facial image and voiceprint information are judged to be successfully matched.

[0062] In one embodiment, step 104 also includes: clustering historical facial images, and taking the historical facial image with the highest confidence among the historical facial images in the same cluster as a reference image; clustering historical voiceprint information, and taking the historical voiceprint information with the highest confidence among the historical voiceprint information in the same cluster as a reference voiceprint; matching the facial image with the reference image, and matching the voiceprint information with the reference voiceprint; based on the reference image with the highest match to the facial image, combined with the reference voiceprint with the highest match to the voiceprint information and the attribute information corresponding to the reference image, predicting the family identity corresponding to the facial image.

[0063] In this embodiment, face recognition based on deep learning is combined with a clustering algorithm to identify and cluster face images. Specifically, yolo-v8 face in deepface is used as a face detector, VGGFace is used as a face feature extractor, and a hierarchical clustering algorithm is used for clustering. Under autonomous learning conditions, the accuracy of face recognition clustering can reach 94%.

[0064] Deepface-based face recognition and clustering pipeline Figure 2As shown. First, face detection is performed on the image data, which refers to the technology of automatically identifying and locating faces in images or videos. Face detection is usually the first step in a facial recognition system, which is used to find the facial area in the image and provide a basis for subsequent facial recognition or analysis steps. Then, face alignment is performed based on the cropped facial image, that is, by adjusting the position, angle or posture of the facial image to align it to a standard or preset position. Face alignment can help improve the accuracy and consistency of facial recognition. Next, the face embedding model is used to convert the facial image into a low-dimensional face embedding vector, which can represent facial features in a smaller space, thereby facilitating tasks such as facial recognition, comparison or retrieval. Finally, clustering operations are performed based on the facial embedding vector, and face images in the same cluster are judged to be the same person. The threshold for face recognition is based on the VGGFace document: if the cosine distance is less than 0.64, it is considered to be the same person. This parameter can be directly used here.

[0065] Voiceprint clustering uses a solution based on feature extraction and unsupervised learning of speech signals. The pipeline is as follows: Figure 3 As shown. First, the input speech data is preprocessed, including denoising and normalization of the speech signal amplitude. The next step is to extract voiceprint features from the processed speech data. This embodiment uses MFCC (Mel-Frequency Cepstral Coefficients), which can efficiently extract the spectral information of the speech signal. Since the spectrum is represented in a high-dimensional space, direct clustering will increase the computational complexity, so it is necessary to reduce the dimension first. Here, PCA (Principal Component Analysis) is used, and clustering is performed after dimensionality reduction. Since this embodiment provides a completely autonomous learning system, the voiceprint clustering algorithm in this system uses DBSCAN (Density-Based Spatial Clustering of Applications with Noise), which is suitable for an unknown number of categories. Regarding the thresholds selected for the voiceprint clustering algorithm in this embodiment: eps is 0.3, and min_sample is 3.

[0066] In this embodiment, both the face image and the voiceprint image refer to the greedy matching algorithm, that is, the face image is matched with the historical face image with the highest confidence, and the voiceprint information is matched with the historical voiceprint information with the highest confidence.

[0067] After clustering, the historical face images in the same cluster will be labeled with a unique face ID. The matching process between the face image and the reference image is to determine whether the similarity between the face image and a reference image reaches a threshold. If so, it is considered a successful match, and the face image is labeled with the face ID corresponding to the reference image. Similarly, voiceprint information also has a unique voiceprint ID, and the labeling method is similar to that of face images. Labeling face ID and voiceprint ID facilitates subsequent management.

[0068] In addition, facial images and voiceprint information are constantly collected, updated and expanded. After each clustering, when data with higher confidence (mainly facial images and voiceprint information) appears, the reference image and reference voiceprint will be updated, and then the historical prediction results will be corrected, so as to achieve better recognition results over time.

[0069] In one embodiment, step 104 further includes: if the number of images in the same cluster of historical face images is less than a threshold, marking them as guests; wherein the threshold is a weighted value of the number of images corresponding to each historical face image in the same cluster.

[0070] The weighted value of the number of images under each face category is used as a threshold. New categories below this threshold are temporarily classified as guests and then discarded.

[0071] In addition, low-quality images and low-quality speech will also be discarded during the clustering and iteration process, such as images with little face exposure or unclear speech.

[0072] In one embodiment, in step 106, based on the identity keywords, and the continuously updated facial images, attribute information and voiceprint information, the family member knowledge graph is iteratively corrected, including: on the same timeline, taking the capture moment of the identity keywords as the base point, searching for facial images, attribute information and voiceprint information acquired within a preset time window before and after the base point, and matching the searched facial images, attribute information and voiceprint information with the family identities represented by the identity keywords respectively; if the match is successful, the searched facial images are associated with the family identities represented by the identity keywords, and the family member knowledge graph is corrected.

[0073] On the same timeline, the time of facial image capture and the time of voice data collection need to match, so as to achieve the purpose of associating identity keywords with faces. When associating voice with faces, the capture time of the identity keywords in the voice will be used as the base point on the timeline, and the face will be searched before and after. If the face is not found, no match will be performed; if the face is searched and the confidence is higher than the threshold, the identity keyword will be associated with the face. Here, the face confidence is obtained based on the yolo-v8 face model, and the threshold is selected as 0.75, which can be used to measure the quality of the face image. In this embodiment, the preset time window is 1s before the base point and 5s after the base point.

[0074] In this embodiment, when associating identity keywords with faces, the voiceprint information and attribute information must still be combined. The gender, age and other features contained in the voiceprint information and attribute information corresponding to the face image must be matched with the gender and age restrictions implied by the family identity represented by the identity keywords. Only when the match is complete can the association between the identity keywords and faces be completed, effectively reducing misjudgments.

[0075] In one embodiment, the family member knowledge graph includes: the relationship between multiple family identities, the facial images corresponding to each family identity, the voiceprint information corresponding to each facial image, and the attribute information corresponding to each facial image.

[0076] With the continuous updating of face clustering and voiceprint clustering, they will definitely tend to stabilize under a certain periodic threshold. At this time, an accurate face ID and the corresponding face image can be obtained, and the voiceprint information and attribute information associated with this face ID can be obtained. In this embodiment, the face image corresponding to the family identity can select the reference image with the highest confidence, and the voiceprint information can select the reference voiceprint with the highest confidence. Similarly, the attribute information can also be gradually refined during the continuous updating of image data, and the attribute information with stable data is stored in the family member knowledge graph. With the iterative correction of family identity, the results will become more and more stable and accurate, completing the autonomous learning of family member situation.

[0077] like Figure 4 As shown, this is the interaction flow chart of the various functional modules of the robot.

[0078] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0079] Based on the same inventive concept, the embodiment of the present application also provides a multimodal family personnel relationship autonomous learning device for implementing the multimodal family personnel relationship autonomous learning method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the multimodal family personnel relationship autonomous learning device provided below can refer to the limitations of the multimodal family personnel relationship autonomous learning method above, and will not be repeated here.

[0080] In one embodiment, Figure 5 As shown, a multi-modal family personnel relationship autonomous learning device is provided, including: a collection module 502, a preliminary prediction module 504 and an iterative update module 506, wherein:

[0081] The acquisition module 502 is used to acquire image data and voice data; wherein the image data includes face images, and the voice data includes voiceprint information and semantic information;

[0082] A preliminary prediction module 504 is used to obtain attribute information corresponding to the face image based on the image data, predict the family identity corresponding to the face image based on the face image combined with the voiceprint information and the attribute information; and obtain a family member knowledge graph based on the predicted multiple family identities;

[0083] The iterative update module 506 is used to monitor identity keywords related to family identity based on the continuously updated semantic information, and iteratively correct the family member knowledge graph based on the identity keywords, as well as the continuously updated facial images, attribute information and voiceprint information, to achieve autonomous learning of family member relationships.

[0084] In one embodiment, the preliminary prediction module 504 is also used to match the facial image and the voiceprint information based on the timestamp when the facial image was captured and the timestamp when the voiceprint information was captured, as well as the facial orientation corresponding to the facial image and the sound source orientation corresponding to the voiceprint information; and predict the family identity corresponding to the facial image based on the attribute information corresponding to the facial image and the voiceprint information that successfully matches the facial image.

[0085] In one embodiment, the preliminary prediction module 504 is also used to cluster historical facial images, and use the historical facial image with the highest confidence among the historical facial images in the same cluster as the reference image; cluster historical voiceprint information, and use the historical voiceprint information with the highest confidence among the historical voiceprint information in the same cluster as the reference voiceprint; match the facial image with the reference image, and match the voiceprint information with the reference voiceprint; based on the reference image with the highest match to the facial image, combined with the reference voiceprint with the highest match to the voiceprint information and the attribute information corresponding to the reference image, predict the family identity corresponding to the facial image.

[0086] In one embodiment, the preliminary prediction module 504 is further configured to mark the image as a guest if the number of images in the historical face images of the same cluster is less than a threshold value; wherein the threshold value is a weighted value of the number of images corresponding to each historical face image of the same cluster.

[0087] In one embodiment, the iterative update module 506 is also used to search for facial images, attribute information and voiceprint information obtained within a preset time window before and after the base point on the same timeline, taking the capture moment of the identity keyword as the base point, and matching the searched facial images, attribute information and voiceprint information with the family identity represented by the identity keyword respectively; if the match is successful, the searched facial image is associated with the family identity represented by the identity keyword, and the family member knowledge graph is corrected.

[0088] In one embodiment, the family member knowledge graph includes: the relationship between multiple family identities, the facial images corresponding to each family identity, the voiceprint information corresponding to each facial image, and the attribute information corresponding to each facial image.

[0089] like Figure 6 Shown is a structural block diagram of another multimodal-based autonomous learning device for family relationships.

[0090] Each module in the above-mentioned multimodal family personnel relationship autonomous learning device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0091] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in all the above method embodiments when executing the computer program.

[0092] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in all the above method embodiments are implemented.

[0093] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in all the above method embodiments when executed by a processor.

[0094] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0095] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processors involved in each embodiment provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc., but are not limited thereto.

[0096] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0097] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multimodal family relationship autonomous learning method, characterized in that: The method comprises: Acquire image data and voice data; wherein the image data includes a face image, and the voice data includes voiceprint information and semantic information; Acquire attribute information corresponding to the face image based on the image data, predict the family identity corresponding to the face image based on the face image combined with the voiceprint information and the attribute information; acquire a family member knowledge graph based on the predicted multiple family identities; Based on the continuously updated semantic information, identity keywords related to the family identity are monitored. Based on the identity keywords, as well as the continuously updated facial images, the attribute information and the voiceprint information, the family member knowledge graph is iteratively corrected to achieve autonomous learning of family member relationships.

2. The method according to claim 1, characterized in that The predicting the family identity corresponding to the face image based on the face image combined with the voiceprint information and the attribute information includes: Matching the facial image with the voiceprint information according to the timestamp when the facial image was captured and the timestamp when the voiceprint information was captured, as well as the facial orientation corresponding to the facial image and the sound source orientation corresponding to the voiceprint information; The family identity corresponding to the facial image is predicted based on the attribute information corresponding to the facial image and the voiceprint information successfully matched with the facial image.

3. The method according to claim 1, characterized in that The method further comprises: Clustering the historical face images, and taking the historical face image with the highest confidence among the historical face images in the same cluster as a reference image; Clustering the historical voiceprint information, and taking the historical voiceprint information with the highest confidence among the historical voiceprint information in the same cluster as the reference voiceprint; The predicting the family identity corresponding to the face image based on the face image combined with the voiceprint information and the attribute information includes: Matching the facial image with the reference image, and matching the voiceprint information with the reference voiceprint; Based on the reference image that has the highest matching degree with the facial image, combined with the reference voiceprint that has the highest matching degree with the voiceprint information and the attribute information corresponding to the reference image, the family identity corresponding to the facial image is predicted.

4. The method according to claim 3, characterized in that The method further comprises: If the number of images in the historical face images of the same cluster is less than the threshold, it is marked as a guest; The threshold is a weighted value of the number of images corresponding to the historical face images in each cluster.

5. The method according to claim 1, characterized in that The iteratively correcting the family member knowledge graph based on the identity keyword, and the continuously updated face image, the attribute information, and the voiceprint information, comprises: On the same timeline, taking the capture moment of the identity keyword as a base point, searching for the facial image, the attribute information and the voiceprint information acquired within a preset time window before and after the base point, and matching the searched facial image, the attribute information and the voiceprint information with the family identity represented by the identity keyword respectively; If the match is successful, the searched facial image is associated with the family identity represented by the identity keyword, and the family member knowledge graph is modified.

6. The method according to any one of claims 1 to 5, characterized in that: The family member knowledge graph includes: the relationship between multiple family identities, the facial images corresponding to each family identity, the voiceprint information corresponding to each facial image, and the attribute information corresponding to each facial image.

7. A multi-modal family relationship autonomous learning device, characterized in that: The device comprises: A collection module, used to obtain image data and voice data; wherein the image data includes a face image, and the voice data includes voiceprint information and semantic information; A preliminary prediction module, for obtaining attribute information corresponding to the face image based on the image data, predicting the family identity corresponding to the face image based on the face image combined with the voiceprint information and the attribute information; and obtaining a family member knowledge graph based on the predicted multiple family identities; An iterative update module is used to monitor identity keywords related to the family identity based on the continuously updated semantic information, and iteratively correct the family member knowledge graph based on the identity keywords, as well as the continuously updated facial images, attribute information and voiceprint information, so as to realize autonomous learning of family member relationships.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images