Digital cloned body construction method, device and system based on intelligent glasses

Abstract

The invention relates to a digital duplication construction method, device and system based on intelligent glasses. The digital cloned body construction method based on the intelligent glasses comprises the following steps: S1, acquiring life data of a user at a first person view angle through the intelligent glasses; s2, performing feature extraction and recognition on the life data to obtain content data; s3, establishing a user portrait according to the content data; and S4, training an intelligent agent according to the user portrait, and taking the trained intelligent agent as a digital avatar model. According to the method, hardware advantages of the intelligent glasses are utilized, the life data of the first person perspective of the user are obtained through the intelligent glasses in an all-weather mode, and the portrait of the user in the real world is constructed according to the life data, so that the digital avatar model with a relatively high simulation degree is trained; the limitation that an existing digital duplicate construction method can only process virtual world trace information is broken through.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for constructing a digital avatar based on smart glasses. Background Art

[0002] A digital avatar is a virtual character created based on artificial intelligence and virtual reality technology. Its essence is an artificial intelligence agent that digitally replicates a specific real person based on the personal information and habits of the real person. A digital avatar can simulate the corresponding real person in the digital world and has similar appearance, voice, personality and behavioral characteristics to the real person.

[0003] Digital avatars can, to a certain extent, represent real people in the interaction of the virtual world and realize the application of various scenarios. For example, digital avatars can replace real people to participate in online or offline meetings, give speeches, and hold discussions; on social media, digital avatars can replace real people to interact, such as chatting, posting status, replying to comments, etc.; in the field of education, digital avatars can serve as virtual teaching assistants to provide online tutoring and answer questions for students. In addition, digital avatars can also be applied to many fields such as corporate marketing, public services, and e-commerce live broadcasting.

[0004] The key problem of building a digital avatar is how to make the intelligent agent understand the personal information and habits of the corresponding real person. For this key problem, existing technical solutions include: 1. Ask users to actively submit their personal information and habits. This solution requires users to actively fill in information and set parameters, which is a cumbersome process and low user enthusiasm. In addition, the character image set by the user is often different from the real character image, and the degree of realism of the constructed digital avatar is low.

[0005] 2. Collect the trace information left by the user in the virtual world (such as chat records, browsing records, and consumption records), and let the intelligent agent summarize the user's personal information and habitual preferences by learning these trace information. This solution does not require the user to actively participate, and is a commonly used digital avatar construction method in the current existing technology. However, the virtual world is only a relatively small part of the user's life. Even if the intelligent agent can obtain all the trace information left by the user in the virtual world, it can only learn one aspect of the user's image. Therefore, the digital avatar constructed by this method still has a low degree of simulation, which limits the further promotion and application of digital avatars. Summary of the invention

[0006] Based on this, the object of the present invention is to provide a method, device and system for constructing a digital avatar based on smart glasses.

[0007] The method for constructing a digital avatar based on smart glasses of the present invention includes: S1, obtaining life data of a user from a first-person perspective through smart glasses; S2, extracting and identifying features of the life data to obtain content data; S3, establishing a user portrait based on the content data; S4, training an intelligent agent based on the user portrait, and using the trained intelligent agent as a digital avatar model.

[0008] The present invention takes advantage of the hardware advantages of smart glasses, obtains the user's first-person perspective life data through smart glasses around the clock, and constructs a portrait of the user in the real world based on this life data, thereby training a digital avatar model with a high degree of simulation, breaking through the limitation of existing digital avatar construction methods that can only process virtual world trace information.

[0009] Furthermore, the life data is multimodal data, which includes: video data, audio data, time data and space data; step S2 specifically includes: S21, performing picture content recognition on the video data to obtain video content data; S22, performing audio content recognition on the audio data to obtain audio content data; S23, performing spatiotemporal association on the video content data and the audio content data according to the time data and the space data to obtain context content data; S24, making the video content data, audio content data and context content data form content data.

[0010] Furthermore, step S22 specifically includes: S221, performing audio segmentation on the audio data to separate the dialogue voice data, ambient sound data and background music data in the audio data; S222, performing content recognition on the dialogue voice data to obtain the dialogue content; S223, performing style recognition on the background music data to obtain the type of background music; S224, performing type recognition on the ambient sound data to obtain the type of ambient sound; S225, making the dialogue content, background music type and ambient sound type constitute the audio content data.

[0011] Furthermore, the video content data described in step S21 includes: scene type, and characters, objects, signs and text information in the scene.

[0012] Furthermore, the user portrait includes: preference and interest portrait, behavior pattern portrait and social and language style portrait; step S3 specifically includes: S31, counting the frequency of occurrence of each element in the content data, screening several high-frequency elements with higher frequency of occurrence, and inferring the user's interest preferences, habitual choices and / or consumption tendencies based on the high-frequency elements, thereby establishing the user's preference and interest portrait; S32, based on the content data, analyzing the user's daily routine and / or activity route, and summarizing the user's behavior patterns in various scenarios, thereby establishing a behavior pattern portrait; S33, based on the content data, analyzing the user's language habits, expression style and / or commonly used vocabulary, and analyzing the user's interaction frequency with different social objects, common topics, expression style and / or emotional characteristics, thereby establishing a social and language style portrait.

[0013] Furthermore, step S4 is specifically as follows: using the user portrait as the input Prompt of the agent, and fine-tuning the parameters of the agent according to the Prompt, thereby obtaining a digital avatar model.

[0014] Furthermore, step S5 is included after step S4: repeating steps S1-S4, continuously acquiring life data through smart glasses, updating the user portrait according to the updated life data, and comparing the difference between the imitation behavior generated by the digital avatar model and the real behavior of the user, and further fine-tuning the parameters of the digital avatar model.

[0015] Furthermore, it also includes a preprocessing step before step S2; the preprocessing step includes: data cleaning of the life data to extract valid life data fragments; time alignment matching of the valid life data fragments to obtain preprocessed data; the step S2 becomes: feature extraction and recognition of the preprocessed data to obtain content data.

[0016] Based on the same inventive concept, the present invention also provides a digital avatar construction device based on smart glasses, including: a data acquisition module, which is used to obtain the user's first-person perspective life data through smart glasses; a content recognition module, which is used to extract and recognize features of the life data to obtain content data; a portrait construction module, which is used to establish a user portrait based on the content data; a training module, which trains an intelligent agent based on the user portrait, and uses the trained intelligent agent as a digital avatar model.

[0017] Based on the same inventive concept, the present invention also provides a digital avatar construction system, including: smart glasses and a processor; the smart glasses collect life data from the user's first-person perspective; the processor executes any of the above-mentioned digital avatar construction methods based on smart glasses according to the life data obtained by the smart glasses, thereby constructing a digital avatar model of the user.

[0018] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of the modules of the digital avatar construction system of the present invention; Figure 2 A schematic diagram of a module of a device for constructing a digital avatar based on smart glasses according to the present invention; Figure 3 The figure is a flow chart of the method for constructing a digital avatar based on smart glasses of the present invention. DETAILED DESCRIPTION

[0020] It should be clear that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0021] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0022] In addition, in the description of this application, unless otherwise specified, "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0023] It should be understood that the embodiments of the present application are not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the embodiments of the present application is limited only by the appended claims.

[0024] Smart glasses are a kind of wearable device that combines the visual assistance function of traditional glasses with the interaction and computing capabilities of modern smart devices. The present invention sets data acquisition modules such as cameras, microphones, and inertial sensors on smart glasses. While the user wears the smart glasses, the smart glasses can collect a large amount of visual, auditory, and motion data from the user's first-person perspective through the data collection module they carry. The present invention enables the intelligent body to grasp the user's personal information, habitual preferences, and interpersonal relationships in real life through these first-person perspective data, thereby replicating a digital avatar model with a high degree of simulation.

[0025] See also Figure 1 , Figure 1 The module diagram of the digital twin construction system of the present invention. The digital twin construction system of the present invention comprises: smart glasses 1 and a processor 2. The smart glasses 1 are provided with data acquisition modules such as a camera, a microphone, an inertial sensor, and a position sensor. While the user wears the smart glasses 1, the smart glasses 1 collects the user's life data from the first-person perspective for a long time through the data acquisition module carried thereon. The processor 2 executes the digital twin construction method based on smart glasses of the present invention according to the life data acquired by the smart glasses 1, thereby constructing a digital twin model of the user.

[0026] In this embodiment, the processor 2 is independent of the smart glasses 1. In other embodiments, the processor 2 may also be disposed inside the smart glasses 1 as a part of the smart glasses 1. The present invention does not limit the relationship between the smart glasses 1 and the processor 2.

[0027] The life data collected by the smart glasses 1 of the present invention has the characteristics of "all-weather" and "first-person perspective". These two characteristics are relative to traditional smart terminals such as mobile phones and computers. Although traditional smart terminals such as mobile phones and computers can also be equipped with data acquisition modules such as cameras and microphones to collect users' life data, due to the user's usage habits (users will not always install their mobile phones on their bodies, nor will they use their mobile phones during work, housework and exercise), the duration of life data that can be collected by traditional smart terminals such as mobile phones and computers is very short, and cannot cover the time period when the user stays awake during the day, nor can it be guaranteed that the content of the collected life data is always related to the user. A considerable number of users have the habit of wearing glasses all day long, so smart glasses can easily realize all-weather data collection, and the collected data is first-person perspective data that is always related to the user.

[0028] See also Figure 2 and Figure 3 , Figure 2 is a module schematic diagram of a device for constructing a digital avatar based on smart glasses according to the present invention, Figure 3 The figure is a flow chart of the method for constructing a digital avatar based on smart glasses of the present invention.

[0029] The digital avatar construction device based on smart glasses of the present invention includes: a data acquisition module M1, a content recognition module M2, a portrait construction module M3, a training module M4 and a circulation module M5.

[0030] The data acquisition module M1 is used to execute step S1: obtain the user's life data from the first-person perspective through the smart glasses 1. The life data is multimodal data, which includes: video data from the first-person perspective collected by the camera of the smart glasses, audio data from the first-person perspective collected by the microphone, head movement data collected by the inertial sensor, and time data obtained through the network, spatial data obtained through the position sensor or the network, etc. In this embodiment, the video data and audio data collected by the camera and the microphone play a major role in the subsequent construction of the digital avatar, and the head movement data, time data and spatial features play an auxiliary role.

[0031] The content recognition module M2 is used to execute step S2: extract and recognize features of the life data to obtain content data.

[0032] In one embodiment, the digital avatar construction device further includes a preprocessing module M20, which is used to perform step S20 before step S2: preprocess the life data to obtain preprocessed data. Step S2 becomes: extract and identify features of the preprocessed data to obtain content data.

[0033] Wherein, step S20 specifically includes: S201, performing data cleaning on the life data to extract valid life data fragments.

[0034] There are many invalid or low-quality life data fragments in continuous life data. For example, when the user takes off smart glasses or places them in the dark, the camera of the smart glasses will capture long-term static invalid video data. When the user is in a high-noise environment, the microphone will record low-quality audio data containing a lot of noise, resulting in unrecognizable audio content. These invalid or low-quality life data fragments are not helpful for the subsequent construction of digital twins, so they need to be removed. The purpose of data cleaning is to identify invalid or low-quality life data fragments. After removing these invalid or low-quality life data fragments, life data fragments that are helpful for the construction of digital twins can be extracted.

[0035] Specifically, step S201 can remove low-quality life data segments containing a large amount of noise through a filtering and denoising algorithm; and can extract valid video data segments or audio data segments that record key events through a video key frame extraction algorithm based on image similarity, or an artificial intelligence-based content recognition algorithm (such as the YOLO target detection algorithm, the ASR speech recognition algorithm).

[0036] S202, time alignment and matching are performed on the valid life data segments to obtain preprocessed data. Since the life data includes multimodal data, these multimodal data must be aligned and matched based on time so that they can be mutually integrated and referenced in the subsequent feature extraction steps. In this embodiment, the time sampling rate of each multimodal data included in the life data is the same, and time alignment and matching can be achieved as long as the timestamp mark of each data point is aligned. In other embodiments with different time sampling rates, other time alignment strategies such as interpolation can also be used for time alignment and matching.

[0037] Specifically, the content recognition module M2 includes: a video content recognition unit M21, an audio content recognition unit M22, a context content recognition unit M23 and a content data merging unit M24.

[0038] The video content recognition unit M21 is used to execute step S21: through the target detection algorithm, the video data in the pre-processed data is subjected to picture content recognition to obtain video content data. The video content data includes: scene type, and characters, objects, signs and text information in the scene. For example, in a scene where a user wears smart glasses and walks into a cafe to drink coffee, through an artificial intelligence-based target detection algorithm, it can be identified that the scene type is a cafe, the characters in the scene include waiters, the objects in the scene include tables and chairs, the signs in the scene include the trademark of the cafe, and the text in the scene includes menu content, etc. Specifically, AI target detection algorithms such as YOLO and SSD can be used to complete the recognition of video content data.

[0039] The audio content recognition unit M22 is used to perform step S22: perform audio content recognition on the audio data in the preprocessed data through an audio recognition algorithm based on artificial intelligence to obtain audio content data. The audio content data includes: the conversation content marked with the conversation participants, the emotional tendency of the conversation content, the type of ambient sound, the type of background music, the emotional tendency of the background music, and the single name of the background music.

[0040] Wherein, step S22 specifically includes: S221, using an audio segmentation algorithm, the audio data is segmented to separate the conversational voice data, ambient sound data, and background music data in the audio data. For example, in a scenario where a user is drinking coffee in a cafe, the microphone of the smart glasses will record the following audio data: ① the conversational voice between the user and the waiter; ② ambient sounds such as the waiter's walking, the collision of cutlery, and the tapping of keyboards; ③ background music played by the speakers of the cafe. Step S321 segments these three types of audio data to facilitate subsequent content recognition of the three types of audio data.

[0041] S222, using a speech recognition algorithm, performing content recognition on the conversation voice data to obtain conversation content with participants marked. For example, when a user is talking to a waiter, the speech recognition algorithm can distinguish the voices of the user and the waiter by analyzing the voiceprint or the sound source location, and output the conversation content between the user and the waiter in the form of text.

[0042] Furthermore, after step S222, the method further includes: analyzing the emotional tendency of the conversation content by using an emotional recognition algorithm. For example, when a user has an argument with a waiter, the emotional recognition algorithm can recognize that the user's emotional tendency is anger based on the conversation content.

[0043] S223, using a music recognition algorithm, the background music data is style-recognized to obtain the type of the background music. For example, the types of background music may include pop, classical, folk, rock, electronic, etc. Further, after step S323, it further includes: analyzing the emotional tendency of the background music by an emotional recognition algorithm. The emotional tendency of the background music may include excitement, relaxation, calmness, sadness, etc. Further, after step S323, the single name of the background music is also identified by searching the online music library.

[0044] S224, using an ambient sound recognition algorithm, the ambient sound data is type-recognized to obtain the type of ambient sound. The type of ambient sound can be used to assist in identifying the type of scene the user is in. For example, the characteristics of ambient sound in the three scene types of library, coffee shop, and commercial street are quite different. The artificial intelligence algorithm can integrate the ambient sound data and the video content data to improve the recognition accuracy of the scene type.

[0045] S225, making the conversation content, the type of background music and the type of ambient sound compose audio content data.

[0046] The context content recognition unit M23 is used to execute step S23: according to the time data and space data in the pre-processed data, the video content data and the audio content data are temporally and spatially associated to extract the context content data. For example, multiple independent video content data / audio content data can only respectively reflect scattered information that lacks connection with each other, such as "the user is taking the subway", "the user is drinking coffee in a cafe", "the user is working", etc. However, after combining the context content data, multiple related video content data / audio content data can reflect complete information with temporal and spatial connection, such as "the user starts taking subway line n from station A at 8:00 in the morning and exits station B at 8:30; after exiting the subway, the user walks for 15 minutes and enters a certain cafe to drink coffee at 8:45; 15 minutes later at 9:00, the user enters the office to work". The context content data reflects the temporal and spatial association characteristics of user behavior and preferences.

[0047] The content data merging unit M24 is used to execute step S24: composing the video content data, the audio content data and the context content data into content data.

[0048] The portrait building module M3 is used to execute step S3: build a user portrait according to the content data. The user portrait includes: preference and interest portrait, behavior pattern portrait, social and language style portrait.

[0049] Specifically, the portrait construction module M3 includes: a preference and interest portrait construction unit M31, a behavior pattern portrait construction unit M32 and a social and language style portrait construction unit M33.

[0050] The preference and interest portrait construction unit M31 is used to execute step S31: count the frequency of occurrence of each element in each category in the content data, screen out several high-frequency elements with higher frequency of occurrence in each category, and infer the user's interest preferences, habitual choices and / or consumption tendencies based on the high-frequency elements, so as to establish the user's preference and interest portrait. For example, assuming that in the category of "consumption places frequented by users", the frequency of occurrence of a certain coffee shop is much higher than other consumption places, and in the category of "single items consumed by users in coffee shops", the frequency of occurrence of American coffee is much higher than other single items, then the intelligent agent can infer the preference and interest portrait of "the user likes to drink coffee, likes to patronize a certain coffee shop, and prefers American coffee". Elements of other categories can also be counted and profiled in the same way to understand the user's preferences and interests in each category.

[0051] The behavior pattern portrait construction unit M32 is used to execute step S32: according to the content data, analyze the user's daily routine and / or activity route, summarize the user's behavior pattern in each scenario, and thus establish a behavior pattern portrait. For example, in the breakfast scenario, the user always visits the same coffee shop and orders the same coffee; in the dinner scenario, he often tries new restaurants and new dishes. In other words, the user has different behavior patterns in the breakfast scenario and in the dinner scenario. The intelligent agent can summarize the differences in the user's behavior patterns in different time and space scenarios based on the contextual content data, thereby establishing a complete behavior pattern portrait.

[0052] The social and language style portrait construction unit M33 is used to execute step S33: according to the content data, analyze the user's overall language habits, expression style and / or common vocabulary, and analyze the user's interaction frequency, common topics, expression style and / or emotional characteristics with different social objects, so as to establish a social and language style portrait. The social and language style portrait is mainly constructed by analyzing the conversation content in the content data, the emotional tendency of the conversation content, and the style differences of the conversation content in different scenarios.

[0053] The training module M4 is used to execute step S4: train the intelligent agent according to the user portrait, and use the trained intelligent agent as the digital avatar model. Step S4 specifically includes: using the user portrait as the input prompt of the intelligent agent, and fine-tuning the parameters of the intelligent agent according to the prompt, so that the intelligent agent imitates the user's behavior.

[0054] The loop module M5 is used to execute step S5: repeatedly call the data acquisition module M1, the content recognition module M2, the portrait construction module M3 and the training module M4 to repeatedly execute steps S1-S4, so as to continuously acquire life data through the smart glasses, update the user portrait according to the updated life data, and compare the difference between the imitation behavior generated by the digital avatar model and the real behavior of the user, and further fine-tune the parameters of the digital avatar model.

[0055] The present invention has the following technical effects: The present invention utilizes the hardware advantages of smart glasses, obtains the user's first-person perspective life data through smart glasses around the clock, and constructs a user's portrait in the real world based on these life data, thereby training a digital avatar model with a high degree of simulation. The present invention uses a variety of content recognition algorithms to extract and recognize features of life data, so that the artificial intelligence entity has the ability to construct a user portrait based on the recognized content, breaking through the limitation of the existing digital avatar construction method that can only process virtual world trace information.

[0056] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and the present invention is also intended to include these modifications and modifications.

Claims

1. A method for constructing a digital avatar based on smart glasses, characterized in that: include: S1, obtains the user’s first-person perspective life data through smart glasses; S2, extracting and identifying features of the life data to obtain content data; S3, establishing a user profile based on the content data; S4, training an intelligent agent according to the user portrait, and using the trained intelligent agent as a digital avatar model.

2. The method for constructing a digital avatar based on smart glasses according to claim 1, characterized in that: The life data is multimodal data, which includes: video data, audio data, time data and space data; Step S2 specifically includes: S21, performing picture content recognition on the video data to obtain video content data; S22, performing audio content recognition on the audio data to obtain audio content data; S23, performing spatiotemporal association on the video content data and the audio content data according to the time data and the space data to obtain context content data; S24, making the video content data, audio content data and context content data form content data.

3. The method for constructing a digital avatar based on smart glasses according to claim 2, characterized in that: Step S22 specifically includes: S221, performing audio segmentation on the audio data to separate the conversation voice data, ambient sound data and background music data in the audio data; S222, performing content recognition on the conversation voice data to obtain the conversation content; S223, performing style recognition on the background music data to obtain the type of the background music; S224, performing type identification on the ambient sound data to obtain the type of the ambient sound; S225, making the conversation content, the type of background music and the type of ambient sound compose audio content data.

4. The method for constructing a digital avatar based on smart glasses according to claim 3, characterized in that: The video content data described in step S21 includes: scene type, and characters, objects, signs and text information in the scene.

5. The method for constructing a digital avatar based on smart glasses according to claim 4, characterized in that: The user portrait includes: preference and interest portrait, behavior pattern portrait and social and language style portrait; Step S3 specifically includes: S31, counting the frequency of occurrence of each element in the content data, selecting several high-frequency elements with high frequency of occurrence, and inferring the user's interest preferences, habitual choices and / or consumption tendencies based on the high-frequency elements, thereby establishing a preference and interest profile of the user; S32, analyzing the user's daily routine and / or activity route according to the content data, summarizing the user's behavior patterns in various scenarios, and thus establishing a behavior pattern portrait; S33, analyzing the user's language habits, expression style and / or common vocabulary based on the content data, and analyzing the user's interaction frequency, common topics, expression style and / or emotional characteristics with different social objects, so as to establish a social and language style portrait.

6. The method for constructing a digital avatar based on smart glasses according to claim 5, characterized in that: Step S4 specifically includes: using the user portrait as the input Prompt of the agent, and fine-tuning the parameters of the agent according to the Prompt, thereby obtaining a digital avatar model.

7. The method for constructing a digital avatar based on smart glasses according to any one of claims 1 to 6, characterized in that: It also includes step S5 after step S4: repeating steps S1-S4, continuously acquiring life data through smart glasses, updating the user portrait according to the updated life data, and comparing the difference between the imitation behavior generated by the digital avatar model and the real behavior of the user, and further fine-tuning the parameters of the digital avatar model.

8. The method for constructing a digital avatar based on smart glasses according to claim 1, characterized in that: Also includes a pre-processing step before step S2; The pre-processing step comprises: Performing data cleaning on the life data to extract valid life data fragments; Performing time alignment and matching on the valid life data segments to obtain preprocessed data; The step S2 becomes: extracting and identifying features of the pre-processed data to obtain content data.

9. A digital avatar construction device based on smart glasses, characterized in that: include: A data acquisition module, which is used to acquire the user's life data from the first-person perspective through smart glasses; A content recognition module, which is used to extract and recognize features of the life data to obtain content data; A portrait building module, which is used to build a user portrait based on the content data; A training module trains an intelligent agent according to the user portrait and uses the trained intelligent agent as a digital avatar model.

10. A digital avatar construction system, characterized in that: include: Smart glasses and processors; The smart glasses collect life data from the user's first-person perspective; The processor executes the method for constructing a digital avatar based on smart glasses according to any one of claims 1-8 based on the life data acquired by the smart glasses, thereby constructing a digital avatar model of the user.

Images