A cemetery digital management system and method with emotional digital human interaction

Abstract

The application relates to the technical field of digital cemetery management, in particular to a digital cemetery management system and method with emotional digital human interaction, a deceased memory bank dynamic updating module in the system is used for generating mouth shape animation and facial micro-expression of a deceased digital human in response to voice in synchronization with a static image model in a digital human interaction feedback model of the deceased according to obtained current interaction emotion tags, and the mouth shape animation and the facial micro-expression are fed back to relatives through a terminal device for interaction; and the initial memory bank of the deceased is dynamically updated according to interaction data. The application simulates the natural forgetting law and memory strengthening law of human memory by dynamically updating the memory weight of each historical memory item in the constructed initial memory bank of the deceased; and the language style of the digital human can continuously evolve along with the interaction process with the relatives of the deceased by taking the cumulative interaction text word number as a trigger condition for language habit feature vector updating.

Description

Technical Field

[0001] This invention relates to the field of digital management technology for cemeteries, specifically to a digital management system and method for cemeteries with emotional digital human interaction. Background Technology

[0002] With the development of digital technology, cemetery management is gradually extending from traditional physical grave management to digital memorial spaces. Currently, some technologies are attempting to combine artificial intelligence with virtual digital humans to create digital avatars of the deceased, enabling family members to interact with these digital avatars to some extent through voice or text. For example, by collecting images, voice recordings, and text records of the deceased, deep learning models are used to simulate the deceased's appearance and demeanor in the form of a digital human. However, in the implementation of existing technologies, the interactive content of digital humans is usually generated once based on a pre-trained model. This prevents dynamic learning during continuous dialogue between the deceased's digital avatar and family members, and also hinders the construction and continuous updating of the deceased's digital avatar's memory bank. Consequently, as the number of interactions between the deceased's digital avatar and family members increases, the responses gradually become repetitive and patterned. Furthermore, existing cemetery management systems primarily focus on security monitoring of graves and reservation management, neglecting to integrate and analyze the interactive functions between the emotionally resonant digital avatar and family members. Therefore, the dialogue between family members and the digital avatar in existing technologies often lacks simulation of the deceased's unique language habits and life memories, resulting in significant discrepancies between the dialogue content and the deceased's expressive habits and life experiences. Therefore, existing technologies have significant shortcomings. Summary of the Invention

[0003] The purpose of this invention is to provide a cemetery digital management system and method with emotional digital human interaction to solve the problems mentioned in the background art.

[0004] To address the aforementioned technical problems, this invention provides the following technical solution: a digital management method for cemeteries with emotional digital human interaction, comprising: S1. Construct a static image model of the deceased and mouth-shaped animation driving parameters based on the deceased's portrait; construct a speech synthesis model of the deceased based on the deceased's audio recording; construct the deceased's language habit feature vector based on the deceased's biographical text; generate a digital human interactive feedback model of the deceased. S2. Combine the pre-set records of the deceased's life trajectory events, the pre-set relationship graph of the deceased, and the obtained language habit feature vector to construct the initial memory bank of the deceased; the initial memory bank of the deceased includes multiple historical memory entries of the deceased, each historical memory entry corresponds to an event, and the event is bound to an emotional tendency label and an initial memory weight. S3. Obtain and parse the current multimodal interaction information of the deceased's family members to obtain the current interaction sentiment label and the current interaction semantic vector based on the current multimodal interaction information; S4. Based on the current interaction semantic vector, retrieve historical memory entries associated with the current multimodal interaction information from the deceased's initial memory bank, and calculate the similarity score for each historical memory; dynamically generate the digital human's response text and response speech based on the obtained current interaction sentiment tags and the similarity scores of associated historical memories; the response text is obtained by rewriting the retrieved matching memory content into sentences, and the response speech is obtained by splicing together Chinese character pronunciation fragments corresponding to the sentiment tags selected from the obtained speech synthesis model; S5. Based on the obtained current interaction emotion tags, drive the static image model in the obtained digital human interaction feedback model of the deceased to generate lip-syncing animation and facial micro-expressions synchronized with the response voice of the obtained digital human, and feed them back to the family members through the terminal device for interaction; and take the current multimodal interaction information of the deceased's family members, the dynamically generated digital human response text, the current interaction emotion tags and the current interaction semantic vector as a set of interaction memory data, and dynamically update the historical memory entries in the deceased's initial memory bank.

[0005] Furthermore, the specific method for constructing the static image model of the deceased and the lip-shape animation driving parameters based on the deceased's photograph in S1 is as follows: The deceased's portrait provided by the family is used as the static image of the digital human. The pronunciation time of each Chinese character is extracted from the deceased's recording, and the short-time energy envelope mean of each Chinese character is calculated. The normalized result corresponding to the short-time energy envelope mean of each Chinese character is used as the reference value of the mouth opening degree of the corresponding Chinese character, and a mapping table between the deceased's text and mouth shape is constructed. When the digital human image of the deceased interacts with the family, the mouth opening degree corresponding to each Chinese character in the Chinese character sequence in the digital human's response text is obtained by querying the mapping table between the deceased's text and mouth shape in turn, and a continuous mouth shape change curve corresponding to the digital human's response text is generated.

[0006] Furthermore, the specific method for constructing the deceased's speech synthesis model based on the deceased's recording in S1 is as follows: Speech detection is performed on the recordings of the deceased, and the independent pronunciation segments of each Chinese character in the recordings are segmented out. An array consisting of each Chinese character and its corresponding emotion category is stored. The emotion categories include neutral, joy, and sadness. In this invention, each emotion category is represented by a constant. When obtaining the response speech corresponding to the digital human's response text, based on the Chinese character sequence in the digital human's response text and the current interaction emotion tag corresponding to the current multimodal interaction information of the deceased's family, the pronunciation segment of each Chinese character in the Chinese character sequence in the digital human's response text that has the same emotion category as the corresponding current interaction emotion tag is selected, and the obtained Chinese character pronunciation segments are spliced ​​together sequentially from the Chinese character sequence in the digital human's response text. The boundary between adjacent Chinese character pronunciation segments in the spliced ​​result is processed by fading in and out to obtain the response speech corresponding to the digital human's response text.

[0007] Furthermore, the specific method for constructing the deceased's language habit feature vector based on the deceased's biographical text in S1 is as follows: Using the deceased's biographical text as a corpus, the top n most frequent content words were counted using Jieba word segmentation to form an n-dimensional word frequency vector for the deceased; the sum of the average sentence length, exclamation mark density, question mark density, and modal particle density in the deceased's biographical text was used as the deceased's style feature vector; the concatenation of the deceased's n-dimensional word frequency vector and the deceased's style feature vector was used as the deceased's language habit feature vector.

[0008] In this invention, the exclamation mark density is equal to the quotient obtained by dividing the number of times the exclamation mark appears by the total number of sentences in the deceased's biographical text; the question mark density is equal to the quotient obtained by dividing the number of times the question mark appears by the total number of sentences in the deceased's biographical text; and the modal particle density is equal to the quotient obtained by dividing the total number of times each predefined modal particle appears by the total number of sentences in the deceased's biographical text.

[0009] Furthermore, the current multimodal interaction information of the deceased's family members in S3 includes the family members' voice input information, text input information, and facial image input information; In the process of obtaining the current interaction sentiment tag based on the current multimodal interaction information, sentiment words are matched in the text input information of the current multimodal interaction information of the deceased's family members. The number of preset positive keywords and negative keywords are counted respectively, and the text sentiment tag is obtained according to the statistical results. Specifically, the difference between the number of positive keywords and the number of negative keywords is divided by the sum of the number of positive keywords, the number of negative keywords, and 1. This quotient is used as the emotional score corresponding to the text input information in the current multimodal interaction information of the deceased's family. If the obtained emotional score is greater than the first threshold, the text emotional label is determined to be joyful; if the obtained emotional score is less than the second threshold, the text emotional label is determined to be sad; otherwise, the text emotional label is determined to be neutral. The voice input information in the current multimodal interaction information of the deceased's family members is converted into text input information for processing, and the processed text sentiment tags are used as the voice sentiment tags corresponding to the voice input information in the current multimodal interaction information of the deceased's family members. Extract a local binary pattern histogram of the facial image input information of the deceased's family members from the current multimodal interaction information, and obtain facial emotion labels through an input vector machine classifier. Specifically, each frame of facial image captured by the camera is detected and the face region is extracted; the detected face region is scaled to 48×48 pixels, a uniform local binary pattern histogram is extracted, divided into 8×8 blocks, each block yields a 59-dimensional histogram, and all blocks are stitched together to obtain a 3776-dimensional feature vector; the obtained stitched vector is input into a pre-trained vector machine classifier to output the corresponding facial emotion label.

[0010] The obtained voice emotion tags, text emotion tags, and facial emotion tags are voted on by majority vote to obtain the final current interaction emotion tag; if the voice emotion tags, text emotion tags, and facial emotion tags are all different, the obtained text emotion tag is used as the final current interaction emotion tag. In obtaining the current interaction semantic vector based on the current multimodal interaction information, the text input information in the current multimodal interaction information of the deceased's family is segmented into words, and the TF-IDF value of each word is calculated. The vector composed of the TF-IDF values ​​of all words is used as the current interaction semantic vector based on the current multimodal interaction information. The calculation formulas involved are as follows:

[0011] in, H represents the relative frequency of term Q in the current text; H represents the total number of descriptive texts corresponding to historical memory entries in the deceased's initial memory bank. This indicates the number of descriptive text entries for term Q in the deceased's initial memory bank.

[0012] Furthermore, the similarity score of each historical memory in S4 is obtained by calculating the cosine similarity between the interaction semantic vector bound to the corresponding historical memory entry in the database and the current interaction semantic vector, the preset intimacy coefficient between the current family member and the deceased (obtained by querying the preset intimacy coefficient list), and the cumulative product of the memory weights bound to the corresponding historical memory entry. In step S4, during the process of dynamically generating the digital human's response text based on the obtained current interaction sentiment tags and the similarity scores of associated historical memories, the historical memory entry with the highest similarity score among the historical memory entries retrieved from the deceased's initial memory bank and associated with the current multimodal interaction information is used as the main matching content, and the descriptive text of the historical memory entry with the highest similarity score is used as the basis; an sentiment prefix is ​​added to the beginning of the sentence based on the current interaction sentiment tags; the third person in the descriptive text of the historical memory entry with the highest similarity score is replaced with the first person, and the synonyms of each term in the descriptive text of the historical memory entry with the highest similarity score are randomly replaced to obtain the digital human's response text; if the length of the digital human's response text is less than the preset length, a descriptive fragment of a historical memory entry related to time or person is selected from the deceased's initial memory bank and spliced ​​to the end of the obtained digital human's response text to obtain a new digital human's response text.

[0013] Furthermore, during the process of dynamically updating the historical memory entries in the initial memory bank of the deceased in S5, the updated objects include the historical memory entries in the memory bank, the memory weight of each historical memory entry, and the language habit feature vector of the deceased. When updating historical memory entries in the memory bank, the generated interactive memory data is stored as a new historical memory entry in the deceased's initial memory bank; The calculation formula involved in updating the memory weight of each historical memory entry in the memory bank is as follows:

[0014] in, This represents the memory weight after the i-th historical memory entry is updated; This represents the memory weight before the i-th historical memory entry was updated; This represents the activation coefficient of the i-th historical memory entry. If the cosine similarity between the interaction semantic vector bound to the i-th historical memory entry in the database and the current interaction semantic vector is greater than a preset value, then it is determined that... The value is equal to the cosine similarity between the interaction semantic vector bound to the i-th historical memory entry in the database and the current interaction semantic vector; otherwise, it is determined that... The value is 0; This indicates the preset learning rate; This represents the preset forgetting rate; min{} represents the function to find the minimum value; The update of the deceased's language habit feature vector is triggered when the accumulated interactive text word count reaches a preset word count threshold; the accumulated interactive text is used as the deceased's biographical text to construct the deceased's language habit feature vector, denoted as the incremental feature vector; the deceased's language habit feature vector is updated through a preset weighted formula, the calculation formulas involved are as follows:

[0015] in, This represents the updated language habit feature vector of the deceased. This represents the language habit feature vector of the deceased before the update. Represents the incremental feature vector; This indicates that the weights are updated incrementally, and The value of the incremental update weight decays with the number of updates; The decay formula with each update is as follows:

[0016] in, This represents the incremental update weight after the g-th update; d represents the initial incremental weight; d is the preset decay coefficient; when the Euclidean distance between the language habit feature vectors before and after the update is less than the preset constant in m consecutive updates, the update state of the deceased's language habit feature vector is determined to be the convergent state, the preset word count threshold is doubled as the new preset word count threshold, and half of the initial incremental weight is used as the new initial incremental weight; m is the preset constant.

[0017] A digital management system for cemeteries with emotional digital human interaction includes: The digital human interaction feedback model construction module constructs a digital human interaction feedback model for the deceased based on the deceased's photographs, audio recordings, and biographical texts. The deceased's memory bank construction module is used to combine pre-set records of deceased life trajectory events, pre-set deceased relationship graphs, and obtained language habit feature vectors to construct an initial memory bank for the deceased; The multimodal interaction information parsing module is used to obtain and parse the current multimodal interaction information of the deceased's family members to obtain the current interaction sentiment tag and the current interaction semantic vector based on the current multimodal interaction information; The digital human feedback information dynamic acquisition module is used to retrieve historical memory entries associated with the current multimodal interaction information in the deceased's initial memory bank based on the current interaction semantic vector, and calculate the similarity score of each historical memory; based on the obtained current interaction emotional tags and the similarity scores of associated historical memories, the module dynamically generates the digital human's response text and response voice. The deceased's memory bank dynamic update module is used to drive the static image model in the digital human interaction feedback model of the deceased to generate lip-sync animation and facial micro-expressions that are synchronized with the response voice of the digital human, based on the obtained current interaction emotion tags, and to provide feedback to the family members through the terminal device for interaction; and to dynamically update the deceased's initial memory bank based on the interaction data.

[0018] Furthermore, the digital human interactive feedback model construction module includes a static image model construction unit, a speech synthesis model construction unit, a language habit feature vector construction unit, and an interactive feedback model generation unit. The static image model construction unit constructs a static image model of the deceased and mouth animation driving parameters based on the deceased's portrait. The speech synthesis model building unit constructs a speech synthesis model of the deceased based on the deceased's recording. The language habit feature vector construction unit constructs the language habit feature vector of the deceased based on the deceased's biographical text. The interactive feedback model generation unit is used to summarize the acquisition results of the static image model construction unit, the speech synthesis model construction unit, and the language habit feature vector construction unit to generate the digital human interactive feedback model of the deceased.

[0019] Furthermore, the deceased's memory bank dynamic update module includes a model-driven unit and a memory bank dynamic update unit. The model driving unit is used to drive the static image model in the digital human interaction feedback model of the deceased to generate mouth animation and facial micro-expressions synchronized with the response voice of the digital human, based on the obtained current interaction emotion tag, and to provide feedback to the family members through the terminal device for interaction. The memory bank dynamic update unit is used to dynamically update the historical memory entries in the deceased's initial memory bank by taking the current multimodal interaction information of the deceased's family members during the interaction process, the dynamically generated digital human response text, the current interaction emotion tags, and the current interaction semantic vector as a set of interaction memory data.

[0020] Compared with the prior art, the beneficial effects achieved by the present invention are: (1) This invention simulates the natural forgetting and memory reinforcement laws of human memory to a certain extent by dynamically updating the memory weight of each historical memory entry in the constructed initial memory bank of the deceased; at the same time, by using the cumulative number of words of interactive text as the trigger condition for updating the language habit feature vector, and updating the language habit feature vector of the deceased based on the input of the deceased's family and the interaction data of the digital human, the language style of the digital human can continue to evolve with the interaction process with the deceased's family. (2) To a certain extent, this invention can realize cross-temporal and spatial emotional dialogue between the deceased's family and the deceased, and through the dynamic update mechanism of the deceased's initial memory bank, it realizes differentiated management of information of different deceased. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the structure of a cemetery digital management system with emotional digital human interaction according to the present invention; Figure 2 This is a flowchart illustrating a digital management method for cemeteries with emotional digital human interaction, as described in this invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Please see Figure 1 This embodiment provides a cemetery digital management system with emotional digital human interaction, including: The digital human interaction feedback model construction module includes a static image model construction unit, a speech synthesis model construction unit, a language habit feature vector construction unit, and an interaction feedback model generation unit. The static image model construction unit constructs a static image model of the deceased and mouth animation driving parameters based on the deceased's portrait. The speech synthesis model building unit constructs a speech synthesis model of the deceased based on the deceased's recording. The language habit feature vector construction unit constructs the language habit feature vector of the deceased based on the deceased's biographical text. The interactive feedback model generation unit is used to summarize the acquisition results of the static image model construction unit, the speech synthesis model construction unit, and the language habit feature vector construction unit to generate the digital human interactive feedback model of the deceased.

[0024] The deceased's memory bank construction module is used to combine pre-set records of deceased life trajectory events, pre-set deceased relationship graphs, and obtained language habit feature vectors to construct an initial memory bank for the deceased; The multimodal interaction information parsing module is used to obtain and parse the current multimodal interaction information of the deceased's family members to obtain the current interaction sentiment tag and the current interaction semantic vector based on the current multimodal interaction information; The digital human feedback information dynamic acquisition module is used to retrieve historical memory entries associated with the current multimodal interaction information in the deceased's initial memory bank based on the current interaction semantic vector, and calculate the similarity score of each historical memory; based on the obtained current interaction emotional tags and the similarity scores of associated historical memories, the module dynamically generates the digital human's response text and response voice. The deceased's memory bank dynamic update module includes a model-driven unit and a memory bank dynamic update unit. The model driving unit is used to drive the static image model in the digital human interaction feedback model of the deceased to generate mouth animation and facial micro-expressions synchronized with the response voice of the digital human, based on the obtained current interaction emotion tag, and to provide feedback to the family members through the terminal device for interaction. The memory bank dynamic update unit is used to dynamically update the historical memory entries in the deceased's initial memory bank by taking the current multimodal interaction information of the deceased's family members during the interaction process, the dynamically generated digital human response text, the current interaction emotion tags, and the current interaction semantic vector as a set of interaction memory data.

[0025] like Figure 2 As shown, this embodiment also provides a method for digital management of cemeteries with emotional digital human interaction, including: S1. Construct a static image model of the deceased and mouth-shaped animation driving parameters based on the deceased's portrait; construct a speech synthesis model of the deceased based on the deceased's audio recording; construct the deceased's language habit feature vector based on the deceased's biographical text; generate a digital human interactive feedback model of the deceased. The specific method for constructing the static image model of the deceased and the mouth animation driving parameters based on the deceased's portrait in S1 is as follows: The deceased's portrait provided by the family is used as the static image of the digital human. The pronunciation time of each Chinese character is extracted from the deceased's recording, and the short-time energy envelope mean of each Chinese character is calculated. The normalized result corresponding to the short-time energy envelope mean of each Chinese character is used as the reference value of the mouth opening degree of the corresponding Chinese character, and a mapping table between the deceased's text and mouth shape is constructed. When the digital human image of the deceased interacts with the family, the mouth opening degree corresponding to each Chinese character in the Chinese character sequence in the digital human's response text is obtained by querying the mapping table between the deceased's text and mouth shape in turn, and a continuous mouth shape change curve corresponding to the digital human's response text is generated.

[0026] The specific method for constructing the deceased's speech synthesis model based on the deceased's recording in S1 is as follows: Performing speech detection on the deceased's recorded audio, segmenting independent pronunciation fragments of each Chinese character in the deceased's recorded audio, and storing an array composed of each Chinese character and the corresponding emotion category of the Chinese character; the emotion categories include neutral, joy and sadness, each emotion category is represented by a constant in the present invention, and in this embodiment, neutral is represented by 0, joy is represented by 1, and sadness is represented by -1; When acquiring the response speech corresponding to the digital human response text, according to the Chinese character sequence in the digital human response text and the current interaction emotion label corresponding to the current multimodal interaction information of the deceased's family members, each Chinese character pronunciation fragment in the Chinese character sequence of the digital human response text whose corresponding emotion category is the same as the corresponding current interaction emotion label is screened out, the obtained Chinese character pronunciation fragments are spliced in sequence according to the Chinese character sequence in the digital human response text, and the junction of adjacent Chinese character pronunciation fragments in the spliced result is processed in a fade-in and fade-out manner, so as to obtain the response speech corresponding to the digital human response text.

[0027] The specific manner of constructing the language habit feature vector of the deceased according to the deceased's life text in S1 is as follows: Taking the deceased's life text as a corpus, using Jieba word segmentation to count the top n real words with the highest frequency, to form an n-dimensional word frequency vector of the deceased; taking the summary set of the average sentence length, exclamation mark density, question mark density and modal particle density in the deceased's life text as the style feature vector of the deceased; taking the splicing result of the n-dimensional word frequency vector of the deceased and the style feature vector of the deceased as the language habit feature vector of the deceased.

[0028] In this embodiment, the exclamation mark density is equal to the quotient obtained by dividing the counted number of occurrences of exclamation marks by the total number of sentences in the deceased's life text; the question mark density is equal to the quotient obtained by dividing the counted number of occurrences of question marks by the total number of sentences in the deceased's life text; the modal particle density is equal to the quotient obtained by dividing the counted total number of occurrences of each predefined modal particle by the total number of sentences in the deceased's life text. The predefined modal particles in this embodiment include ne, a, o, la, ma, ya, ba, bei, lo, yo, he, ha, etc.

[0029] S2. Constructing an initial memory base of the deceased by combining the pre-set life trajectory event records of the deceased, the pre-set relationship graph of the deceased and the obtained language habit feature vector; the initial memory base of the deceased includes a plurality of historical memory entries of the deceased, each historical memory entry corresponds to an event, and the event is bound with an emotion tendency label and an initial memory weight; S3. Acquiring and parsing the current multimodal interaction information of the deceased's family members, to obtain a current interaction emotion label and a current interaction semantic vector based on the current multimodal interaction information; The current multimodal interaction information of the deceased's family members in S3 includes the family members' voice input information, text input information and facial image input information; In the process of obtaining the current interaction sentiment tag based on the current multimodal interaction information, sentiment words are matched to the text input information in the current multimodal interaction information of the deceased's family. The number of preset positive keywords and negative keywords are counted respectively, and the text sentiment tag is obtained according to the statistical results. Specifically, the difference between the number of positive keywords and the number of negative keywords is calculated and divided by the sum of the number of positive keywords, the number of negative keywords, and 1. This quotient is used as the sentiment score corresponding to the text input information in the current multimodal interaction information of the deceased's family. If the obtained sentiment score is greater than the first threshold, the text sentiment tag is determined to be of the joy type, and the constant corresponding to the text sentiment tag is 1. If the obtained sentiment score is less than the second threshold, the text sentiment tag is determined to be of the sadness type, and the constant corresponding to the text sentiment tag is -1. Otherwise, the text sentiment tag is determined to be of the neutral type, and the constant corresponding to the text sentiment tag is 0. The voice input information in the current multimodal interaction information of the deceased's family members is converted into text input information for processing, and the processed text sentiment tags are used as the voice sentiment tags corresponding to the voice input information in the current multimodal interaction information of the deceased's family members. A local binary pattern histogram of a preset size is extracted from the facial image input information in the current multimodal interaction information of the deceased's family members, and then fed into a vector machine classifier to obtain facial emotion labels. Specifically, each frame of facial image captured by the camera is detected, and the face region is extracted. The detected face region is scaled to 48×48 pixels, and a uniform local binary pattern histogram is extracted, divided into 8×8 blocks, each block yielding a 59-dimensional histogram. After all blocks are concatenated, a 3776-dimensional feature vector is obtained. The obtained concatenated vector is input into a pre-trained vector machine classifier, which outputs the corresponding facial emotion label.

[0030] The obtained voice emotion tags, text emotion tags, and facial emotion tags are voted on by majority vote to obtain the final current interaction emotion tag; if the voice emotion tags, text emotion tags, and facial emotion tags are all different, the obtained text emotion tag is used as the final current interaction emotion tag. In obtaining the current interaction semantic vector based on the current multimodal interaction information, the text input information in the current multimodal interaction information of the deceased's family is segmented into words, and the TF-IDF value of each word is calculated. The vector composed of the TF-IDF values ​​of all words is used as the current interaction semantic vector based on the current multimodal interaction information. The calculation formulas involved are as follows:

[0031] in, H represents the relative frequency of term Q in the current text; H represents the total number of descriptive texts corresponding to historical memory entries in the deceased's initial memory bank. This indicates the number of descriptive text entries for term Q in the deceased's initial memory bank.

[0032] S4. Based on the current interaction semantic vector, retrieve historical memory entries associated with the current multimodal interaction information from the deceased's initial memory bank, and calculate the similarity score for each historical memory; dynamically generate the digital human's response text and response speech based on the obtained current interaction sentiment tags and the similarity scores of associated historical memories; the response text is obtained by rewriting the retrieved matching memory content into sentences, and the response speech is obtained by splicing together Chinese character pronunciation fragments corresponding to the sentiment tags selected from the obtained speech synthesis model; The similarity score of each historical memory in S4 is obtained by calculating the cosine similarity between the interaction semantic vector bound to the corresponding historical memory entry in the database and the current interaction semantic vector, the preset intimacy coefficient between the current family member and the deceased (obtained by querying the preset intimacy coefficient list), and the cumulative product of the memory weights bound to the corresponding historical memory entry. In step S4, during the process of dynamically generating the digital human's response text based on the obtained current interaction sentiment tags and the similarity scores of associated historical memories, the historical memory entry with the highest similarity score among the historical memory entries retrieved from the deceased's initial memory bank and associated with the current multimodal interaction information is used as the main matching content, and the description text of the historical memory entry with the highest similarity score is used as the basis; an sentiment prefix is ​​added to the beginning of the sentence based on the current interaction sentiment tags; the third person in the description text of the historical memory entry with the highest similarity score is replaced with the first person, and the synonyms of each term in the description text of the historical memory entry with the highest similarity score are randomly replaced to obtain the digital human's response text; if the length of the digital human's response text is less than the preset length, a description fragment of a historical memory entry related to a time or person is selected from the deceased's initial memory bank and spliced ​​to the end of the obtained digital human's response text to obtain a new digital human's response text.

[0033] S5. Based on the obtained current interaction emotion tags, drive the static image model in the obtained digital human interaction feedback model of the deceased to generate lip-syncing animation and facial micro-expressions synchronized with the response voice of the obtained digital human, and feed them back to the family members through the terminal device for interaction; and take the current multimodal interaction information of the deceased's family members, the dynamically generated digital human response text, the current interaction emotion tags and the current interaction semantic vector as a set of interaction memory data, and dynamically update the historical memory entries in the deceased's initial memory bank.

[0034] In the process of S5 dynamically updating the historical memory entries in the initial memory bank of the deceased, the updated objects include the historical memory entries in the memory bank, the memory weight of each historical memory entry, and the language habit feature vector of the deceased. When updating historical memory entries in the memory bank, the generated interactive memory data is stored as a new historical memory entry in the deceased's initial memory bank; The calculation formula involved in updating the memory weight of each historical memory entry in the memory bank is as follows:

[0035] in, This represents the memory weight after the i-th historical memory entry is updated; This represents the memory weight before the i-th historical memory entry was updated; This represents the activation coefficient of the i-th historical memory entry. If the cosine similarity between the interaction semantic vector bound to the i-th historical memory entry in the database and the current interaction semantic vector is greater than a preset value, then it is determined that... The value is equal to the cosine similarity between the interaction semantic vector bound to the i-th historical memory entry in the database and the current interaction semantic vector; otherwise, it is determined that... The value is 0; This indicates the preset learning rate; This represents the preset forgetting rate; min{} represents the function to find the minimum value; The update of the deceased's language habit feature vector is triggered when the accumulated interactive text word count reaches a preset word count threshold; the accumulated interactive text is used as the deceased's biographical text to construct the deceased's language habit feature vector, denoted as the incremental feature vector; the deceased's language habit feature vector is updated through a preset weighted formula, the calculation formulas involved are as follows:

[0036] in, This represents the updated language habit feature vector of the deceased. This represents the language habit feature vector of the deceased before the update. Represents the incremental feature vector; This indicates that the weights are updated incrementally, and The value of the incremental update weight decays with the number of updates; The decay formula with each update is as follows:

[0037] in, This represents the incremental update weight after the g-th update; The initial incremental weight is 0.2 by default; the default value for the preset word count threshold is 1000 words; d is the preset decay coefficient, which is 0.9 by default; when the Euclidean distance between the language habit feature vectors before and after the update is less than the preset constant in m consecutive updates, the update state of the deceased's language habit feature vector is determined to be in a convergent state, the preset word count threshold is doubled as the new preset word count threshold, and half of the initial incremental weight is used as the new initial incremental weight; m is a preset constant.

[0038] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0039] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for digital management of cemeteries with emotional digital human interaction, characterized in that, include: S1. Construct a static image model of the deceased and mouth animation driving parameters based on the deceased's portrait; Construct a speech synthesis model of the deceased based on audio recordings; construct a language habit feature vector of the deceased based on biographical texts of the deceased; generate a digital human interactive feedback model of the deceased. S2. Combine the pre-set records of the deceased's life trajectory events, the pre-set relationship graph of the deceased, and the obtained language habit feature vector to construct the initial memory bank of the deceased; the initial memory bank of the deceased includes multiple historical memory entries of the deceased, each historical memory entry corresponds to an event, and the event is bound to an emotional tendency label and an initial memory weight. S3. Obtain and parse the current multimodal interaction information of the deceased's family members to obtain the current interaction sentiment label and the current interaction semantic vector based on the current multimodal interaction information; S4. Based on the current interaction semantic vector, retrieve historical memory entries associated with the current multimodal interaction information from the deceased's initial memory bank, and calculate the similarity score for each historical memory; based on the obtained current interaction sentiment tags and the similarity scores of associated historical memories, dynamically generate the digital human's response text and response voice. S5. Based on the obtained current interactive emotion tags, drive the static image model in the obtained digital human interaction feedback model of the deceased to generate mouth animation and facial micro-expressions synchronized with the response voice of the obtained digital human, and feed them back to the family members through the terminal device for interaction. The current multimodal interaction information of the deceased's family members, the dynamically generated digital human response text, the current interaction emotion tags, and the current interaction semantic vectors are collected as a set of interaction memory data, which are then used to dynamically update the historical memory entries in the deceased's initial memory bank.

2. The method for digital management of cemeteries with emotional digital human interaction according to claim 1, characterized in that, The specific method for constructing the static image model of the deceased and the mouth animation driving parameters based on the deceased's portrait in S1 is as follows: The deceased's portrait provided by the family is used as the static image of the digital human. The pronunciation time of each Chinese character is extracted from the deceased's recording, and the short-time energy envelope mean of each Chinese character is calculated. The normalized result corresponding to the short-time energy envelope mean of each Chinese character is used as the reference value of the mouth opening degree of the corresponding Chinese character, and a mapping table between the deceased's text and mouth shape is constructed. When the digital human image of the deceased interacts with the family, the mouth opening degree corresponding to each Chinese character in the Chinese character sequence in the digital human's response text is obtained by querying the mapping table between the deceased's text and mouth shape in turn, and a continuous mouth shape change curve corresponding to the digital human's response text is generated.

3. The method for digital management of cemeteries with emotional digital human interaction according to claim 1, characterized in that, The specific method for constructing the deceased's speech synthesis model based on the deceased's recording in S1 is as follows: Speech detection is performed on the recordings of the deceased, and the independent pronunciation segments of each Chinese character in the recordings are segmented out. An array consisting of each Chinese character and its corresponding emotion category is stored. The emotion categories include neutral, joy, and sadness. When obtaining the response speech corresponding to the digital human's response text, based on the Chinese character sequence in the digital human's response text and the current interaction emotion tag corresponding to the current multimodal interaction information of the deceased's family, the pronunciation segment of each Chinese character in the Chinese character sequence in the digital human's response text that has the same emotion category as the corresponding current interaction emotion tag is selected, and the obtained Chinese character pronunciation segments are spliced ​​together sequentially from the Chinese character sequence in the digital human's response text. The boundary between adjacent Chinese character pronunciation segments in the spliced ​​result is processed by fading in and out to obtain the response speech corresponding to the digital human's response text.

4. The method for digital management of cemeteries with emotional digital human interaction according to claim 1, characterized in that, The specific method for constructing the deceased's language habit feature vector based on the deceased's biographical text in S1 is as follows: Using the deceased's biographical text as a corpus, the top n most frequent content words were counted using Jieba word segmentation to form an n-dimensional word frequency vector for the deceased; the sum of the average sentence length, exclamation mark density, question mark density, and modal particle density in the deceased's biographical text was used as the deceased's style feature vector; the concatenation of the deceased's n-dimensional word frequency vector and the deceased's style feature vector was used as the deceased's language habit feature vector.

5. A method for digital management of cemeteries with emotional digital human interaction as described in claim 1, characterized in that, The current multimodal interaction information of the deceased's family members in S3 includes the family members' voice input information, text input information, and facial image input information; In the process of obtaining the current interaction sentiment tag based on the current multimodal interaction information, sentiment words are matched in the text input information of the current multimodal interaction information of the deceased's family members. The number of preset positive keywords and negative keywords are counted respectively, and the text sentiment tag is obtained according to the statistical results. The voice input information in the current multimodal interaction information of the deceased's family members is converted into text input information for processing, and the processed text sentiment tags are used as the voice sentiment tags corresponding to the voice input information in the current multimodal interaction information of the deceased's family members. Extract a local binary pattern histogram of the facial image input information of the deceased's family members from the current multimodal interaction information, and obtain facial emotion labels through an input vector machine classifier. The obtained voice emotion tags, text emotion tags, and facial emotion tags are voted on by majority vote to obtain the final current interaction emotion tag; if the voice emotion tags, text emotion tags, and facial emotion tags are all different, the obtained text emotion tag is used as the final current interaction emotion tag. In obtaining the current interaction semantic vector based on the current multimodal interaction information, the text input information in the current multimodal interaction information of the deceased's family is segmented into words, and the TF-IDF value of each word is calculated. The vector composed of the TF-IDF values ​​of all words is used as the current interaction semantic vector based on the current multimodal interaction information. The calculation formulas involved are as follows: in, H represents the relative frequency of term Q in the current text; H represents the total number of descriptive texts corresponding to historical memory entries in the deceased's initial memory bank. This indicates the number of descriptive text entries for term Q in the deceased's initial memory bank.

6. The method for digital management of cemeteries with emotional digital human interaction according to claim 1, characterized in that, The similarity score of each historical memory in S4 is obtained by multiplying the cosine similarity between the interaction semantic vector bound to the corresponding historical memory entry in the database and the current interaction semantic vector, the preset intimacy coefficient between the current family member and the deceased, and the memory weight bound to the corresponding historical memory entry. In step S4, during the process of dynamically generating the digital human's response text based on the obtained current interaction sentiment tags and the similarity scores of associated historical memories, the historical memory entry with the highest similarity score among the historical memory entries retrieved from the deceased's initial memory bank and associated with the current multimodal interaction information is used as the main matching content, and the description text of the historical memory entry with the highest similarity score is used as the basis; an sentiment prefix is ​​added to the beginning of the sentence based on the current interaction sentiment tags; the third person in the description text of the historical memory entry with the highest similarity score is replaced with the first person, and the synonyms of each term in the description text of the historical memory entry with the highest similarity score are randomly replaced to obtain the digital human's response text; if the length of the digital human's response text is less than the preset length, a description fragment of a historical memory entry related to a time or person is selected from the deceased's initial memory bank and spliced ​​to the end of the obtained digital human's response text to obtain a new digital human's response text.

7. A method for digital management of cemeteries with emotional digital human interaction as described in claim 1, characterized in that, In the process of S5 dynamically updating the historical memory entries in the initial memory bank of the deceased, the updated objects include the historical memory entries in the memory bank, the memory weight of each historical memory entry, and the language habit feature vector of the deceased. When updating historical memory entries in the memory bank, the generated interactive memory data is stored as a new historical memory entry in the deceased's initial memory bank; The calculation formula involved in updating the memory weight of each historical memory entry in the memory bank is as follows: in, This represents the memory weight after the i-th historical memory entry is updated; This represents the memory weight before the i-th historical memory entry was updated; This represents the activation coefficient of the i-th historical memory entry. If the cosine similarity between the interaction semantic vector bound to the i-th historical memory entry in the database and the current interaction semantic vector is greater than a preset value, then it is determined that... The value is equal to the cosine similarity between the interaction semantic vector bound to the i-th historical memory entry in the database and the current interaction semantic vector; otherwise, it is determined that... The value is 0; This indicates the preset learning rate; Indicates the preset forgetting rate; min{} represents a function that finds the minimum value; The update of the deceased's language habit feature vector is triggered when the accumulated interactive text word count reaches a preset word count threshold; the accumulated interactive text is used as the deceased's biographical text to construct the deceased's language habit feature vector, denoted as the incremental feature vector; the deceased's language habit feature vector is updated through a preset weighted formula, the calculation formulas involved are as follows: in, This represents the updated language habit feature vector of the deceased. This represents the language habit feature vector of the deceased before the update. Represents the incremental feature vector; This indicates that the weights are updated incrementally, and The value of the incremental update weight decays with the number of updates; The decay formula with each update is as follows: in, This represents the incremental update weight after the g-th update; d represents the initial incremental weight; d is the preset decay coefficient; when the Euclidean distance between the language habit feature vectors before and after the update is less than the preset constant in m consecutive updates, the update state of the deceased's language habit feature vector is determined to be the convergent state, the preset word count threshold is doubled as the new preset word count threshold, and half of the initial incremental weight is used as the new initial incremental weight; m is the preset constant.

8. A cemetery digital management system with emotional digital human interaction, characterized in that, include: The digital human interaction feedback model construction module constructs a digital human interaction feedback model for the deceased based on the deceased's photographs, audio recordings, and biographical texts. The deceased's memory bank construction module is used to combine pre-set records of deceased life trajectory events, pre-set deceased relationship graphs, and obtained language habit feature vectors to construct an initial memory bank for the deceased; The multimodal interaction information parsing module is used to obtain and parse the current multimodal interaction information of the deceased's family members to obtain the current interaction sentiment tag and the current interaction semantic vector based on the current multimodal interaction information; The digital human feedback information dynamic acquisition module is used to retrieve historical memory entries associated with the current multimodal interaction information in the deceased's initial memory bank based on the current interaction semantic vector, and calculate the similarity score of each historical memory; based on the obtained current interaction emotional tags and the similarity scores of associated historical memories, the module dynamically generates the digital human's response text and response voice. The deceased's memory bank dynamic update module is used to drive the static image model in the digital human interaction feedback model of the deceased to generate lip-sync animation and facial micro-expressions that are synchronized with the response voice of the digital human, based on the obtained current interaction emotion tags, and to provide feedback to the family members through the terminal device for interaction; and to dynamically update the deceased's initial memory bank based on the interaction data.

9. A cemetery digital management system with emotional digital human interaction as described in claim 8, characterized in that, The digital human interactive feedback model construction module includes a static image model construction unit, a speech synthesis model construction unit, a language habit feature vector construction unit, and an interactive feedback model generation unit. The static image model construction unit constructs a static image model of the deceased and mouth animation driving parameters based on the deceased's portrait. The speech synthesis model building unit constructs a speech synthesis model of the deceased based on the deceased's recording. The language habit feature vector construction unit constructs the language habit feature vector of the deceased based on the deceased's biographical text. The interactive feedback model generation unit is used to summarize the acquisition results of the static image model construction unit, the speech synthesis model construction unit, and the language habit feature vector construction unit to generate the digital human interactive feedback model of the deceased.

10. A cemetery digital management system with emotional digital human interaction as described in claim 8, characterized in that, The deceased's memory bank dynamic update module includes a model-driven unit and a memory bank dynamic update unit. The model driving unit is used to drive the static image model in the digital human interaction feedback model of the deceased to generate mouth animation and facial micro-expressions synchronized with the response voice of the digital human, based on the obtained current interaction emotion tag, and to provide feedback to the family members through the terminal device for interaction. The memory bank dynamic update unit is used to dynamically update the historical memory entries in the deceased's initial memory bank by taking the current multimodal interaction information of the deceased's family members during the interaction process, the dynamically generated digital human response text, the current interaction emotion tags, and the current interaction semantic vector as a set of interaction memory data.

Images