A method and system for training a general-knowledge large-scale artificial intelligence model based on digital human

Abstract

The present invention discloses a method and system for training a general-knowledge big model of artificial intelligence based on digital human, wherein the method includes collecting all multimodal sample data of all digital human, constructing text recognition key vectors and audio recognition key vectors according to all text modal data and audio modal data, calculating text mode deepening index and audio mode deepening index, constructing image recognition key vectors according to all image modal data, calculating image variation factor, obtaining image mode deepening index, and training a general-knowledge big model of artificial intelligence based on text mode deepening index, image mode deepening index, audio mode deepening index, text recognition key vectors, image recognition key vectors, and audio recognition key vectors. The embodiment of the present invention effectively integrates the text, voice, and image information of digital human, and improves the comprehensive ability and adaptability of the general-knowledge big model of artificial intelligence.

Description

Technical Field

[0001] The present invention relates to the field of large model technology, and in particular to a method and system for training a general large model of artificial intelligence based on digital humans. Background Art

[0002] Digital human is a virtual character created by computer graphics technology, artificial intelligence technology and related hardware and software equipment. It is designed to have human appearance, behavior and interaction capabilities and is used in various application scenarios. Artificial intelligence general knowledge big model, also known as universal big model or general meaning big model, is a special type of artificial intelligence model. Artificial intelligence general knowledge big model is usually trained with large-scale data sets, has a very large number of parameters, can handle various types of tasks, and has excellent performance in a wide range of fields.

[0003] When digital humans interact with users, they need to understand multiple types of input and respond in appropriate ways. Therefore, digital humans need to intelligently integrate multiple learning forms, but how to effectively integrate information from multiple modalities is a complex technical problem.

[0004] It can be seen that how to effectively integrate the information of multiple modes of digital humans to improve the comprehensive ability and adaptability of the artificial intelligence general knowledge model has become a technical problem that technical personnel in this field need to solve urgently. Summary of the invention

[0005] The present invention provides a training method and system for an artificial intelligence general knowledge large model based on digital human, so as to solve the technical problem of how to effectively integrate the information of multiple modes of digital human, so as to improve the comprehensive ability and adaptability of the artificial intelligence general knowledge large model.

[0006] In order to solve the above technical problems, an embodiment of the present invention provides a method and system for training a general large model of artificial intelligence based on digital humans.

[0007] Collecting all multimodal sample data of all digital humans, wherein the multimodal sample data includes text modality data, image modality data and audio modality data;

[0008] Constructing a text recognition key vector for each digital person according to all the text modality data, and obtaining a text mode deepening index for the current digital person based on the current text recognition key vector of the current digital person and the similarity between the current text recognition key vector and other text recognition key vectors;

[0009] Convert each of the audio modal data into text form to obtain each audio format conversion data, construct an audio recognition key vector for each of the digital persons based on all the audio format conversion data, and obtain the audio mode deepening index of the current digital person based on the current audio recognition key vector of the current digital person and the similarity between the current audio recognition key vector and other audio recognition key vectors;

[0010] constructing each image recognition key vector according to the grayscale distribution of all the image modality data, obtaining the image variation factor of each digital person based on the complexity of the texture structure of all the image modality data of each digital person and the difference of the grayscale distribution of each image modality data of each digital person, and calculating the image mode deepening index of the current digital person based on the image variation factor of the current digital person and the similarity between the current image recognition key vector and other image recognition key vectors;

[0011] The artificial intelligence general knowledge model is trained based on all the text pattern deepening indexes, the image pattern deepening indexes, the audio pattern deepening indexes, the text recognition key vectors, the image recognition key vectors, and the audio recognition key vectors.

[0012] As one of the preferred solutions, the step of constructing a text recognition key vector for each digital person based on all the text modality data includes:

[0013] Using all text modality data of each digital person as input of the TF-IDF algorithm, obtaining a first TF-IDF value of each first word, screening all the first TF-IDF values, and determining each first general knowledge training keyword of each digital person based on the screening result;

[0014] Sorting the first TF-IDF values ​​of the first general knowledge training keywords of each digital person, and determining the text recognition key vector of each digital person based on the sorting result;

[0015] The step of constructing an audio recognition key vector for each digital person according to all the audio format conversion data includes:

[0016] Using all the audio format conversion data of each digital person as the input of the TF-IDF algorithm, obtaining the second TF-IDF value of each second word, screening all the second TF-IDF values, and determining each second general knowledge training keyword of each digital person based on the screening result;

[0017] The second TF-IDF values ​​of the second general knowledge training keywords of each digital person are sorted, and the audio recognition key vector of each digital person is determined based on the sorting result.

[0018] As one of the preferred solutions, the method of obtaining the text mode deepening index of the current digital person based on the current text recognition key vector of the current digital person and the similarity between the current text recognition key vector and other text recognition key vectors includes:

[0019] Calculating the sequence similarity between any two of the text recognition key vectors, and taking the sum of the sequence similarities between the current text recognition key vector and other text recognition key vectors as the text pattern overlap factor of the current digital person;

[0020] The average value of all data in the text recognition key vector of each digital person is used as the text recognition key index of each digital person, and the text pattern deepening index of each digital person is calculated according to the linear relationship between the text recognition key index of each digital person and the text pattern overlap factor;

[0021] The step of obtaining the audio mode deepening index of the current digital person based on the current audio recognition key vector of the current digital person and the similarity between the current audio recognition key vector and other audio recognition key vectors includes:

[0022] Calculating the sequence similarity between any two of the audio recognition key vectors, and taking the sum of the sequence similarities between the current audio recognition key vector and other audio recognition key vectors as the audio mode overlap factor of the current digital human;

[0023] The average value of all data in the audio recognition key vector of each digital person is used as the audio recognition key index of each digital person. According to the linear relationship between the audio recognition key index of each digital person and the audio pattern overlap factor, the audio pattern deepening index of each digital person is calculated.

[0024] As one of the preferred solutions, the step of constructing each image recognition key vector according to the grayscale distribution of all the image modality data includes:

[0025] constructing a histogram of each of the image modality data;

[0026] Taking the histogram of each image modality data of each digital human as a row, obtaining the image general feature matrix of each digital human, and decomposing each image general feature matrix to obtain each image general singular matrix;

[0027] Part of the data on the diagonal of the image common singular matrix is ​​selected as each image common singular value of each digital person, and the feature vectors corresponding to all the image common singular values ​​of each digital person are concatenated as the image recognition key vector of each digital person.

[0028] As one of the preferred solutions, the image variation factor of each digital human being is obtained based on the complexity of the texture structure of all the image modality data of each digital human being and the difference of the gray level distribution of each image modality data of each digital human being, including:

[0029] Constructing a gray level co-occurrence matrix for each of the image modality data;

[0030] All the image modality data of each digital human are combined in pairs, the distance between the histograms of the two image modality data of each combination is recorded as the intra-group grayscale difference factor of each combination, and the sum of the entropy of the grayscale co-occurrence matrix of the two image modality data of each combination is recorded as the intra-group texture bi-phase complexity index of each combination;

[0031] The image variation factor of each digital human is calculated based on the linear relationship between the intra-group grayscale difference factor and the intra-group texture bi-phase complexity index of all the combinations.

[0032] As one of the preferred solutions, the image pattern deepening index of the current digital human being is calculated based on the image variation factor of the current digital human being and the similarity between the current image recognition key vector and other image recognition key vectors, including:

[0033] The sum of the sequence similarities between the current image recognition key vector and other image recognition key vectors is used as the image pattern overlap factor of the current digital human;

[0034] The image pattern deepening index of the current digital human is calculated based on the linear relationship between the image pattern overlapping factor and the image variation factor of the current digital human.

[0035] As one of the preferred solutions, the training of the artificial intelligence general knowledge big model based on all the text pattern deepening indexes, the image pattern deepening indexes, the audio pattern deepening indexes, the text recognition key vectors, the image recognition key vectors, and the audio recognition key vectors includes:

[0036] Processing all the text mode deepening indexes, the image mode deepening indexes and the audio mode deepening indexes to obtain training text learning weights, training image learning weights and training audio learning weights respectively;

[0037] All the text recognition key vectors, the image recognition key vectors and the audio recognition key vectors are used as inputs of the transformer neural network, the training text learning weights, the training image learning weights and the training audio learning weights are used as corresponding weights, and the transformer neural network is used to train the artificial intelligence general knowledge model.

[0038] As one of the preferred solutions, the processing of all the text mode deepening indexes, the image mode deepening indexes and the audio mode deepening indexes to obtain training text learning weights, training image learning weights and training audio learning weights respectively includes:

[0039] Acquire training text key factors, training image key factors and training audio key factors based on the correlation degree among all the text mode deepening indexes, the image mode deepening indexes and the audio mode deepening indexes;

[0040] According to the linear relationship between the training text key factor and all the text pattern deepening indexes, the training text learning weight to be optimized is calculated; according to the linear relationship between the training image key factor and all the image pattern deepening indexes, the training image learning weight to be optimized is calculated; according to the linear relationship between the training audio key factor and all the audio pattern deepening indexes, the training audio learning weight to be optimized is calculated;

[0041] The training text learning weight to be optimized, the training image learning weight to be optimized and the training audio learning weight to be optimized are normalized to obtain the training text learning weight, the training image learning weight and the training audio learning weight respectively.

[0042] As one of the preferred solutions, the step of obtaining the training text key factors, the training image key factors and the training audio key factors based on the correlation between all the text mode deepening indexes, the image mode deepening indexes and the audio mode deepening indexes includes:

[0043] The sorting results of all the text mode deepening indexes are used as a text deepening feature sequence; the sorting results of all the audio mode deepening indexes are used as an audio deepening feature sequence; the sorting results of all the image mode deepening indexes are used as an image deepening feature sequence;

[0044] The sum of the absolute values ​​of the correlation coefficients between the text deepening feature sequence and the image deepening feature sequence and the audio deepening feature sequence is used as the training text key factor; the sum of the absolute values ​​of the correlation coefficients between the image deepening feature sequence and the text deepening feature sequence and the audio deepening feature sequence is used as the training image key factor; the sum of the absolute values ​​of the correlation coefficients between the audio deepening feature sequence and the image deepening feature sequence and the text deepening feature sequence is used as the training audio key factor.

[0045] Yet another embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the above-mentioned method for training a general large model of artificial intelligence based on digital human is implemented.

[0046] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0047] (1) Considering the huge differences in the form, structure and dimension of data of different modalities, we first extract features from multimodal sample data, and construct the text recognition key vector, audio recognition key vector and image recognition key vector of each digital person based on all the text modal data, audio modal data and image modal data of each digital person. We convert the multimodal sample data into a unified format that is easy to process, thus improving the comprehensive ability and adaptability of the artificial intelligence general knowledge model.

[0048] (2) Analyze the amount of key information contained in the text modal data of digital humans and the similarity of patterns between the text modal data of different digital humans. Obtain the text pattern deepening index of each digital human based on the text recognition key vectors of all digital humans and the similarity between the text recognition key vectors of each digital human and other digital humans, so as to measure the degree of learning required for the text modal data of digital humans. Obtain the audio pattern deepening index of each digital human based on the audio recognition key vectors of all digital humans and the similarity between the audio recognition key vectors of different digital humans. Obtain the image variation factor of each digital human based on the complexity of the texture structure of all image modal data of each digital human and the difference in grayscale distribution. Calculate the image pattern deepening index of each digital human based on the similarity of the image recognition key vectors of all digital humans.

[0049] (3) Based on the correlation between the text mode deepening index, image mode deepening index and audio mode deepening index of all digital humans, the training text key factors, training image key factors and training audio key factors are obtained. Taking into account the internal characteristics of each modal data of the digital human, the similarity of the patterns between the same modal data and the correlation between multimodal sample data, the training text learning weights, training image learning weights and training audio learning weights are obtained based on the text mode deepening index, image mode deepening index and audio mode deepening index of all digital humans. The artificial intelligence general knowledge big model is trained, and different weights are set for the different modal data of the digital human to learn the different modes of the multimodal sample data of the digital human in a more balanced way, thereby further improving the comprehensive ability and adaptability of the artificial intelligence general knowledge big model. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of the process of training an artificial intelligence general knowledge model in one embodiment of the present invention;

[0051] Figure 2 It is a schematic diagram of obtaining a text mode deepening index in one embodiment of the present invention;

[0052] Figure 3 It is a schematic diagram of obtaining the image mode deepening index in one embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] In the description of this application, the terms "first", "second", "third", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", "third", etc. may explicitly or implicitly include one or more of the feature. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0055] In the description of the present application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. The terms "vertical", "horizontal", "left", "right", "upper", "lower" and similar expressions used herein are only for illustrative purposes, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0056] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by those skilled in the art. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood by specific circumstances.

[0057] An embodiment of the present invention provides a method for training a general-purpose large-scale artificial intelligence model based on a digital human. For details, see Figure 1 , Figure 1 Shown is a schematic diagram of the process of training an artificial intelligence general knowledge model in one embodiment of the present invention.

[0058] Step S1: Collect all multimodal sample data of all digital humans.

[0059] Digital humans usually refer to virtual characters created through computer technology, and multimodal sample data refers to data from different sources or different perceptual modalities, such as images, audio, text, etc. Training digital humans with multimodal sample data enables digital humans to provide a more natural and realistic interactive experience, thereby improving user satisfaction and loyalty.

[0060] Therefore, in order to train the general knowledge model of artificial intelligence, it is necessary to first collect a large amount of multimodal sample data of digital humans. It should be noted that multimodal sample data includes three types: text modal data, image modal data, and audio modal data.

[0061] Step S2: construct a text recognition key vector for each digital person based on all text modal data of each digital person, and obtain a text mode deepening index for each digital person based on the text recognition key vector of each digital person and the degree of similarity between the text recognition key vectors of each digital person and other digital persons.

[0062] Since data of different modalities have huge differences in form, structure and dimension, in order to effectively fuse the multimodal sample data of digital humans, it is necessary to first extract features from the multimodal sample data and convert the multimodal sample data into a unified and easy-to-process format. The importance of a word depends not only on its frequency of occurrence in a document, but also on its prevalence in the entire document collection. The TF-IDF algorithm is a commonly used statistical method for evaluating the importance of a word in a document or document collection.

[0063] First, feature extraction is performed on the text modal data of the digital person. Specifically, all text modal data of each digital person is used as the input of the TF-IDF algorithm to obtain the first TF-IDF value of each first word, and all first TF-IDF values ​​are screened. Based on the screening results, the first general training keywords of each digital person are determined, the first TF-IDF values ​​of the first general training keywords of each digital person are sorted, and the text recognition key vector of each digital person is determined based on the sorting results.

[0064] As an embodiment of the present application, the first TF-IDF values ​​of all the first words are arranged in descending order, the first m words are used as the first general training keywords of each digital person, and the first TF-IDF values ​​of all the first general training keywords of each digital person are arranged in descending order to obtain the text recognition key vector of each digital person.

[0065] It should be noted that the TF-IDF algorithm is a well-known technology and will not be described in detail in this application. The number m of the first general training keywords for each digital person is a preset value. In this embodiment, the number m of the general training keywords for each digital person is 10. As for the number of the first general training keywords for each digital person, as other implementation methods, the implementer can select it at will, and this application does not impose any special restrictions on this.

[0066] It should be further explained that the frequencies of occurrence of various modes in the different modal data of digital humans are different. In order to learn the different modes of the multimodal sample data of digital humans in a more balanced way, different weights need to be set for the different modal data of digital humans.

[0067] Specifically, the sequence similarity between the text recognition key vectors of any two digital persons is calculated, and the sum of the sequence similarities between each digital person and the text recognition key vectors of the remaining digital persons is taken as the text pattern overlap factor of each digital person.

[0068] It should be noted that the calculation of sequence similarity is a well-known technology and will not be elaborated in this application. In this embodiment, DTW distance is selected to calculate the sequence similarity between text recognition key vectors. For the calculation of sequence similarity, as other implementation methods, the implementer can choose them at will, and this application does not impose any special restrictions on this.

[0069] The average value of all data in the text recognition key vector of each digital person is taken as the text recognition key index of each digital person. According to the linear relationship between the text recognition key index of each digital person and the text pattern overlap factor, the text pattern deepening index of each digital person is calculated.

[0070] It should be noted that a linear relationship refers to a linear function relationship between two variables.

[0071] As an embodiment of the present application, the difference between the text recognition key index of each digital person and the text pattern overlap factor is used as the text pattern deepening index of each digital person.

[0072] It should be further explained that when the digital person's text recognition key index is larger and the text pattern overlap factor is smaller, it means that the digital person's text modal data contains more key information and more patterns appear, and the more it is necessary to strengthen the learning of the digital person's text modal data, and the larger the text pattern deepening index value.

[0073] In one embodiment of the present invention, the schematic diagram of obtaining the text mode deepening index is as follows: Figure 2 shown.

[0074] Step S3: construct an audio recognition key vector for each digital person based on all audio modal data of each digital person, and obtain the audio mode deepening index of each digital person based on the audio recognition key vector of each digital person and the similarity between each digital person and the audio recognition key vectors of other digital persons.

[0075] Speech recognition technology is used to convert each audio modality data into text form, and all audio format conversion data of each digital person is used as the input of the TF-IDF algorithm to obtain the second TF-IDF value of each second word. All the second TF-IDF values ​​are screened, and the second general training keywords of each digital person are determined based on the screening results. The second TF-IDF values ​​of the second general training keywords of each digital person are sorted, and the audio recognition key vector of each digital person is determined based on the sorting results.

[0076] As an embodiment of the present application, the second TF-IDF values ​​of all second words are arranged in descending order, the first t words are used as the second general training keywords of each digital person, and the second TF-IDF values ​​of all second general training keywords of each digital person are arranged in descending order to obtain the audio recognition key vector of each digital person.

[0077] It should be noted that the number t of the second general training keywords for each digital person is a preset value. In this embodiment, the number t of the general training keywords for each digital person is 10. For the number of second general training keywords for each digital person, as other implementation methods, the implementer can choose it at his own discretion, and this application does not impose any special restrictions on this.

[0078] The sequence similarity between the audio recognition key vectors of any two digital persons is calculated, and the sum of the sequence similarities between each digital person and the audio recognition key vectors of the remaining digital persons is taken as the audio pattern overlap factor of each digital person.

[0079] It should be noted that the calculation of sequence similarity is a well-known technology and will not be elaborated in this application. In this embodiment, DTW distance is selected to calculate the sequence similarity between audio recognition key vectors. For the calculation of sequence similarity, as other implementation methods, the implementer can choose it at will, and this application does not impose any special restrictions on this.

[0080] The average value of all data in the audio recognition key vector of each digital person is taken as the audio recognition key index of each digital person. According to the linear relationship between the audio recognition key index of each digital person and the audio pattern overlap factor, the audio pattern deepening index of each digital person is calculated.

[0081] As an embodiment of the present application, the difference between the audio recognition key index of each digital person and the audio pattern overlap factor is used as the audio pattern deepening index of each digital person.

[0082] It should be further explained that when the digital person's audio recognition key index is larger and the audio pattern overlap factor is smaller, it means that the digital person's audio modal data contains more key information and more patterns appear, and the more it is necessary to strengthen the learning of the digital person's audio modal data, and the larger the audio pattern deepening index value.

[0083] It should be noted that speech recognition technology is a well-known technology and will not be elaborated in detail in this application. The greater the audio mode deepening index of the digital human, the more it is necessary to strengthen the learning of the audio modality data of the digital human.

[0084] Step S4: construct the image recognition key vector of each digital person according to the grayscale distribution of all image modal data of each digital person, obtain the image variation factor of each digital person based on the complexity of the texture structure of all image modal data of each digital person and the difference of grayscale distribution, and calculate the image pattern deepening index of each digital person based on the image variation factor of each digital person and the similarity between each digital person and the image recognition key vectors of other digital persons.

[0085] It should be noted that the histogram of an image is a statistical chart that describes the grayscale distribution in an image. It shows the frequency or probability of each grayscale. In order to extract the grayscale distribution features of each digital human image modality data, a histogram of each image modality data is constructed.

[0086] The histogram of each image modality data of each digital person is taken as a row to obtain the image general feature matrix of each digital person, and each image general feature matrix is ​​decomposed to obtain each image general singular matrix, and part of the data on the diagonal of the image general singular matrix is ​​selected as each image general singular value of each digital person. The feature vectors corresponding to all the image general singular values ​​of each digital person are spliced ​​as the key vector for image recognition of each digital person.

[0087] It should be noted that matrix decomposition is a well-known technology and will not be described in detail in this application.

[0088] As an embodiment of the present application, a singular value decomposition algorithm is used to decompose the image common feature matrix of each digital person to obtain the image common singular matrix of each digital person, and the first s data on the diagonal of the image common singular matrix of each digital person are used as the image common singular values ​​of each digital person.

[0089] It should be noted that the construction of the histogram and the singular value decomposition algorithm are both well-known technologies and are not elaborated in this application. The number s of common singular values ​​of the image of each digital person is a preset value. In this embodiment, the number s of common singular values ​​of the image of each digital person is 3. The value of the number of common singular values ​​of the image of each digital person can be selected by the implementer as other implementation methods, and this application does not impose any special restrictions on this.

[0090] It should be further explained that the gray-level co-occurrence matrix describes the texture features of an image by calculating the probability of gray values ​​in the image appearing in pairs in a specific direction. The entropy of the gray-level co-occurrence matrix measures the complexity of the texture structure in the image. In order to analyze the complexity of the texture structure of all digital human image modal data and the differences in grayscale distribution, the gray-level co-occurrence matrix of each image modal data is constructed.

[0091] Furthermore, all image modality data of each digital human are combined in pairs, the distance between the histograms of the two image modality data of each combination is recorded as the intra-group grayscale difference factor of each combination, the sum of the entropies of the grayscale co-occurrence matrices of the two image modality data of each combination is recorded as the intra-group texture biphase complexity index of each combination, and the image variation factor of each digital human is calculated based on the linear relationship between the intra-group grayscale difference factors of all combinations and the intra-group texture biphase complexity index.

[0092] As an embodiment of the present application, the sum of the grayscale difference factor within the group and the bi-phase complexity index of the texture within the group of all combinations of each digital human is used as the image variation factor of each digital human.

[0093] It should be further explained that the construction of the grayscale co-occurrence matrix is ​​a well-known technology and will not be elaborated in this application. When the grayscale difference factor within each group of the image modality data of the digital human is larger and the texture bi-phase complexity index within the group is larger, it means that the texture structure of the image modality data of the digital human is more complex, the grayscale distribution difference between each image modality data is larger, and the image variation factor value of the digital human is larger. When training the multimodal sample data of the digital human, the learning of the image modality data of the digital human should be strengthened.

[0094] The sum of the sequence similarities between the image recognition key vectors of each digital person and the other digital persons is taken as the image pattern overlap factor of each digital person. According to the linear relationship between the image pattern overlap factor and the image variation factor of each digital person, the image pattern deepening index of each digital person is calculated.

[0095] As an embodiment of the present application, the difference between the image variation factor of each digital human and the image pattern overlap factor is used as the image pattern deepening index of each digital human.

[0096] It should be further explained that when the image variation factor of the digital human is larger and the image pattern overlap factor is smaller, it means that the information contained in the image modal data of the digital human is more complex and the more patterns appear, the more it is necessary to strengthen the learning of the image modal data of the digital human, and the larger the image pattern deepening index value.

[0097] In one embodiment of the present invention, the schematic diagram of obtaining the image mode deepening index is as follows: Figure 3 shown.

[0098] Step S5: Train the artificial intelligence general knowledge model based on the text pattern deepening index, image pattern deepening index, audio pattern deepening index, text recognition key vector, image recognition key vector, and audio recognition key vector of all digital humans.

[0099] It should be noted that in order to create a more realistic, immersive interactive experience and a more realistic, natural and interactive virtual experience, there is usually a certain correlation between the multimodal sample data of digital humans.

[0100] In order to analyze the correlation between the multimodal sample data of digital humans, all digital humans are randomly sorted to obtain the number of each digital human, and the text mode deepening indexes of all digital humans are arranged in ascending order of the number to obtain the text deepening feature sequence; the image mode deepening indexes of all digital humans are arranged in ascending order of the number to obtain the image deepening feature sequence, and the audio mode deepening indexes of all digital humans are arranged in ascending order of the number to obtain the audio deepening feature sequence.

[0101] Furthermore, the sum of the absolute values ​​of the correlation coefficients between the text deepening feature sequence and the image deepening feature sequence and the audio deepening feature sequence is used as the training text key factor; the sum of the absolute values ​​of the correlation coefficients between the image deepening feature sequence and the text deepening feature sequence and the audio deepening feature sequence is used as the training image key factor; the sum of the absolute values ​​of the correlation coefficients between the audio deepening feature sequence and the image deepening feature sequence and the text deepening feature sequence is used as the training audio key factor.

[0102] It should be noted that the calculation of the correlation coefficient between random sorting and sequences is a well-known technology and will not be elaborated in this application. The larger the correlation coefficient between a certain modality data and the other modality data, the more likely the modality data is to be a key factor in training a digital human. When training the multimodal sample data of a digital human, the more the learning of the modality data should be strengthened.

[0103] According to the linear relationship between the key factors of the training text and the deepening indexes of all text patterns, the learning weights of the training text to be optimized are calculated; according to the linear relationship between the key factors of the training image and the deepening indexes of all image patterns, the learning weights of the training image to be optimized are calculated; according to the linear relationship between the key factors of the training audio and the deepening indexes of all audio patterns, the learning weights of the training audio to be optimized are calculated.

[0104] As an embodiment of the present application, the average value of the text mode deepening index of all digital humans plus the training text key factor is used as the training text learning weight to be optimized, the average value of the image mode deepening index of all digital humans plus the training image key factor is used as the training image learning weight to be optimized, and the average value of the audio mode deepening index of all digital humans plus the training audio key factor is used as the training audio learning weight to be optimized.

[0105] The training text learning weight to be optimized, the training image learning weight to be optimized and the training audio learning weight to be optimized are normalized to obtain the training text learning weight, the training image learning weight and the training audio learning weight respectively.

[0106] It should be noted that, when the training text key factor is larger and the text mode deepening index of each digital human is larger, it means that the text modal data is more likely to be the key factor for training the digital human, the learning of text modal data should be strengthened, and the training text learning weight value is larger; when the training image key factor is larger and the image mode deepening index of each digital human is larger, it means that the image modal data is more likely to be the key factor for training the digital human, the learning of image modal data should be strengthened, and the training image learning weight value is larger; when the training audio key factor is larger and the audio mode deepening index of each digital human is larger, it means that the audio modal data is more likely to be the key factor for training the digital human, the learning of audio modal data should be strengthened, and the training audio learning weight value is larger.

[0107] Furthermore, the text recognition key vectors, image recognition key vectors and audio recognition key vectors of all digital humans are used as inputs of the transformer neural network, and the training text learning weights, training image learning weights and training audio learning weights are used as corresponding weights. The transformer neural network is used to train the artificial intelligence general knowledge model.

[0108] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0109] Accordingly, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the steps in the training method of a general large model of artificial intelligence based on digital human as described in the above embodiment, for example Figure 1 Steps S1 to S5 described in .

[0110] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for training a general-purpose large-scale artificial intelligence model based on digital human, characterized in that: include: Collecting all multimodal sample data of all digital humans, wherein the multimodal sample data includes text modality data, image modality data and audio modality data; Constructing a text recognition key vector for each digital person according to all the text modality data, and obtaining a text mode deepening index for the current digital person based on the current text recognition key vector of the current digital person and the similarity between the current text recognition key vector and other text recognition key vectors; Convert each of the audio modal data into text form to obtain each audio format conversion data, construct an audio recognition key vector for each of the digital persons based on all the audio format conversion data, and obtain the audio mode deepening index of the current digital person based on the current audio recognition key vector of the current digital person and the similarity between the current audio recognition key vector and other audio recognition key vectors; constructing each image recognition key vector according to the grayscale distribution of all the image modality data, obtaining the image variation factor of each digital person based on the complexity of the texture structure of all the image modality data of each digital person and the difference of the grayscale distribution of each image modality data of each digital person, and calculating the image mode deepening index of the current digital person based on the image variation factor of the current digital person and the similarity between the current image recognition key vector and other image recognition key vectors; Based on all the text pattern deepening indexes, the image pattern deepening indexes, the audio pattern deepening indexes, the text recognition key vectors, the image recognition key vectors, and the audio recognition key vectors, train an artificial intelligence general knowledge model; The obtaining of the text mode deepening index of the current digital person based on the current text recognition key vector of the current digital person and the similarity between the current text recognition key vector and other text recognition key vectors includes: Calculating the sequence similarity between any two of the text recognition key vectors, and taking the sum of the sequence similarities between the current text recognition key vector and other text recognition key vectors as the text pattern overlap factor of the current digital person; The average value of all data in the text recognition key vector of each digital person is used as the text recognition key index of each digital person, and the text pattern deepening index of each digital person is calculated according to the linear relationship between the text recognition key index of each digital person and the text pattern overlap factor; The step of obtaining the audio mode deepening index of the current digital person based on the current audio recognition key vector of the current digital person and the similarity between the current audio recognition key vector and other audio recognition key vectors includes: Calculating the sequence similarity between any two of the audio recognition key vectors, and taking the sum of the sequence similarities between the current audio recognition key vector and other audio recognition key vectors as the audio mode overlap factor of the current digital human; The average value of all data in the audio recognition key vector of each digital person is used as the audio recognition key index of each digital person, and the audio pattern deepening index of each digital person is calculated based on the linear relationship between the audio recognition key index of each digital person and the audio pattern overlap factor; The step of constructing each image recognition key vector according to the grayscale distribution of all the image modality data comprises: constructing a histogram of each of the image modality data; Taking the histogram of each image modality data of each digital human as a row, obtaining the image general feature matrix of each digital human, and decomposing each image general feature matrix to obtain each image general singular matrix; Selecting part of the data on the diagonal of the image-general singular matrix as the image-general singular values ​​of each digital person, and concatenating the feature vectors corresponding to all the image-general singular values ​​of each digital person as the image recognition key vector of each digital person; The step of obtaining the image variation factor of each digital person based on the complexity of the texture structure of all the image modality data of each digital person and the difference in the grayscale distribution of each of the image modality data of each digital person comprises: Constructing a gray level co-occurrence matrix for each of the image modality data; All the image modality data of each digital human are combined in pairs, the distance between the histograms of the two image modality data of each combination is recorded as the intra-group grayscale difference factor of each combination, and the sum of the entropy of the grayscale co-occurrence matrix of the two image modality data of each combination is recorded as the intra-group texture bi-phase complexity index of each combination; The image variation factor of each digital human is calculated based on the linear relationship between the grayscale difference factor within the group and the texture bi-phase complexity index within the group of all the combinations; The calculating of the image pattern deepening index of the current digital human based on the image variation factor of the current digital human and the similarity between the current image recognition key vector and other image recognition key vectors includes: The sum of the sequence similarities between the current image recognition key vector and other image recognition key vectors is used as the image pattern overlap factor of the current digital human; The image pattern deepening index of the current digital human is calculated based on the linear relationship between the image pattern overlapping factor and the image variation factor of the current digital human.

2. The method for training a general-knowledge large-scale model of artificial intelligence based on digital human according to claim 1 is characterized in that: The step of constructing a text recognition key vector for each digital person according to all the text modality data includes: Using all text modality data of each digital person as input of the TF-IDF algorithm, obtaining a first TF-IDF value of each first word, screening all the first TF-IDF values, and determining each first general knowledge training keyword of each digital person based on the screening result; Sorting the first TF-IDF values ​​of the first general knowledge training keywords of each digital person, and determining the text recognition key vector of each digital person based on the sorting result; The step of constructing an audio recognition key vector for each digital person according to all the audio format conversion data includes: Using all the audio format conversion data of each digital person as the input of the TF-IDF algorithm, obtaining the second TF-IDF value of each second word, screening all the second TF-IDF values, and determining each second general knowledge training keyword of each digital person based on the screening result; The second TF-IDF values ​​of the second general knowledge training keywords of each digital person are sorted, and the audio recognition key vector of each digital person is determined based on the sorting result.

3. The method for training a general large-scale model of artificial intelligence based on digital human according to claim 1 is characterized in that: The method of training the artificial intelligence general knowledge big model based on all the text pattern deepening indexes, the image pattern deepening indexes, the audio pattern deepening indexes, the text recognition key vectors, the image recognition key vectors, and the audio recognition key vectors includes: Processing all the text mode deepening indexes, the image mode deepening indexes and the audio mode deepening indexes to obtain training text learning weights, training image learning weights and training audio learning weights respectively; All the text recognition key vectors, the image recognition key vectors and the audio recognition key vectors are used as inputs of the transformer neural network, the training text learning weights, the training image learning weights and the training audio learning weights are used as corresponding weights, and the transformer neural network is used to train the artificial intelligence general knowledge model.

4. The method for training a general-knowledge large-scale model of artificial intelligence based on digital human according to claim 3 is characterized in that: The processing of all the text mode deepening indexes, the image mode deepening indexes and the audio mode deepening indexes to obtain training text learning weights, training image learning weights and training audio learning weights respectively includes: Acquire training text key factors, training image key factors and training audio key factors based on the correlation degree among all the text mode deepening indexes, the image mode deepening indexes and the audio mode deepening indexes; According to the linear relationship between the training text key factor and all the text pattern deepening indexes, the training text learning weight to be optimized is calculated; according to the linear relationship between the training image key factor and all the image pattern deepening indexes, the training image learning weight to be optimized is calculated; according to the linear relationship between the training audio key factor and all the audio pattern deepening indexes, the training audio learning weight to be optimized is calculated; The training text learning weight to be optimized, the training image learning weight to be optimized and the training audio learning weight to be optimized are normalized to obtain the training text learning weight, the training image learning weight and the training audio learning weight respectively.

5. The method for training a general large-scale model of artificial intelligence based on digital human according to claim 4 is characterized in that: The step of acquiring the training text key factor, the training image key factor and the training audio key factor based on the correlation between all the text mode deepening indexes, the image mode deepening indexes and the audio mode deepening indexes comprises: The sorting results of all the text mode deepening indexes are used as a text deepening feature sequence; the sorting results of all the audio mode deepening indexes are used as an audio deepening feature sequence; the sorting results of all the image mode deepening indexes are used as an image deepening feature sequence; The sum of the absolute values ​​of the correlation coefficients between the text deepening feature sequence and the image deepening feature sequence and the audio deepening feature sequence is used as the training text key factor; the sum of the absolute values ​​of the correlation coefficients between the image deepening feature sequence and the text deepening feature sequence and the audio deepening feature sequence is used as the training image key factor; the sum of the absolute values ​​of the correlation coefficients between the audio deepening feature sequence and the image deepening feature sequence and the text deepening feature sequence is used as the training audio key factor.

6. An artificial intelligence general knowledge large model training system based on digital human, characterized by: It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements the artificial intelligence general knowledge large model training method based on digital human as described in any one of claims 1 to 5.

Images