Animation video generation method, device, equipment and storage medium

Abstract

The present invention relates to artificial intelligence and provides a method, apparatus, device, and storage medium for generating an animated video. Upon receiving a video generation request, the method can obtain text information based on the video generation request; input the text information into a pre-trained video generation model to obtain an initial video; identify human feature points in each frame of the initial video; generate posture information of the user in each frame based on the human feature points; if the posture information is a preset posture, adjust the posture information based on the human feature points to obtain a second video; analyze the text information based on a pre-trained audio generation model to obtain audio information; and generate an animated video based on the second video and the audio information. The present invention can improve the generation efficiency and quality of animated videos. Furthermore, the present invention also relates to blockchain technology, and the animated video can be stored in a blockchain.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an animation video generation method, device, equipment and storage medium. Background Art

[0002] In educational settings for children, animated video instruction can stimulate students' interest and enthusiasm for learning. With the development of artificial intelligence, animated video instruction has also evolved. However, the current animation video production process involves steps such as storyboard writing, storyboard design, live-action filming, illustration creation, animation production, and post-production editing. This results in inefficient production of complete animated videos. Furthermore, due to the varying perspectives of different video production users, the quality of the generated videos cannot be guaranteed. Summary of the Invention

[0003] In view of the above, it is necessary to provide an animation video generation method, device, equipment and storage medium that can improve the generation efficiency and generation quality of animation videos.

[0004] In one aspect, the present invention provides a method for generating an animation video, the method comprising:

[0005] When a video generation request is received, obtaining text information according to the video generation request;

[0006] Inputting the text information into a pre-trained video generation model to obtain an initial video;

[0007] Identifying human feature points in each frame of the initial video;

[0008] Generating user posture information in each frame of image according to the human body feature points;

[0009] If the posture information is a preset posture, adjusting the posture information according to the human body feature points to obtain a second video;

[0010] Analyze the text information based on a pre-trained audio generation model to obtain audio information;

[0011] An animation video is generated according to the second video and the audio information.

[0012] According to a preferred embodiment of the present invention, before inputting the text information into a pre-trained video generation model to obtain an initial video, the method further includes:

[0013] Acquire multiple video training samples, each video training sample including a training video and a training text corresponding to the training video;

[0014] Constructing a learner, wherein the learner includes an encoding layer and a decoding layer;

[0015] Performing text encoding processing on the training text to obtain a text vector;

[0016] Analyze the text vector based on the coding layer to obtain feature information of the training text;

[0017] Analyze the feature information based on the decoding layer to obtain an output vector;

[0018] Performing mapping processing on the training video based on a preset mapping table to obtain an image vector of the training video;

[0019] Calculating the similarity between the text vector and the output vector to obtain a first similarity, and calculating the similarity between the text vector and the image vector to obtain a second similarity;

[0020] calculating a ratio of the first similarity to the second similarity to obtain a learning index of the learner;

[0021] The network parameters in the learner are adjusted until the learning index no longer increases, thereby obtaining the video generation model.

[0022] According to a preferred embodiment of the present invention, identifying the human body feature points of each frame image in the initial video includes:

[0023] Detect each frame of image based on a preset detector to obtain the human body area in each frame of image;

[0024] Performing grayscale processing on the human body region to obtain a plurality of pixel points in the human body region and a pixel grayscale value corresponding to each pixel point;

[0025] Calculating the pixel difference between each pixel and the preset feature point according to the pixel grayscale value and the feature grayscale value of the preset feature point;

[0026] Determine the pixel points whose pixel difference is less than a preset threshold as initial feature points;

[0027] Constructing a coordinate system based on each frame of image, and obtaining initial coordinate information of the initial feature points on each frame of image;

[0028] The human body feature points are screened out from the initial feature points according to the initial coordinate information.

[0029] According to a preferred embodiment of the present invention, filtering out the human body feature points from the initial feature points according to the initial coordinate information includes:

[0030] For any initial feature point, calculating a feature distance between the initial feature point and a target feature point according to the initial coordinate information, wherein the target feature point refers to the remaining feature points among the initial feature points except the initial feature point;

[0031] Determine the feature distance with the minimum value as the target distance, and determine the target feature point corresponding to the target distance as the adjacent feature point of any initial feature point;

[0032] Performing normal distribution processing on the target distance to obtain a probability value of the target distance;

[0033] The initial feature point corresponding to the target distance whose probability value is greater than the preset probability value is determined as the human body feature point.

[0034] According to a preferred embodiment of the present invention, generating the user's posture information in each frame of image according to the human body feature points includes:

[0035] Acquire the coordinate information of the human body feature points as human body coordinate information according to the coordinate system;

[0036] Acquire any two adjacent feature points from the human body feature points as a feature point pair;

[0037] Calculating the Euler angle of each feature point pair according to the human body coordinate information and the abscissa axis in the coordinate system;

[0038] An average value of the Euler angles is calculated to obtain an angle mean value, and preset posture information corresponding to the angle mean value is determined as the posture information.

[0039] According to a preferred embodiment of the present invention, the audio generation model includes an emotion recognition network layer and a speech conversion network layer, and the audio information obtained by analyzing the text information based on the pre-trained audio generation model includes:

[0040] Analyze the text information based on the emotion recognition network layer to obtain the text emotion of the text information;

[0041] Acquire emotional speech features of the text emotion from a speech feature library;

[0042] Processing the text information based on the speech conversion network layer to obtain speech information, and obtaining text speech features from the speech information;

[0043] The text speech feature and the emotional speech feature are subjected to audio mixing processing to obtain the audio information.

[0044] According to a preferred embodiment of the present invention, generating an animation video based on the second video and the audio information includes:

[0045] Counting the duration of the second video to obtain a first duration;

[0046] Counting the duration of the audio information to obtain a second duration;

[0047] If the first duration is not equal to the second duration, obtaining information with the largest duration from the second video and the audio information as information to be processed;

[0048] Compressing the information to be processed until the lengths of the processed second video and the processed audio information are equal;

[0049] The processed second video and the processed audio information are combined to obtain the animation video.

[0050] On the other hand, the present invention further provides an animation video generating device, the animation video generating device comprising:

[0051] an acquiring unit, configured to acquire text information according to a video generation request when a video generation request is received;

[0052] An input unit, configured to input the text information into a pre-trained video generation model to obtain an initial video;

[0053] A recognition unit, configured to recognize human feature points in each frame of the initial video;

[0054] A generating unit, configured to generate posture information of the user in each frame of image according to the human body feature points;

[0055] an adjusting unit, configured to adjust the posture information according to the human body feature points to obtain a second video if the posture information is a preset posture;

[0056] An analysis unit, configured to analyze the text information based on a pre-trained audio generation model to obtain audio information;

[0057] The generating unit is configured to generate an animation video based on the second video and the audio information.

[0058] On the other hand, the present invention further provides an electronic device, comprising:

[0059] a memory storing computer-readable instructions; and

[0060] The processor executes the computer-readable instructions stored in the memory to implement the animation video generation method.

[0061] On the other hand, the present invention further proposes a computer-readable storage medium, in which computer-readable instructions are stored. The computer-readable instructions are executed by a processor in an electronic device to implement the animation video generation method.

[0062] It can be seen from the above technical solution that the present invention analyzes the text information through the video generation model, and can quickly generate the initial video, thereby improving the generation efficiency of the animation video. Furthermore, by identifying the human body feature points, the posture information of the user in each frame of the image can be accurately determined, and then when the posture information is a preset posture, the posture information is adjusted, which can avoid the presence of bad posture information such as the preset posture in the second video. Since good posture information can play a certain educational role for users, the quality of the second video can be improved by avoiding the presence of bad posture information such as the preset posture in the second video. The audio generation model can accurately generate audio information corresponding to the text information, and the generation quality of the animation video can be improved based on the audio information and the second video. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flow chart of a preferred embodiment of the animation video generation method of the present invention.

[0064] Figure 2 It is a functional module diagram of a preferred embodiment of the animation video generating device of the present invention.

[0065] Figure 3 It is a structural diagram of an electronic device according to a preferred embodiment of the present invention for realizing the method for generating animation video. DETAILED DESCRIPTION

[0066] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] like Figure 1 FIG. 1 is a flow chart of a preferred embodiment of the method for generating an animated video according to the present invention. The order of the steps in the flow chart may be changed and some steps may be omitted according to different requirements.

[0068] The animation video generation method can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0069] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0070] The method for generating an animated video is applied to the field of smart education, thereby promoting the development of smart cities. The method for generating an animated video is applied to one or more electronic devices, wherein the electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored computer-readable instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0071] The electronic device can be any electronic product that can interact with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0072] The electronic device may include a network device and / or a user device, wherein the network device includes, but is not limited to, a single network electronic device, a group of electronic devices consisting of multiple network electronic devices, or a cloud based on cloud computing consisting of a large number of hosts or network electronic devices.

[0073] The network where the electronic device is located includes, but is not limited to: the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0074] S10: When a video generation request is received, text information is acquired according to the video generation request.

[0075] In at least one embodiment of the present invention, the triggering user of the video generation request is different depending on the application scenario of the video generation request. For example, if the application scenario of the video generation request is in the field of education, the triggering user of the video generation request may be a teacher, etc.

[0076] The video generation request may include, but is not limited to: text path, preset tags, etc.

[0077] The text information refers to text information that needs to be converted into a video. For example, the text information may be a teacher's lecture notes.

[0078] In at least one embodiment of the present invention, the electronic device obtaining text information according to the video generation request includes:

[0079] Parsing the video generation request message to obtain data information carried in the message;

[0080] extracting the text path from the data information according to the preset tag;

[0081] The text information is obtained from the text path.

[0082] The preset tag is a tag used to indicate a path, for example, the preset tag may be a storage location.

[0083] The text path can be accurately extracted through the preset tag, so that the text information can be accurately obtained, which is conducive to the generation of the corresponding animation video.

[0084] S11, inputting the text information into a pre-trained video generation model to obtain an initial video.

[0085] In at least one embodiment of the present invention, the video generation model is a model capable of converting text into video. The video generation model includes an encoding layer, a decoding layer, and a preset mapping table. The preset mapping table stores a mapping relationship between pixel values and vectors.

[0086] The initial video refers to a video generated after the video generation model analyzes the text information. The initial video does not contain voice information.

[0087] In at least one embodiment of the present invention, before inputting the text information into a pre-trained video generation model to obtain an initial video, the method further includes:

[0088] Acquire multiple video training samples, each video training sample including a training video and a training text corresponding to the training video;

[0089] Constructing a learner, wherein the learner includes an encoding layer and a decoding layer;

[0090] Performing text encoding processing on the training text to obtain a text vector;

[0091] Analyze the text vector based on the coding layer to obtain feature information of the training text;

[0092] Analyze the feature information based on the decoding layer to obtain an output vector;

[0093] Performing mapping processing on the training video based on a preset mapping table to obtain an image vector of the training video;

[0094] Calculating the similarity between the text vector and the output vector to obtain a first similarity, and calculating the similarity between the text vector and the image vector to obtain a second similarity;

[0095] calculating a ratio of the first similarity to the second similarity to obtain a learning index of the learner;

[0096] The network parameters in the learner are adjusted until the learning index no longer increases, thereby obtaining the video generation model.

[0097] The text vector is used to represent the training text.

[0098] The learning indicator is used to evaluate the accuracy of the learner.

[0099] The network parameters include pre-set parameters in the encoding layer and the decoding layer. For example, if the encoding layer includes a convolution layer, the network parameters may be the size of the convolution kernel in the convolution layer.

[0100] By generating the learning indicator through the similarity between the training text and the predicted video and the similarity between the training text and the training video, and then adjusting the network parameters according to the learning indicator, the video generation model's ability to represent text information can be improved, thereby improving the accuracy of video generation.

[0101] In at least one embodiment of the present invention, the manner in which the electronic device analyzes the text information based on the video generation model is similar to the manner in which the electronic device analyzes the training text based on the learner, and the present invention will not elaborate on this.

[0102] S12: Identify human feature points in each frame of the initial video.

[0103] In at least one embodiment of the present invention, the human body feature points include, but are not limited to: key feature points of the face, such as pupil centers, etc.; hand joints and bone joints, etc.

[0104] In at least one embodiment of the present invention, the electronic device identifying the human feature points of each frame of the initial video includes:

[0105] Detect each frame of image based on a preset detector to obtain the human body area in each frame of image;

[0106] Performing grayscale processing on the human body region to obtain a plurality of pixel points in the human body region and a pixel grayscale value corresponding to each pixel point;

[0107] Calculating the pixel difference between each pixel and the preset feature point according to the pixel grayscale value and the feature grayscale value of the preset feature point;

[0108] Determine the pixel points whose pixel difference is less than a preset threshold as initial feature points;

[0109] Constructing a coordinate system based on each frame of image, and obtaining initial coordinate information of the initial feature points on each frame of image;

[0110] The human body feature points are screened out from the initial feature points according to the initial coordinate information.

[0111] The preset detector may be used to identify person information in an image.

[0112] The preset feature points include hand joints and bone joints, etc. The feature grayscale value can be determined based on pixel information corresponding to the preset feature points of multiple preset users.

[0113] The preset threshold can be set according to requirements.

[0114] The coordinate system includes a horizontal axis and a vertical axis.

[0115] By detecting each frame of image through the preset detector, not only can the interference of background information in each frame of image on the human feature points be eliminated, thereby improving the recognition accuracy of the human feature points, but also the number of pixel points analyzed can be reduced, thereby improving the recognition efficiency of the human feature points, and then through the analysis of the pixel grayscale value and the feature grayscale value, the initial feature point can be quickly determined, and then the determination accuracy of the human feature point can be improved according to the initial coordinate information of the initial feature point.

[0116] In at least one embodiment of the present invention, the electronic device screening out the human body feature points from the initial feature points according to the initial coordinate information includes:

[0117] For any initial feature point, calculating a feature distance between the initial feature point and a target feature point according to the initial coordinate information, wherein the target feature point refers to the remaining feature points among the initial feature points except the initial feature point;

[0118] Determine the feature distance with the minimum value as the target distance, and determine the target feature point corresponding to the target distance as the adjacent feature point of any initial feature point;

[0119] Performing normal distribution processing on the target distance to obtain a probability value of the target distance;

[0120] The initial feature point corresponding to the target distance whose probability value is greater than the preset probability value is determined as the human body feature point.

[0121] The preset probability value may be set according to requirements, for example, the preset probability value may be 99.44%.

[0122] By analyzing the characteristic distance, the adjacent characteristic points of any initial characteristic point can be quickly determined. By performing normal distribution processing on the target distance and further analyzing the probability value of the target distance, the human body characteristic points can be accurately screened out from the initial characteristic points.

[0123] S13: Generate user posture information in each frame of image according to the human body feature points.

[0124] In at least one embodiment of the present invention, the posture information refers to the posture of the user in each frame of image. For example, the posture information may be lowering the head, or looking up the head.

[0125] In at least one embodiment of the present invention, the electronic device generating the user's posture information in each frame of image according to the human body feature points includes:

[0126] Acquire the coordinate information of the human body feature points as human body coordinate information according to the coordinate system;

[0127] Acquire any two adjacent feature points from the human body feature points as a feature point pair;

[0128] Calculating the Euler angle of each feature point pair according to the human body coordinate information and the abscissa axis in the coordinate system;

[0129] An average value of the Euler angles is calculated to obtain an angle mean value, and preset posture information corresponding to the angle mean value is determined as the posture information.

[0130] Among them, the feature point pair refers to any two adjacent feature points obtained from the human body feature points. Further, the any two adjacent feature points refer to human body feature points with adjacent feature distances, for example, human body feature point A, human body feature point B, human body feature point C, and human body feature point D. The feature distance between the human body feature point A and the human body feature point B is 5, the feature distance between the human body feature point A and the human body feature point C is 2, and the feature distance between the human body feature point A and the human body feature point D is 3, then the human body feature point C is the adjacent feature point of the human body feature point A.

[0131] By calculating the Euler angles of any two adjacent feature points, it is possible to avoid interference of human feature points that are far apart on the posture information, thereby improving the accuracy of determining the posture information.

[0132] Specifically, the posture information can be determined based on a mapping table between angles and preset posture information, wherein the preset posture information can be marked by a user.

[0133] S14: If the posture information is a preset posture, adjust the posture information according to the human body feature points to obtain a second video.

[0134] In at least one embodiment of the present invention, the preset posture may include, but is not limited to: lowering the head, raising the head, and other bad postures.

[0135] The user posture of each frame image in the second video is not the preset posture.

[0136] In at least one embodiment of the present invention, the electronic device adjusting the posture information according to the human body feature points to obtain the second video includes:

[0137] Get the pose angle of the standard pose from the pose mapping table;

[0138] comparing the angle mean with the posture angle;

[0139] If the angle mean is greater than the posture angle, comparing the Euler angle with the angle mean;

[0140] Obtaining feature point pairs corresponding to Euler angles having values greater than the angle mean as feature points to be processed;

[0141] The position of the feature point to be processed in the image is adjusted until the adjusted posture information is not a preset posture, thereby obtaining the second video.

[0142] The posture mapping table stores mappings between multiple preset posture information and angles, including the standard posture and undesirable postures such as pitch. The multiple preset posture information in the posture mapping table can be annotated by the user. The angles in the posture mapping table are calculated similarly to the mean angle value in each frame of the image, and this description will not be repeated herein.

[0143] By comparing the angle mean with the posture angle, and comparing the Euler angle with the angle mean, the human body feature points that affect the posture information can be quickly determined and then adjusted to improve the quality of the second video.

[0144] S15, analyzing the text information based on a pre-trained audio generation model to obtain audio information.

[0145] In at least one embodiment of the present invention, the audio generation model is used to convert the text information into speech.

[0146] The audio information refers to the voice corresponding to the text information.

[0147] In at least one embodiment of the present invention, the audio generation model includes an emotion recognition network layer and a speech conversion network layer, and the electronic device analyzes the text information based on the pre-trained audio generation model to obtain audio information including:

[0148] Analyze the text information based on the emotion recognition network layer to obtain the text emotion of the text information;

[0149] Acquire emotional speech features of the text emotion from a speech feature library;

[0150] Processing the text information based on the speech conversion network layer to obtain speech information, and obtaining text speech features from the speech information;

[0151] Perform audio mixing processing on the text speech features and the emotional speech features to obtain the audio information.

[0152] The emotion recognition network layer is used to analyze the emotion corresponding to the text, which can be happiness, sadness, etc.

[0153] The speech conversion network layer is used to convert text into speech.

[0154] By performing audio mixing processing on the text speech features and the emotional speech features, the text emotion is included in the audio information, thereby improving the interest of the audio information.

[0155] S16: Generate an animation video based on the second video and the audio information.

[0156] In at least one embodiment of the present invention, the animation video refers to a video including the audio information and the second video.

[0157] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned animation video, the above-mentioned animation video can also be stored in a blockchain node.

[0158] In at least one embodiment of the present invention, the electronic device generating an animation video according to the second video and the audio information includes:

[0159] Counting the duration of the second video to obtain a first duration;

[0160] Counting the duration of the audio information to obtain a second duration;

[0161] If the first duration is not equal to the second duration, obtaining information with the largest duration from the second video and the audio information as information to be processed;

[0162] Compressing the information to be processed until the lengths of the processed second video and the processed audio information are equal;

[0163] The processed second video and the processed audio information are combined to obtain the animation video.

[0164] Through the above implementation, when the first duration is not equal to the second duration, the information with the largest duration is compressed, which can ensure that the duration of the processed second video and the processed audio information are equal, and thus facilitate the direct merging of the processed second video and the processed audio information, thereby improving the generation efficiency of the animation video.

[0165] Specifically, the electronic device combines the processed second video and the processed audio information to obtain the animation video, including:

[0166] Obtaining soundtrack information of the processed second video in a soundtrack dimension;

[0167] The soundtrack information is replaced with the processed audio information to obtain the animation video.

[0168] By replacing the soundtrack information with the processed audio information, the animation video can be quickly generated.

[0169] It can be seen from the above technical solution that the present invention analyzes the text information through the video generation model, and can quickly generate the initial video, thereby improving the generation efficiency of the animation video. Furthermore, by identifying the human body feature points, the posture information of the user in each frame of the image can be accurately determined, and then when the posture information is a preset posture, the posture information is adjusted, which can avoid the presence of bad posture information such as the preset posture in the second video. Since good posture information can play a certain educational role for users, the quality of the second video can be improved by avoiding the presence of bad posture information such as the preset posture in the second video. The audio generation model can accurately generate audio information corresponding to the text information, and the generation quality of the animation video can be improved based on the audio information and the second video.

[0170] like Figure 2 , which is a functional module diagram of a preferred embodiment of the animation video generation device of the present invention. The animation video generation device 11 includes an acquisition unit 110, an input unit 111, a recognition unit 112, a generation unit 113, an adjustment unit 114, an analysis unit 115, a construction unit 116, an encoding unit 117, a mapping unit 118, and a calculation unit 119. The modules / units referred to in the present invention refer to a series of computer-readable instruction segments that can be acquired by the processor 13 and can perform fixed functions, which are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0171] When receiving a video generation request, the acquisition unit 110 acquires text information according to the video generation request.

[0172] In at least one embodiment of the present invention, the triggering user of the video generation request is different depending on the application scenario of the video generation request. For example, if the application scenario of the video generation request is in the field of education, the triggering user of the video generation request may be a teacher, etc.

[0173] The video generation request may include, but is not limited to: text path, preset tags, etc.

[0174] The text information refers to text information that needs to be converted into a video. For example, the text information may be a teacher's lecture notes.

[0175] In at least one embodiment of the present invention, the acquiring unit 110 acquiring text information according to the video generation request includes:

[0176] Parsing the video generation request message to obtain data information carried in the message;

[0177] extracting the text path from the data information according to the preset tag;

[0178] The text information is obtained from the text path.

[0179] The preset tag is a tag used to indicate a path, for example, the preset tag may be a storage location.

[0180] The text path can be accurately extracted through the preset tag, so that the text information can be accurately obtained, which is conducive to the generation of the corresponding animation video.

[0181] The input unit 111 inputs the text information into a pre-trained video generation model to obtain an initial video.

[0182] In at least one embodiment of the present invention, the video generation model is a model capable of converting text into video. The video generation model includes an encoding layer, a decoding layer, and a preset mapping table. The preset mapping table stores a mapping relationship between pixel values and vectors.

[0183] The initial video refers to a video generated after the video generation model analyzes the text information. The initial video does not contain voice information.

[0184] In at least one embodiment of the present invention, before inputting the text information into a pre-trained video generation model to obtain an initial video, the acquisition unit 110 acquires a plurality of video training samples, each video training sample including a training video and a training text corresponding to the training video;

[0185] The construction unit 116 constructs a learner, wherein the learner includes an encoding layer and a decoding layer;

[0186] The encoding unit 117 performs text encoding processing on the training text to obtain a text vector;

[0187] The analyzing unit 115 analyzes the text vector based on the coding layer to obtain feature information of the training text;

[0188] The analyzing unit 115 analyzes the feature information based on the decoding layer to obtain an output vector;

[0189] The mapping unit 118 performs mapping processing on the training video based on a preset mapping table to obtain an image vector of the training video;

[0190] The calculation unit 119 calculates the similarity between the text vector and the output vector to obtain a first similarity, and calculates the similarity between the text vector and the image vector to obtain a second similarity;

[0191] The calculation unit 119 calculates the ratio of the first similarity to the second similarity to obtain a learning index of the learner;

[0192] The adjusting unit 114 adjusts the network parameters in the learner until the learning index no longer increases, thereby obtaining the video generation model.

[0193] The text vector is used to represent the training text.

[0194] The learning indicator is used to evaluate the accuracy of the learner.

[0195] The network parameters include pre-set parameters in the encoding layer and the decoding layer. For example, if the encoding layer includes a convolution layer, the network parameters may be the size of the convolution kernel in the convolution layer.

[0196] By generating the learning indicator through the similarity between the training text and the predicted video and the similarity between the training text and the training video, and then adjusting the network parameters according to the learning indicator, the video generation model's ability to represent text information can be improved, thereby improving the accuracy of video generation.

[0197] In at least one embodiment of the present invention, the manner of analyzing the text information based on the video generation model is similar to the manner of analyzing the training text based on the learner, and the present invention will not elaborate on this.

[0198] The recognition unit 112 recognizes human feature points in each frame of the initial video.

[0199] In at least one embodiment of the present invention, the human body feature points include, but are not limited to: key feature points of the face, such as pupil centers, etc.; hand joints and bone joints, etc.

[0200] In at least one embodiment of the present invention, the identifying unit 112 identifies the human feature points of each frame of the initial video, including:

[0201] Detect each frame of image based on a preset detector to obtain the human body area in each frame of image;

[0202] Performing grayscale processing on the human body region to obtain a plurality of pixel points in the human body region and a pixel grayscale value corresponding to each pixel point;

[0203] Calculating the pixel difference between each pixel and the preset feature point according to the pixel grayscale value and the feature grayscale value of the preset feature point;

[0204] Determine the pixel points whose pixel difference is less than a preset threshold as initial feature points;

[0205] Constructing a coordinate system based on each frame of image, and obtaining initial coordinate information of the initial feature points on each frame of image;

[0206] The human body feature points are screened out from the initial feature points according to the initial coordinate information.

[0207] The preset detector may be used to identify person information in an image.

[0208] The preset feature points include hand joints and bone joints, etc. The feature grayscale value can be determined based on pixel information corresponding to the preset feature points of multiple preset users.

[0209] The preset threshold can be set according to requirements.

[0210] The coordinate system includes a horizontal axis and a vertical axis.

[0211] By detecting each frame of image through the preset detector, not only can the interference of background information in each frame of image on the human feature points be eliminated, thereby improving the recognition accuracy of the human feature points, but also the number of pixel points analyzed can be reduced, thereby improving the recognition efficiency of the human feature points, and then through the analysis of the pixel grayscale value and the feature grayscale value, the initial feature point can be quickly determined, and then the determination accuracy of the human feature point can be improved according to the initial coordinate information of the initial feature point.

[0212] In at least one embodiment of the present invention, the identifying unit 112 filters out the human body feature points from the initial feature points according to the initial coordinate information, including:

[0213] For any initial feature point, calculating a feature distance between the initial feature point and a target feature point according to the initial coordinate information, wherein the target feature point refers to the remaining feature points among the initial feature points except the initial feature point;

[0214] Determine the feature distance with the minimum value as the target distance, and determine the target feature point corresponding to the target distance as the adjacent feature point of any initial feature point;

[0215] Performing normal distribution processing on the target distance to obtain a probability value of the target distance;

[0216] The initial feature point corresponding to the target distance whose probability value is greater than the preset probability value is determined as the human body feature point.

[0217] The preset probability value may be set according to requirements, for example, the preset probability value may be 99.44%.

[0218] By analyzing the characteristic distance, the adjacent characteristic points of any initial characteristic point can be quickly determined. By performing normal distribution processing on the target distance and further analyzing the probability value of the target distance, the human body characteristic points can be accurately screened out from the initial characteristic points.

[0219] The generating unit 113 generates the user's posture information in each frame of image according to the human body feature points.

[0220] In at least one embodiment of the present invention, the posture information refers to the posture of the user in each frame of image. For example, the posture information may be lowering the head, or looking up the head.

[0221] In at least one embodiment of the present invention, the generating unit 113 generates the user's posture information in each frame of image according to the human body feature points, including:

[0222] Acquire the coordinate information of the human body feature points as human body coordinate information according to the coordinate system;

[0223] Acquire any two adjacent feature points from the human body feature points as a feature point pair;

[0224] Calculating the Euler angle of each feature point pair according to the human body coordinate information and the abscissa axis in the coordinate system;

[0225] An average value of the Euler angles is calculated to obtain an angle mean value, and preset posture information corresponding to the angle mean value is determined as the posture information.

[0226] Among them, the feature point pair refers to any two adjacent feature points obtained from the human body feature points. Further, the any two adjacent feature points refer to human body feature points with adjacent feature distances, for example, human body feature point A, human body feature point B, human body feature point C, and human body feature point D. The feature distance between the human body feature point A and the human body feature point B is 5, the feature distance between the human body feature point A and the human body feature point C is 2, and the feature distance between the human body feature point A and the human body feature point D is 3, then the human body feature point C is the adjacent feature point of the human body feature point A.

[0227] By calculating the Euler angles of any two adjacent feature points, it is possible to avoid interference of human feature points that are far apart on the posture information, thereby improving the accuracy of determining the posture information.

[0228] Specifically, the posture information can be determined based on a mapping table between angles and preset posture information, wherein the preset posture information can be marked by a user.

[0229] If the posture information is a preset posture, the adjustment unit 114 adjusts the posture information according to the human body feature points to obtain a second video.

[0230] In at least one embodiment of the present invention, the preset posture may include, but is not limited to: lowering the head, raising the head, and other bad postures.

[0231] The user posture of each frame image in the second video is not the preset posture.

[0232] In at least one embodiment of the present invention, the adjusting unit 114 adjusts the posture information according to the human body feature points to obtain the second video, including:

[0233] Get the pose angle of the standard pose from the pose mapping table;

[0234] comparing the angle mean with the posture angle;

[0235] If the angle mean is greater than the posture angle, comparing the Euler angle with the angle mean;

[0236] Obtaining feature point pairs corresponding to Euler angles having values greater than the angle mean as feature points to be processed;

[0237] The position of the feature point to be processed in the image is adjusted until the adjusted posture information is not a preset posture, thereby obtaining the second video.

[0238] The posture mapping table stores mappings between multiple preset posture information and angles, including the standard posture and undesirable postures such as pitch. The multiple preset posture information in the posture mapping table can be annotated by the user. The angles in the posture mapping table are calculated similarly to the mean angle value in each frame of the image, and this description will not be repeated herein.

[0239] By comparing the angle mean with the posture angle, and comparing the Euler angle with the angle mean, the human body feature points that affect the posture information can be quickly determined and then adjusted to improve the quality of the second video.

[0240] The analyzing unit 115 analyzes the text information based on a pre-trained audio generation model to obtain audio information.

[0241] In at least one embodiment of the present invention, the audio generation model is used to convert the text information into speech.

[0242] The audio information refers to the voice corresponding to the text information.

[0243] In at least one embodiment of the present invention, the audio generation model includes an emotion recognition network layer and a speech conversion network layer. The analysis unit 115 analyzes the text information based on the pre-trained audio generation model to obtain audio information including:

[0244] Analyze the text information based on the emotion recognition network layer to obtain the text emotion of the text information;

[0245] Acquire emotional speech features of the text emotion from a speech feature library;

[0246] Processing the text information based on the speech conversion network layer to obtain speech information, and obtaining text speech features from the speech information;

[0247] The text speech feature and the emotional speech feature are subjected to audio mixing processing to obtain the audio information.

[0248] The emotion recognition network layer is used to analyze the emotion corresponding to the text, which can be happiness, sadness, etc.

[0249] The speech conversion network layer is used to convert text into speech.

[0250] By performing audio mixing processing on the text voice features and the emotional voice features, the text emotion is included in the audio information, thereby improving the interest of the audio information.

[0251] The generating unit 113 generates an animation video according to the second video and the audio information.

[0252] In at least one embodiment of the present invention, the animation video refers to a video including the audio information and the second video.

[0253] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned animation video, the above-mentioned animation video can also be stored in a blockchain node.

[0254] In at least one embodiment of the present invention, the generating unit 113 generates the animation video according to the second video and the audio information, including:

[0255] Counting the duration of the second video to obtain a first duration;

[0256] Counting the duration of the audio information to obtain a second duration;

[0257] If the first duration is not equal to the second duration, obtaining information with the largest duration from the second video and the audio information as information to be processed;

[0258] Compressing the information to be processed until the lengths of the processed second video and the processed audio information are equal;

[0259] The processed second video and the processed audio information are combined to obtain the animation video.

[0260] Through the above implementation, when the first duration is not equal to the second duration, the information with the largest duration is compressed, which can ensure that the duration of the processed second video and the processed audio information are equal, and thus facilitate the direct merging of the processed second video and the processed audio information, thereby improving the generation efficiency of the animation video.

[0261] Specifically, the generating unit 113 combines the processed second video and the processed audio information to obtain the animation video including:

[0262] Obtaining soundtrack information of the processed second video in a soundtrack dimension;

[0263] The soundtrack information is replaced with the processed audio information to obtain the animation video.

[0264] By replacing the soundtrack information with the processed audio information, the animation video can be quickly generated.

[0265] It can be seen from the above technical solution that the present invention analyzes the text information through the video generation model, and can quickly generate the initial video, thereby improving the generation efficiency of the animation video, and then through the recognition of the human body feature points, it can accurately determine the user's posture information in each frame image, and then when the posture information is a preset posture, the posture information is adjusted, and the presence of bad posture information such as the preset posture in the second video can be avoided. Since good posture information can play a certain educational role for users, the quality of the second video can be improved by avoiding the presence of bad posture information such as the preset posture in the second video. The audio generation model can accurately generate audio information corresponding to the text information, and the generation quality of the animation video can be improved based on the audio information and the second video.

[0266] like Figure 3 FIG. 1 is a schematic diagram of the structure of an electronic device according to a preferred embodiment of the present invention for implementing the method for generating animation video.

[0267] In one embodiment of the present invention, the electronic device 1 includes, but is not limited to, a memory 12 , a processor 13 , and computer-readable instructions stored in the memory 12 and executable on the processor 13 , such as an animation video generation program.

[0268] Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 1 and does not constitute a limitation on the electronic device 1. The electronic device 1 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 1 may also include input and output devices, network access devices, buses, etc.

[0269] The processor 13 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 13 is the computing core and control center of the electronic device 1, connecting various parts of the entire electronic device 1 using various interfaces and lines, and executing the operating system of the electronic device 1 as well as various installed applications, program codes, etc.

[0270] Exemplarily, the computer-readable instructions may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to implement the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, which describe the execution process of the computer-readable instructions in the electronic device 1. For example, the computer-readable instructions may be divided into an acquisition unit 110, an input unit 111, an identification unit 112, a generation unit 113, an adjustment unit 114, an analysis unit 115, a construction unit 116, an encoding unit 117, a mapping unit 118, and a calculation unit 119.

[0271] The memory 12 can be used to store the computer-readable instructions and / or modules. The processor 13 implements the various functions of the electronic device 1 by running or executing the computer-readable instructions and / or modules stored in the memory 12 and calling the data stored in the memory 12. The memory 12 can mainly include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created based on the use of the electronic device. The memory 12 can include non-volatile and volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, at least one disk storage device, a flash memory device, or other storage devices.

[0272] The memory 12 may be an external memory and / or an internal memory of the electronic device 1. Furthermore, the memory 12 may be a physical memory, such as a memory stick, a TF card (Trans-flash Card), and the like.

[0273] If the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the processes in the above-mentioned embodiment methods by instructing related hardware through computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium. When the computer-readable instructions are executed by a processor, the steps of each of the above-mentioned method embodiments can be implemented.

[0274] The computer-readable instructions include computer-readable instruction codes, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer-readable instruction codes, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), or a random access memory (RAM).

[0275] Blockchain, as used in this article, refers to a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each block contains information about a batch of online transactions, used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0276] Combine Figure 1 The memory 12 in the electronic device 1 stores computer-readable instructions to implement an animation video generation method, and the processor 13 can execute the computer-readable instructions to implement:

[0277] When a video generation request is received, obtaining text information according to the video generation request;

[0278] Inputting the text information into a pre-trained video generation model to obtain an initial video;

[0279] Identifying human feature points in each frame of the initial video;

[0280] Generating user posture information in each frame of image according to the human body feature points;

[0281] If the posture information is a preset posture, adjusting the posture information according to the human body feature points to obtain a second video;

[0282] Analyze the text information based on a pre-trained audio generation model to obtain audio information;

[0283] An animation video is generated according to the second video and the audio information.

[0284] Specifically, the specific implementation method of the processor 13 for the above computer readable instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0285] In the several embodiments provided herein, it should be understood that the disclosed systems, devices, and methods may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical functional division, and actual implementation may employ other division methods.

[0286] The computer-readable storage medium stores computer-readable instructions, wherein the computer-readable instructions, when executed by the processor 13, are used to implement the following steps:

[0287] When a video generation request is received, obtaining text information according to the video generation request;

[0288] Inputting the text information into a pre-trained video generation model to obtain an initial video;

[0289] Identifying human feature points in each frame of the initial video;

[0290] Generating user posture information in each frame of image according to the human body feature points;

[0291] If the posture information is a preset posture, adjusting the posture information according to the human body feature points to obtain a second video;

[0292] Analyze the text information based on a pre-trained audio generation model to obtain audio information;

[0293] An animation video is generated according to the second video and the audio information.

[0294] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0295] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0296] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0297] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices described may also be implemented by a single unit or device through software or hardware. Terms such as first and second are used to indicate names and do not imply any particular order.

[0298] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for generating an animation video, characterized in that: The animation video generation method comprises: When a video generation request is received, obtaining text information according to the video generation request; Inputting the text information into a pre-trained video generation model to obtain an initial video; Identifying human feature points in each frame of the initial video includes: detecting each frame of the image based on a preset detector to obtain a human body region in each frame of the image; performing grayscale processing on the human body region to obtain a plurality of pixel points in the human body region and a pixel grayscale value corresponding to each pixel point; calculating a pixel difference between each pixel point and a preset feature point based on the pixel grayscale value and the feature grayscale value of the preset feature point; determining a pixel point whose pixel difference is less than a preset threshold as an initial feature point; constructing a coordinate system based on each frame of the image, and obtaining initial coordinate information of the initial feature point on each frame of the image; and screening out the human feature point from the initial feature points based on the initial coordinate information; Generating user posture information in each frame of image according to the human body feature points; If the posture information is a preset posture, adjusting the posture information according to the human body feature points to obtain a second video; Analyze the text information based on a pre-trained audio generation model to obtain audio information; An animation video is generated according to the second video and the audio information.

2. The method for generating an animation video according to claim 1, wherein: Before inputting the text information into a pre-trained video generation model to obtain an initial video, the method further includes: Acquire multiple video training samples, each video training sample including a training video and a training text corresponding to the training video; Constructing a learner, wherein the learner includes an encoding layer and a decoding layer; Performing text encoding processing on the training text to obtain a text vector; Analyze the text vector based on the coding layer to obtain feature information of the training text; Analyze the feature information based on the decoding layer to obtain an output vector; Performing mapping processing on the training video based on a preset mapping table to obtain an image vector of the training video; Calculating the similarity between the text vector and the output vector to obtain a first similarity, and calculating the similarity between the text vector and the image vector to obtain a second similarity; calculating a ratio of the first similarity to the second similarity to obtain a learning index of the learner; The network parameters in the learner are adjusted until the learning index no longer increases, thereby obtaining the video generation model.

3. The method for generating an animation video according to claim 1, wherein: The step of selecting the human body feature points from the initial feature points according to the initial coordinate information includes: For any initial feature point, calculating a feature distance between the initial feature point and a target feature point according to the initial coordinate information, wherein the target feature point refers to the remaining feature points among the initial feature points except the initial feature point; Determine the feature distance with the minimum value as the target distance, and determine the target feature point corresponding to the target distance as the adjacent feature point of any initial feature point; Performing normal distribution processing on the target distance to obtain a probability value of the target distance; The initial feature point corresponding to the target distance whose probability value is greater than the preset probability value is determined as the human body feature point.

4. The method for generating an animation video according to claim 1, wherein: Generating the user's posture information in each frame of image according to the human body feature points includes: Acquire the coordinate information of the human body feature points as human body coordinate information according to the coordinate system; Acquire any two adjacent feature points from the human body feature points as a feature point pair; Calculating the Euler angle of each feature point pair according to the human body coordinate information and the abscissa axis in the coordinate system; An average value of the Euler angles is calculated to obtain an angle mean value, and preset posture information corresponding to the angle mean value is determined as the posture information.

5. The method for generating an animation video according to claim 1, wherein: The audio generation model includes an emotion recognition network layer and a speech conversion network layer. The audio information obtained by analyzing the text information based on the pre-trained audio generation model includes: Analyze the text information based on the emotion recognition network layer to obtain the text emotion of the text information; Acquire emotional speech features of the text emotion from a speech feature library; Processing the text information based on the speech conversion network layer to obtain speech information, and acquiring text-to-speech features from the speech information; The text speech feature and the emotional speech feature are subjected to audio mixing processing to obtain the audio information.

6. The method for generating an animation video according to claim 1, wherein: Generating an animation video according to the second video and the audio information includes: Counting the duration of the second video to obtain a first duration; Counting the duration of the audio information to obtain a second duration; If the first duration is not equal to the second duration, obtaining information with the largest duration from the second video and the audio information as information to be processed; Compressing the information to be processed until the lengths of the processed second video and the processed audio information are equal; The processed second video and the processed audio information are combined to obtain the animation video.

7. An animation video generating device, characterized in that: The animation video generating device comprises: an acquiring unit, configured to acquire text information according to a video generation request when a video generation request is received; An input unit, configured to input the text information into a pre-trained video generation model to obtain an initial video; an identification unit for identifying human feature points in each frame of the initial video, comprising: detecting each frame of the image based on a preset detector to obtain a human body region in each frame of the image; performing grayscale processing on the human body region to obtain a plurality of pixel points in the human body region and a pixel grayscale value corresponding to each pixel point; calculating a pixel difference between each pixel point and a preset feature point based on the pixel grayscale value and the feature grayscale value of the preset feature point; determining a pixel point whose pixel difference is less than a preset threshold as an initial feature point; constructing a coordinate system based on each frame of the image, and obtaining initial coordinate information of the initial feature point on each frame of the image; and screening out the human feature point from the initial feature points based on the initial coordinate information; A generating unit, configured to generate posture information of the user in each frame of image according to the human body feature points; an adjusting unit, configured to adjust the posture information according to the human body feature points to obtain a second video if the posture information is a preset posture; An analysis unit, configured to analyze the text information based on a pre-trained audio generation model to obtain audio information; The generating unit is configured to generate an animation video based on the second video and the audio information.

8. An electronic device, characterized in that: The electronic device comprises: a memory storing computer-readable instructions; and A processor executes computer-readable instructions stored in the memory to implement the animation video generation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions are executed by a processor in an electronic device to implement the animation video generation method according to any one of claims 1 to 6.

Images