Auxiliary voice generation method and system based on convolutional neural network and long short-term memory network

Abstract

The application discloses an auxiliary voice production method and system based on a convolutional neural network and a long-term short-term memory network, and relates to the technical field of image processing applications. The method comprises the following steps: constructing a prediction model framework through three neural network parts of LSTM, CNN and FCN; collecting voice production action videos, and taking frames of the voice production action videos; using part of the data files processed through frame taking to adjust parameters of the prediction model framework, and exporting a prediction model; inputting facial actions into the prediction model, and outputting semantic information of the facial actions; through combination of two deep neural networks of LSTM and CNN, processing videos of facial actions in a voice production process of a character, through combination of semantic labels of the videos, and through the ability of computer vision technology to learn video data features, and through training and optimization of the deep neural network, finally, a voice semantic prediction model with high-precision prediction ability is obtained.

Description

Technical Field

[0001] This application relates to the field of neural network technology, specifically to an auxiliary speech generation method and system based on convolutional neural networks and long short-term memory networks. Background Technology

[0002] Voice disorders refer to the phenomenon of repetition of sounds or interruption of words or sentences during speech. The vocal organs may experience twitching movements and spasms in various muscle groups. Speech is produced by the normal combined activity of the vocal cords (created by the respiratory tract of the diaphragm) and the tongue, lips, and palate (which constitute speech). When almost all of these organs spasm and lose their free movement, unclear pronunciation results. How to assist individuals with voice disorders in vocalization and semantic communication is a valuable question. Because the human vocal process involves the coordinated work of multiple organs such as the trachea, vocal cords, and larynx, analyzing and translating the vocal process solely through physiological acoustics and pathology is a relatively difficult task.

[0003] However, due to the functions of related organs during vocalization, the human face, throat, and neck undergo corresponding mechanical movements. Therefore, by establishing the relationship between vocal semantics and the external features of the mechanical movements of the face, throat, and neck during vocalization, a vocalization prediction model can be constructed, and the optimal solution of this model can be explored and studied. Once the optimal model is established, it can be used to predict the semantic expressions of individuals with vocal disorders.

[0004] Currently, there is no vocalization prediction model that establishes a relationship between vocalization semantics and the external features of the mechanical movements of the face, throat, and neck during the vocalization process. Therefore, it is worthwhile to study how to establish such a vocalization prediction model. Summary of the Invention

[0005] The purpose of this invention is to address the current lack of a vocalization prediction model that establishes a relationship between the external features of facial, throat, and neck mechanical movements during vocalization and semantics. This invention combines two deep neural networks, LSTM and CNN, to process captured videos of facial movements during vocalization. By integrating semantic tags from the videos and leveraging the ability of computer vision technology to learn video data features, and through training and optimization of the model using deep neural networks, a vocalization semantic prediction model with high accuracy is ultimately obtained.

[0006] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0007] In a first aspect of this application, an assisted speech generation method based on convolutional neural networks and long short-term memory networks is provided, comprising:

[0008] A prediction model framework is constructed using three neural networks: LSTM, CNN, and FCN.

[0009] Collect videos of vocalization actions and extract frames from the videos of vocalization actions;

[0010] The data files after frame extraction are used to adjust the parameters of the prediction model framework and export the prediction model.

[0011] The prediction model takes facial actions as input and outputs semantic information about those facial actions.

[0012] In one embodiment of this application, the step of constructing a prediction model framework using three neural networks—LSTM, CNN, and FCN—specifically includes the following steps:

[0013] The two-layer LSTM network and the one-layer CNN network are used to process the video data of the sound action. Specifically, the LSTM layer is used to process the time series video stream data, and the CNN layer is used to learn the local spatial features of each frame of the video and perform feature compression and extraction.

[0014] The FCN component is used to output the network's prediction of semantic categories.

[0015] In one embodiment of this application, the step of using a portion of the data file after frame extraction for parameter adjustment of the prediction model framework specifically includes the following steps:

[0016] The prediction model is forward-propagated, which is the process by which the neural network predicts data, and the forward propagation is used to obtain the loss value;

[0017] The prediction model is backpropagated, which is the process of calculating the partial derivatives of the loss function with respect to the parameters in different layers of the neural network function. The backpropagation is used to adjust the parameters based on the value of the loss function.

[0018] In one embodiment of this application, the step of collecting video of vocalization actions and extracting frames from the video of vocalization actions specifically includes the following steps:

[0019] Designate a person with a normal voice to read the relevant text aloud, and use video equipment to capture the process.

[0020] Video data cleaning involves filtering and selecting videos that are abnormal or contain incorrect information.

[0021] The video data is processed by extracting frames at a certain ratio using the Open-CV library, forming a Numpy binary file.

[0022] In one embodiment of this application, after the step of forming a NumPy binary file, the method further includes the following steps:

[0023] The NumPy binary files form a dataset, which is then divided into training, validation, and test sets according to a set ratio.

[0024] Determine the dimensions of the input and output data, as well as the amount of training data, and then determine the number of input and output layer nodes in the LSTM and FCN layers used in the modeling process.

[0025] The size of the convolutional kernel, stride, and padding in the CNN layer is determined based on the image size.

[0026] The number of network layers and the number of nodes in each layer in the CNN are determined based on the size of the dataset, and the number of layers and the number of nodes in each layer of the CNN are adjusted accordingly.

[0027] The prediction model can be optimized by adjusting the number of layers and nodes in the LSTM and CNN networks. Alternatively, a random search method can be used to find the hyperparameters that minimize the loss function from a set of pre-determined hyperparameters, including combinations of layer and node numbers.

[0028] In one embodiment of this application, the forward propagation specifically involves inputting the same batch of NumPy data from the training set into a convolutional neural network. The matrix calculation for the convolution operation is as follows:

[0029]

[0030] The For the nth input neuron of the convolutional neural network, the The bias of the nth neuron is W, where W is the weight coefficient. The weight coefficients corresponding to the output of the i1th neuron and the j1th neuron in the next layer are described. The output result is the summation of the input signals according to the weight ratio. for The final output result after nonlinear mapping, wherein the nonlinear mapping is: ,Right now:

[0031] ;

[0032] The loss function of the backpropagation model is as follows:

[0033] ;

[0034] Among them, the This is the loss value. For predicted values, The actual value;

[0035] ;

[0036] Using the error as the objective function, the weights are adjusted through multiple reverse iterations until they stabilize, thus obtaining the optimal value.

[0037] In one embodiment of this application, the step of having a designated person with normal voice read aloud relevant speech material, while simultaneously capturing video of the process using a camera device, specifically includes the following steps:

[0038] The corpus should be selected based on the specific usage scenario and scope.

[0039] The subject reads the corpus aloud multiple times and the recordings are made. The subject's speaking speed, volume, and head posture are varied each time the recording is made.

[0040] In a second aspect of this application, an assisted speech system based on convolutional neural networks and long short-term memory networks is provided, the device comprising:

[0041] Framework component modules: The prediction model framework is constructed using three neural networks: LSTM, CNN, and FCN.

[0042] Data acquisition module: collects video of vocalization actions and extracts frames from the video of vocalization actions;

[0043] Parameter adjustment module: Uses a portion of the data file after frame extraction processing to adjust the parameters of the prediction model framework and exports the prediction model;

[0044] Model output module: The prediction model takes facial action as input and outputs the semantic information of the facial action.

[0045] In one embodiment of this application, the frame component module specifically includes:

[0046] The network consists of a sub-module: a two-layer LSTM network and a CNN network are used to process video data of vocalization actions. Specifically, the LSTM layer is used to process time-series video stream data, and the CNN layer is used to learn the local spatial features of each frame of the video and perform feature compression and extraction.

[0047] The FCN component is used to output the network's prediction of semantic categories.

[0048] In one embodiment of this application, the parameter adjustment module specifically includes:

[0049] Forward propagation submodule: performs forward propagation on the prediction model, whereby forward propagation is the process by which the neural network predicts data, and is used to obtain the loss value;

[0050] Backpropagation submodule: Performs backpropagation on the prediction model. Backpropagation is the process of calculating the partial derivatives of the loss function with respect to the parameters in different layers of the neural network function. Backpropagation is used to adjust the parameters based on the value of the loss function.

[0051] In one embodiment of this application, the data acquisition module specifically includes:

[0052] Video capture submodule: Designate a person with normal speech to read the relevant text aloud, and use a camera to capture the process.

[0053] Video cleaning submodule: Cleans video data, that is, filters and selects videos with abnormalities and incorrect information from the video data;

[0054] The frame extraction submodule uses the Open-CV library to extract frames from the video data at a certain ratio, forming a NumPy binary file.

[0055] In one embodiment of this application, it further includes:

[0056] Data segmentation submodule: The NumPy binary files form a dataset, which is then divided into training set, validation set and test set according to the specified proportions;

[0057] Parameter tuning submodule: Determines the dimensions of the input and output data, as well as the amount of training data, and then determines the number of input and output layer nodes in the LSTM layer and the FCN layer in the modeling process;

[0058] The size of the convolutional kernel, stride, and padding in the CNN layer is determined based on the image size.

[0059] The number of network layers and the number of nodes in each layer in the CNN are determined based on the size of the dataset, and the number of layers and the number of nodes in each layer of the CNN are adjusted accordingly.

[0060] The prediction model can be optimized by adjusting the number of layers and nodes in the LSTM and CNN networks. Alternatively, a random search method can be used to find the hyperparameters that minimize the loss function from a set of pre-determined hyperparameters, including combinations of layer and node numbers.

[0061] In one embodiment of this application, the video acquisition submodule specifically includes:

[0062] Corpus selection submodule: Select corpora based on specific use cases and scope of use;

[0063] Differential recording module: The subject reads the corpus multiple times and records the recording. The subject's speaking speed, volume, and head posture are varied each time the recording is made.

[0064] This application has the following beneficial effects:

[0065] In this application, the solution combines two deep neural networks, LSTM and CNN, to process the captured video of facial movements during a person's vocalization process. By combining the semantic tags of the video with the ability of computer vision technology to learn video data features, and after training and optimizing the model through the deep neural network, semantic prediction is finally performed through the FCN deep neural network to obtain a vocalization semantic prediction model with high-precision prediction capabilities. The neural network model can more accurately reflect the effect of vocalization and solve the inaccuracies and errors of previous manual corrections. It can reflect the effect and error of the user's pronunciation in real time, provide timely feedback and correction, and improve efficiency. Attached Figure Description

[0066] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0067] Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of this application.

[0068] Figure 2 This is a schematic diagram of the system architecture of an embodiment of this application.

[0069] Figure 3 This is a flowchart illustrating the steps of the speech-assisted method based on convolutional neural networks and long short-term memory networks provided in this application embodiment.

[0070] Figure 4 This is a schematic diagram of the functional modules of the auxiliary voice system based on convolutional neural networks and long short-term memory networks provided in the embodiments of this application.

[0071] Figure 5 This is a schematic diagram of the forward propagation function module.

[0072] Figure 6 This is a schematic diagram of the parameter adjustment function module. Detailed Implementation

[0073] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0074] The scheme of this application will be further described below with reference to the accompanying drawings.

[0075] Reference Figure 1 , Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of this application.

[0076] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0077] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0078] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and electronic programs.

[0079] exist Figure 1In the illustrated electronic device, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be disposed in the electronic device. The electronic device calls the auxiliary voice system based on convolutional neural network and long short-term memory network stored in the memory 1005 through the processor 1001, and executes the auxiliary voice method based on convolutional neural network and long short-term memory network provided in the embodiments of this application.

[0080] Reference Figure 2 The diagram illustrates a system architecture according to an embodiment of this application. Figure 2 As shown, the system architecture may include a first device 201, a second device 202, a third device 203, a fourth device 204, and a network 205. The network 205 serves as a medium for providing communication links between the first device 201, the second device 202, the third device 203, and the fourth device 204. The network 205 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0081] In this embodiment, the first device 201 can process data and / or information obtained from the second device 202, the third device 203, and / or the fourth device 204. For example, the first device 201 can process video data obtained by the second device 202 to obtain frame-wise video data. In some embodiments, the first device 201 can be a single server or a group of servers. The server group can be centralized or distributed. In some embodiments, the first device 201 can be local or remote. For example, the first device 201 can access information and / or data from the second device 202, the third device 203, and / or the fourth device 204 via network 205. As another example, the processing device 120 can be directly connected to the second device 202, the third device 203, and / or the fourth device 204 to access information and / or data. In some embodiments, the first device 201 can be implemented on a cloud platform. For example, the cloud platform can include private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, inter-cloud cloud, multi-cloud, etc., or any combination thereof.

[0082] In this embodiment, the second device 202 is a video recording device, which may specifically be or include a photography device.

[0083] In this embodiment, the fourth device 204 can be either hardware or software that supports network connectivity to provide various network services. When the device is hardware, it can be various electronic devices, including but not limited to smartphones 204-1, tablets, projectors 204-2, desktop computers 204-3, and servers. In this case, as hardware, it can be implemented as a distributed device group consisting of multiple devices, or as a single device. When the device is software, it can be installed on the devices listed above. In this case, as software, it can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0084] In this embodiment, the third device 203 is a storage device that can store data (e.g., raw video data, frame-processed video data, etc.), instructions, and / or any other information. In some embodiments, the storage device can store data acquired from the first device 201, the second device 202, and / or the fourth device 204. For example, the storage device can store raw video data acquired from the second device 202. Similarly, the storage device can store frame-processed video data acquired from the first device 201. In some embodiments, the storage device can store data and / or instructions that the first device 201 can execute or use to perform the exemplary methods described in this specification.

[0085] In some embodiments, the storage device may include one or a combination of several of the following: mass storage, removable memory, volatile read-write memory, read-only memory (ROM). In some embodiments, the storage device may be implemented using the means described herein.

[0086] In some embodiments, the storage device may be connected to network 205 to communicate with one or more components (e.g., first device 201, fourth device 204) in an assisted speech system based on convolutional neural networks and long short-term memory networks. One or more components in a CT image restoration processing apparatus may read data or instructions from a third device 203 via network 205. In some embodiments, the third device 203 may be part of the first device 201 or may be independent and directly or indirectly connected to the first device 201.

[0087] Network 205 may include any suitable network capable of facilitating information and / or data exchange between a voice-assisting system based on convolutional neural networks and long short-term memory networks. In some embodiments, one or more components of the voice-assisting system based on convolutional neural networks and long short-term memory networks (e.g., first device 201, second device 202, third device 203, fourth device 204) may exchange information and / or data with one or more components of the voice-assisting system based on convolutional neural networks and long short-term memory networks via network 205. For example, first device 201 may acquire a target image to be processed from second device 202 via network 205. In some embodiments, network 205 may include one or more combinations of public networks (e.g., the Internet), private networks (e.g., local area networks (LANs), wide area networks (WANs)), wired networks (e.g., Ethernet), wireless networks (e.g., 802.11 networks, Wi-Fi networks), cellular networks (e.g., LTE networks), Frame Relay networks, virtual private networks (VPNs), satellite networks, telephone networks, routers, hubs, server computers, etc. In some embodiments, network 205 may include one or more network access points. For example, network 205 may include wired and / or wireless network access points, such as base stations and / or Internet switching points, through which one or more components of an assistive speech system based on convolutional neural networks and long short-term memory networks can connect to network 205 to exchange data and / or information.

[0088] In practice, devices can provide network services by installing corresponding client or server applications. After installing a client application, the device can function as a client in network communication. Similarly, after installing a server application, it can function as a server in network communication.

[0089] Both the second device 202 and the fourth device 204 can be understood as terminals. A terminal can be a terminal with wireless transceiver capabilities or a chip or chip system that can be installed on the terminal. This terminal device can also be referred to as a user equipment (UE), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent, or user equipment. In the embodiments of this application, the terminal device can be a mobile phone, tablet computer, computer with wireless transceiver capabilities, virtual reality (VR) terminal device, augmented reality (AR) terminal device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical care, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, vehicle-mounted terminal, RSU with terminal functionality, etc.

[0090] It should be understood that Figure 2 The number of networks and devices shown is merely illustrative. Any number of networks and devices can be included depending on implementation needs.

[0091] Reference Figure 3 Based on the aforementioned hardware operating environment and system architecture, embodiments of this application provide an assisted speech generation method based on convolutional neural networks and long short-term memory networks, including:

[0092] S301: A prediction model framework is constructed using three neural networks: LSTM, CNN, and FCN.

[0093] It should be noted that the LSTM neural network is a Long Short Term Memory Network (LSTM), which is an improved recurrent neural network and a special type of RNN. It mainly controls the accumulation rate of information by introducing a gating mechanism, including selectively adding new information and selectively forgetting previously accumulated information, thereby improving the long-range dependency problem of recurrent neural networks (RNNs) and alleviating the gradient vanishing problem in the long sequence training process.

[0094] It should be noted that the CNN neural network mentioned is a Convolutional Neural Network (CNN), a type of feedforward neural network. Its artificial neurons can respond to a portion of the surrounding units within their coverage area, exhibiting excellent performance in large-scale image processing. It includes convolutional layers and pooling layers;

[0095] It should be noted that the FCN neural network is a framework for image semantic segmentation using Fully Convolutional Networks (FCN). The FCN replaces the fully connected layers at the back of the traditional CNN with convolutional layers, so that the output of the network will be a heatmap rather than a category. At the same time, in order to solve the problem of image size reduction caused by convolution and pooling, upsampling is used to restore the image size.

[0096] S302: Collect the video of the vocalization action and extract frames from the video of the vocalization action;

[0097] In this embodiment, a vocalization prediction model is needed to obtain the relationship between the external features of the vocalization semantics and the mechanical movements of the human face, throat, and neck during the vocalization process. Videos of vocalization movements need to be collected and processed to obtain a dataset for training and validating the prediction model.

[0098] S303: Use a portion of the data file after frame extraction for parameter adjustment of the prediction model framework, and export the prediction model;

[0099] S304: The prediction model takes facial action as input and outputs the semantic information of the facial action.

[0100] In this embodiment, by combining two deep neural networks, LSTM and CNN, the video of the captured facial movements during a person's vocalization is processed. By combining the semantic tags of the video and the ability of computer vision technology to learn video data features, and after training and optimizing the model through the deep neural network, the semantic prediction is finally performed through the FCN deep neural network to obtain a vocalization semantic prediction model with high-precision prediction capabilities.

[0101] In one feasible implementation, the step of constructing a prediction model framework using three neural networks—LSTM, CNN, and FCN—specifically includes the following steps:

[0102] The two-layer LSTM network and the one-layer CNN network are used to process the video data of the sound action. Specifically, the LSTM layer is used to process the time series video stream data, and the CNN layer is used to learn the local spatial features of each frame of the video and perform feature compression and extraction.

[0103] The FCN component is used to output the network's prediction of semantic categories.

[0104] In one feasible implementation, the step of using a portion of the data file after frame extraction for parameter adjustment of the prediction model framework specifically includes the following steps:

[0105] S401: Perform forward propagation on the prediction model, where forward propagation is the prediction process of the neural network on the data, and forward propagation is used to obtain the loss value;

[0106] S402: Perform backpropagation on the prediction model. Backpropagation is the process of calculating the partial derivatives of the loss function with respect to the parameters in different layers of the neural network function. Backpropagation is used to adjust the parameters using the value of the loss function.

[0107] It's important to understand that a neural network can be simply understood as a function approximator, approximating a mapping function from input x to output y. The network parameters we train are essentially fitting the unknowns of this mapping function. Training a neural network can be divided into two steps: forward propagation and backward propagation. Forward propagation moves from the input layer to the output layer: starting from the input layer, it proceeds through layers, continuously calculating the result of each layer and the output result of the activation function, ultimately yielding the output. Backward propagation, having calculated the output value (i.e., the predicted value) in forward propagation, allows us to calculate the loss based on the difference between the output and target values. Backward propagation involves calculating the partial derivatives of the loss function in the reverse direction, changing the weights of each layer from the last layer forward—essentially updating the parameters. The core principle is the chain rule of taking the partial derivative of the loss function with respect to each parameter of each network layer.

[0108] In one feasible implementation, the step of collecting the video of the vocalization action and extracting frames from the video of the vocalization action specifically includes the following steps:

[0109] S501: Designate a person with normal speech to read the relevant text aloud, and simultaneously use a camera to capture the video of the process;

[0110] S502: Clean the video data, that is, filter and screen videos with abnormalities and incorrect information in the video data;

[0111] S503: The Open-CV library is used to extract frames from video data at a certain ratio to form Numpy binary files.

[0112] It should be noted that OpenCV (Open Source Computer Vision Library) is an open-source computer vision library that provides many functions that efficiently implement computer vision algorithms.

[0113] It should be noted that NumPy (Numerical Python) is an open-source numerical computing extension for Python that supports large-dimensional arrays and matrix operations. By using the OpenCV library to process video data into frames at a certain ratio, NumPy binary files are generated, making the video data usable for training predictive models.

[0114] In one feasible implementation, after the step of forming the NumPy binary file, the method further includes the following steps:

[0115] The NumPy binary files form a dataset, which is then divided into training, validation, and test sets according to a set ratio.

[0116] Determine the dimensions of the input and output data, as well as the amount of training data, and then determine the number of input and output layer nodes in the LSTM and FCN layers used in the modeling process.

[0117] The size of the convolutional kernel, stride, and padding in the CNN layer is determined based on the image size.

[0118] The number of network layers and the number of nodes in each layer in the CNN are determined based on the size of the dataset, and the number of layers and the number of nodes in each layer of the CNN are adjusted accordingly.

[0119] The prediction model can be optimized by adjusting the number of layers and nodes in the LSTM and CNN networks. Alternatively, a random search method can be used to find the hyperparameters that minimize the loss function from a set of pre-determined hyperparameters, including combinations of layer and node numbers.

[0120] In one feasible implementation, see Figure 5 and Figure 6 As shown, the forward propagation specifically involves inputting the same batch of NumPy data from the training set into the convolutional neural network. The matrix calculation for the convolution operation is as follows:

[0121]

[0122] The For the nth input neuron of the convolutional neural network, the The bias of the nth neuron is W, where W is the weight coefficient. The weight coefficients corresponding to the output of the i1th neuron and the j1th neuron in the next layer are described. The output result is the summation of the input signals according to the weight ratio. for The final output result after nonlinear mapping, wherein the nonlinear mapping is: ,Right now:

[0123] ;

[0124] The loss function of the backpropagation model is as follows:

[0125] ;

[0126] Among them, the For loss function, For predicted values, The actual value;

[0127] ;

[0128] Using the error as the objective function, the weights are adjusted through multiple reverse iterations until they stabilize, thus obtaining the optimal value.

[0129] In one feasible implementation, the step of having a designated person with a normal voice read aloud the relevant corpus while simultaneously capturing video of the process using a camera device specifically includes the following steps:

[0130] The corpus should be selected based on the specific usage scenario and scope.

[0131] The subject reads the corpus aloud multiple times and the recordings are made. The subject's speaking speed, volume, and head posture are varied each time the recording is made.

[0132] In a second aspect of this application, an assisted speech system 400 based on a convolutional neural network and a long short-term memory network is provided, the device comprising:

[0133] Framework component module 401: Constructs a prediction model framework using three neural networks: LSTM, CNN, and FCN;

[0134] Data acquisition module 402: collects video of vocalization actions and extracts frames from the video of vocalization actions;

[0135] Parameter adjustment module 403: Uses a portion of the data file after frame extraction for parameter adjustment of the prediction model framework, and exports the prediction model;

[0136] Model output module 404: The prediction model takes a facial action as input and outputs the semantic information of the facial action.

[0137] In one feasible implementation, the frame component module specifically includes:

[0138] The network consists of a sub-module: a two-layer LSTM network and a CNN network are used to process video data of vocalization actions. Specifically, the LSTM layer is used to process time-series video stream data, and the CNN layer is used to learn the local spatial features of each frame of the video and perform feature compression and extraction.

[0139] The FCN component is used to output the network's prediction of semantic categories.

[0140] In one feasible implementation, the parameter adjustment module specifically includes:

[0141] Forward propagation submodule: performs forward propagation on the prediction model, whereby forward propagation is the process by which the neural network predicts data, and is used to obtain the loss value;

[0142] Backpropagation submodule: Performs backpropagation on the prediction model. Backpropagation is the process of calculating the partial derivatives of the loss function with respect to the parameters in different layers of the neural network function. Backpropagation is used to adjust the parameters based on the value of the loss function.

[0143] In one feasible implementation, the data acquisition module specifically includes:

[0144] Video capture submodule: Designate a person with normal speech to read the relevant text aloud, and use a camera to capture the process.

[0145] Video cleaning submodule: Cleans video data, that is, filters and selects videos with abnormalities and incorrect information from the video data;

[0146] The frame extraction submodule uses the Open-CV library to extract frames from the video data at a certain ratio, forming a NumPy binary file.

[0147] In one feasible implementation, it also includes:

[0148] Data segmentation submodule: The NumPy binary files form a dataset, which is then divided into training set, validation set and test set according to the specified proportions;

[0149] Parameter tuning submodule: Determines the dimensions of the input and output data, as well as the amount of training data, and then determines the number of input and output layer nodes in the LSTM layer and the FCN layer in the modeling process;

[0150] The size of the convolutional kernel, stride, and padding in the CNN layer is determined based on the image size.

[0151] The number of network layers and the number of nodes in each layer in the CNN are determined based on the size of the dataset, and the number of layers and the number of nodes in each layer of the CNN are adjusted accordingly.

[0152] The prediction model can be optimized by adjusting the number of layers and nodes in the LSTM and CNN networks. Alternatively, a random search method can be used to find the hyperparameters that minimize the loss function from a set of pre-determined hyperparameters, including combinations of layer and node numbers.

[0153] In one feasible implementation, the video acquisition submodule specifically includes:

[0154] Corpus selection submodule: Select corpora based on specific use cases and scope of use;

[0155] Differential recording module: The subject reads the corpus multiple times and records the recording. The subject's speaking speed, volume, and head posture are varied each time the recording is made.

[0156] It should be noted that the specific implementation of the auxiliary voice generation system 400 based on convolutional neural networks and long short-term memory networks in this application embodiment refers to the specific implementation of the auxiliary voice generation method based on convolutional neural networks and long short-term memory networks proposed in the first aspect of the above-mentioned application embodiment, and will not be repeated here.

[0157] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a device including one or any combination of the above-mentioned memories. The computer may be a variety of computing devices, including smart terminals and servers.

[0158] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0159] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0160] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0161] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the article or device that includes that element.

[0162] The above provides a detailed description of the assisted speech generation method and system based on convolutional neural networks and long short-term memory networks. Specific examples are used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the information push method for blockchain networks and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An assisted speech generation method based on convolutional neural networks and long short-term memory networks, characterized in that, include: A prediction model framework is constructed using three neural networks: LSTM, CNN, and FCN. Collect videos of vocalization actions and extract frames from the videos of vocalization actions; The data files after frame extraction are used to adjust the parameters of the prediction model framework and export the prediction model. The prediction model takes facial actions as input and outputs the semantic information of those facial actions. The steps for constructing a prediction model framework using three neural networks—LSTM, CNN, and FCN—specifically include the following steps: The two-layer LSTM network and the one-layer CNN network are used to process the video data of the sound action. Specifically, the LSTM layer is used to process the time series type video stream data, and the CNN layer is used to learn the local spatial features of each frame of the video and perform feature compression and extraction. The FCN component is used to output the network's prediction of semantic categories.

2. The assisted speech method based on convolutional neural networks and long short-term memory networks as described in claim 1, characterized in that, The step of using a portion of the data file after frame extraction for parameter adjustment of the prediction model framework specifically includes the following steps: The prediction model is forward-propagated, which is the process by which the neural network predicts data, and the forward propagation is used to obtain the loss value; The prediction model is backpropagated, which is the process of calculating the partial derivatives of the loss function with respect to the parameters in different layers of the neural network function. The backpropagation is used to adjust the parameters based on the value of the loss function.

3. The assisted speech method based on convolutional neural networks and long short-term memory networks as described in claim 2, characterized in that, The step of collecting video of vocalization actions and extracting frames from the video of vocalization actions specifically includes the following steps: Designate a person with a normal voice to read the relevant text aloud, and use video equipment to capture the process. Video data cleaning involves filtering and selecting videos that are abnormal or contain incorrect information. The video data is processed by extracting frames at a certain ratio using the Open-CV library, forming a Numpy binary file.

4. The assisted speech method based on convolutional neural networks and long short-term memory networks as described in claim 3, characterized in that, After the step of forming the NumPy binary file, the method further includes the following steps: The NumPy binary files form a dataset, which is then divided into training, validation, and test sets according to a set ratio. Determine the dimensions of the input and output data, as well as the amount of training data, and then determine the number of input and output layer nodes in the LSTM and FCN layers used in the modeling process. The size of the convolutional kernel, stride, and padding in the CNN layer is determined based on the image size. The number of network layers and the number of nodes in each layer in the CNN are determined based on the size of the dataset, and the number of layers and the number of nodes in each layer of the CNN are adjusted accordingly. The prediction model can be optimized by adjusting the number of layers and nodes in the LSTM and CNN networks. Alternatively, a random search method can be used to find the hyperparameters that minimize the loss function from a set of pre-determined hyperparameters, including combinations of layer and node numbers.

5. The assisted speech method based on convolutional neural networks and long short-term memory networks as described in claim 4, characterized in that, The forward propagation specifically involves inputting the same batch of NumPy data from the training set into the convolutional neural network. The matrix calculation for the convolution operation is as follows: ; The For the nth input neuron of the convolutional neural network, the The bias of the nth neuron is W, where W is the weight coefficient. The weight coefficients corresponding to the output of the i1th neuron and the j1th neuron in the next layer are described. The output result is the summation of the input signals according to the weight ratio. The above is the The final output result after nonlinear mapping is: ,Right now: ； The loss function of the backpropagation model is as follows: ； Among them, the This is the error value. For predicted values, The actual value; ； Using the error as the objective function, the weights are adjusted through multiple reverse iterations until they stabilize, thus obtaining the optimal value.

6. The assisted speech method based on convolutional neural networks and long short-term memory networks as described in claim 3, characterized in that, The method involves having a designated person with normal speech read aloud the relevant text, while simultaneously capturing the process on video using a camera. The specific steps of this method include: The corpus should be selected based on the specific usage scenario and scope. The subject reads the corpus aloud multiple times and the recordings are made. The subject's speaking speed, volume, and head posture are varied each time the recording is made.

7. An assisted speech system based on convolutional neural networks and long short-term memory networks, characterized in that, include: Framework component modules: The prediction model framework is constructed using three neural networks: LSTM, CNN, and FCN. Data acquisition module: collects video of vocalization actions and extracts frames from the video of vocalization actions; Parameter adjustment module: Uses a portion of the data file after frame extraction processing to adjust the parameters of the prediction model framework and exports the prediction model; Model output module: The prediction model takes facial action as input and outputs the semantic information of the facial action; The frame component module specifically includes: The network consists of a sub-module: a two-layer LSTM network and a CNN network are used to process video data of vocalization actions. Specifically, the LSTM layer is used to process time-series video stream data, and the CNN layer is used to learn the local spatial features of each frame of the video and perform feature compression and extraction. The FCN component is used to output the network's prediction of semantic categories.

8. An auxiliary sound-generating electronic device based on convolutional neural networks and long short-term memory networks, characterized in that, It includes a processor and a memory; the memory is used to store a computer program, which, when executed by the processor, causes the electronic device to perform the assisted speech method based on a convolutional neural network and a long short-term memory network as described in any one of claims 1-6.

9. An assisted speech-readable storage medium based on convolutional neural networks and long short-term memory networks, characterized in that, A program or instructions are stored on a readable storage medium, which, when executed by a processor, implement the various processes of the assisted speech method based on a convolutional neural network and a long short-term memory network as described in any one of claims 1-6.

Images