Method and device for converting voice into sign language animation, equipment, medium and program product

Abstract

The embodiment of the invention provides a voice-to-sign language animation method and device, equipment, a medium and a program product, and relates to the field of audios and videos. In the method, a plurality of standard voice segments are obtained by carrying out preprocessing, segmentation and correction processing on an obtained initial voice; performing voice-to-text processing on the standard voice segment to obtain a standard text; and performing text-to-image processing on the standard text to obtain a sign language image, and finally generating a sign language animation according to the sign language image. According to the scheme, the voice is segmented to obtain the voice segments, the sign language image is obtained through voice-to-text and text-to-image processing, the sign language animation is generated according to the sign language image, the voice is converted into the sign language animation, and the hearing-impaired person can be assisted in completing service handling.

Description

Technical Field

[0001] The present application relates to the field of audio and video, and in particular to a method, device, equipment, medium and program product for converting speech to sign language animation. Background Art

[0002] When people conduct business in financial institutions, staff usually provide guidance and assist customers in conducting business. It is essential for staff to talk to customers during the guidance process. Since hearing-impaired people have hearing impairments, how to assist hearing-impaired people in conducting business is a concern for people.

[0003] In the prior art, in order to assist the hearing-impaired in handling business, the staff is usually trained in sign language, and the staff uses sign language to communicate with the hearing-impaired to complete the business handling.

[0004] However, when the staff does not understand sign language, there is no way to assist the hearing-impaired to complete business transactions, so there is an urgent need for a speech-to-sign language animation method that can assist the hearing-impaired to complete business transactions when the staff does not understand sign language. Summary of the invention

[0005] The present application provides a method, device, equipment, medium and program product for converting speech into sign language animation, which is used to solve the problem that a method for converting speech into sign language animation is urgently needed in the prior art, and can assist hearing-impaired people to complete business transactions when the staff does not understand sign language.

[0006] In a first aspect, the present application provides a method for converting speech to sign language animation, comprising:

[0007] Preprocessing and segmenting the acquired initial speech to obtain multiple initial speech segments;

[0008] Processing the multiple initial speech segments according to a preset speech database and a speech correction model to obtain multiple standard speech segments, wherein the speech correction model is a pre-trained neural network model for correcting speech segments;

[0009] According to a speech-to-text model, the plurality of standard speech segments are converted to obtain a standard text corresponding to each of the standard speech segments, wherein the speech-to-text model is a pre-trained neural network model for obtaining text according to speech;

[0010] According to a preset image database and a text-to-sign language image model, each of the standard texts is converted to obtain a sign language image corresponding to each of the standard speech segments, wherein the text-to-sign language image model is a pre-trained neural network model that obtains images from texts;

[0011] A sign language animation is generated according to the sign language image corresponding to each of the standard speech segments.

[0012] In a second aspect, the present application provides a speech-to-sign language animation device, comprising:

[0013] Processing modules for:

[0014] Preprocessing and segmenting the acquired initial speech to obtain multiple initial speech segments;

[0015] Processing the multiple initial speech segments according to a preset speech database and a speech correction model to obtain multiple standard speech segments, wherein the speech correction model is a pre-trained neural network model for correcting speech segments;

[0016] Conversion modules for:

[0017] According to a speech-to-text model, the plurality of standard speech segments are converted to obtain a standard text corresponding to each of the standard speech segments, wherein the speech-to-text model is a pre-trained neural network model for obtaining text according to speech;

[0018] According to a preset image database and a text-to-sign language image model, each of the standard texts is converted to obtain a sign language image corresponding to each of the standard speech segments, wherein the text-to-sign language image model is a pre-trained neural network model that obtains images from texts;

[0019] A generating module is used to generate a sign language animation according to the sign language image corresponding to each of the standard speech segments.

[0020] In a third aspect, the present application provides an electronic device, including:

[0021] Processor, memory, communication interface;

[0022] The memory is used to store executable instructions of the processor;

[0023] Wherein, the processor is configured to execute the speech-to-sign language animation method described in any one of the first aspects by executing the executable instructions.

[0024] In a fourth aspect, the present application provides a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the speech-to-sign language animation method described in any one of the first aspects.

[0025] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the speech-to-sign language animation method described in any one of the first aspects.

[0026] The speech-to-sign language animation method, device, equipment, medium and program product provided in the present application obtain multiple standard speech segments by preprocessing, segmenting and correcting the acquired initial speech. The standard speech segments are then processed by speech-to-text to obtain standard text; the standard text is then processed by text-to-image to obtain a sign language image, and finally a sign language animation is generated based on the sign language image. This solution achieves the conversion of speech into sign language animation by segmenting the speech to obtain speech segments, and then performing speech-to-text and text-to-image processing to obtain a sign language image, and then generating a sign language animation based on the sign language image, which can assist the hearing-impaired in completing business transactions. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0028] Figure 1 A flowchart of the first embodiment of the method for converting speech to sign language animation provided by the present application;

[0029] Figure 2 A flowchart of the second embodiment of the method for converting speech to sign language animation provided by the present application;

[0030] Figure 3 A flowchart of Embodiment 3 of the method for converting speech to sign language animation provided in this application;

[0031] Figure 4 A flowchart of a fourth embodiment of the method for converting speech to sign language animation provided in this application;

[0032] Figure 5 A schematic diagram of the structure of an embodiment of a speech-to-sign language animation device provided in this application;

[0033] Figure 6 A schematic diagram of the structure of an electronic device provided in this application.

[0034] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0035] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0036] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0038] It should be noted that the speech-to-sign language animation method, device, equipment, medium and program product provided in this application can be used in the audio and video field, and can also be used in any field other than audio and video. The application field of the speech-to-sign language animation method, device, equipment, medium and program product in this application is not limited.

[0039] In financial institutions, in order to improve the efficiency of people handling business, staff are usually required to assist people in handling business. In the process of assisting in handling business, staff are required to communicate with customers. When the customer is hearing-impaired, staff can only communicate with the customer using sign language.

[0040] In the prior art, in order to meet the needs of hearing-impaired people in handling business, staff must be trained in sign language. Therefore, when the staff does not understand sign language, there is no way to assist the hearing-impaired people in completing business. Therefore, a method of converting speech to sign language animation is urgently needed, which can help the hearing-impaired people complete business when the staff does not understand sign language.

[0041] In view of the problems existing in the prior art, the inventor discovered during the research on the method of converting speech to sign language animation that in order to assist the hearing-impaired to complete business transactions when the staff does not understand sign language, the speech spoken by the staff can be converted into sign language animation, and the customer can view the sign language animation to process the business. In order to realize the conversion of speech to sign language animation, the speech can be preprocessed, segmented and corrected to obtain multiple standard speech segments. The standard speech segments are then processed by speech-to-text to obtain standard text; the standard text is then processed by text-to-image to obtain a sign language image, and finally a sign language animation is generated based on the sign language image. Based on the above-mentioned inventive concept, the speech-to-sign language animation scheme in this application is designed.

[0042] The executor of the speech-to-sign language animation method in the present application may be a computer, or a server, a terminal device, etc. The present application does not limit it and the following description will be made using a computer as an example.

[0043] The following is an example of the application scenario of the speech-to-sign language animation provided by this application.

[0044] For example, in this application scenario, a hearing-impaired person needs to conduct business at a financial institution. The staff and the hearing-impaired person conduct business in front of a computer. The staff speaks a voice, and the computer can obtain the initial voice through a microphone.

[0045] Then, the computer performs preprocessing and segmentation processing on the initial speech to obtain a plurality of initial speech segments; and then processes the plurality of initial speech segments according to a preset speech database and a speech correction model to obtain a plurality of standard speech segments.

[0046] The computer then converts multiple standard speech segments according to the speech-to-text model to obtain a standard text corresponding to each standard speech segment. According to the preset image database and the text-to-sign language image model, each standard text is converted to obtain a sign language image corresponding to each standard speech segment.

[0047] The computer generates a sign language animation based on the sign language image corresponding to each standard voice segment and displays the sign language animation on the screen. The hearing-impaired people can watch the sign language animation and conduct business according to the meaning of the sign language animation.

[0048] It should be noted that the above scenario is only an example of an application scenario provided by an embodiment of the present application. The embodiment of the present application does not limit the actual form of the various devices included in the scenario. In the specific application of the solution, it can be set according to actual needs.

[0049] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0050] Figure 1 This is a flow chart of the first embodiment of the speech-to-sign language animation method provided in this application. This embodiment of the application describes the situation where a computer performs speech-to-text conversion and text-to-image conversion to obtain a sign language image after initial speech segmentation, and then generates a sign language animation. The method in this embodiment can be implemented by software, hardware, or a combination of software and hardware. Figure 1 As shown, the speech-to-sign language animation method specifically includes the following steps:

[0051] S101: Preprocessing and segmenting the acquired initial speech to obtain a plurality of initial speech segments.

[0052] In this step, in order to realize the voice-to-animation conversion, the acquired initial voice needs to be preprocessed and segmented to obtain multiple initial voice segments.

[0053] Preprocessing the initial speech can reduce the impact of accents and noise, and segmentation can process the speech more carefully and improve animation accuracy.

[0054] S102: Processing a plurality of initial speech segments according to a preset speech database and a speech correction model to obtain a plurality of standard speech segments.

[0055] In this step, after the computer obtains the initial speech segments, some of the initial speech segments are non-standard, which will affect the accuracy of the generated animation. Therefore, according to the preset speech database and speech correction model, multiple initial speech segments are processed to obtain multiple standard speech segments.

[0056] Among them, the speech correction model is a pre-trained neural network model for correcting speech fragments.

[0057] Specifically, for each initial voice segment, if the initial voice segment exists in the preset voice database, the initial voice segment is used as the standard voice segment. Since the voice segments stored in the preset voice database are all standard voice segments and can be accurately converted into text, the initial voice segment in the preset voice database is a standard voice segment.

[0058] For each initial speech segment, if the initial speech segment does not exist in the preset speech database, it means that the initial speech segment is not standard and cannot be accurately converted into text. Then the initial speech segment is input into the speech correction model to obtain the standard speech segment.

[0059] It should be noted that when determining whether the initial voice segment exists in the preset voice database, the initial voice segment can be input into the preset voice database for query. When querying, all initial voice segments can be input into the preset voice database for query to improve efficiency. Then, the initial voice segment in the preset voice database is marked as 1, and the initial voice segment not in the preset voice database is marked as 0. Then, the initial voice segment marked as 0 is input into the voice correction model to obtain the standard voice segment.

[0060] By converting the initial speech segments into standard speech segments, the accuracy of the speech segments can be improved, thereby improving the accuracy of the sign language animation.

[0061] S103: According to the speech-to-text model, multiple standard speech segments are converted to obtain a standard text corresponding to each standard speech segment.

[0062] In this step, after the computer obtains the standard speech segment, in order to generate the sign language image, it is necessary to convert the multiple standard speech segments according to the speech-to-text model to obtain the standard text corresponding to each standard speech segment.

[0063] Among them, the speech-to-text model is a pre-trained neural network model that obtains text based on speech.

[0064] Specifically, for each standard speech segment, the standard speech segment is input into a speech-to-text model to obtain an initial text corresponding to the standard speech segment.

[0065] According to the preset text mapping relationship and the initial text corresponding to each standard voice segment, the standard text corresponding to each standard voice segment is determined.

[0066] Since some initial texts are not standard, which will affect the accuracy of the generated voice animation, the initial texts are mapped to standard texts according to the preset text mapping relationship.

[0067] Exemplarily, the text mapping relationship is: text 1, text 2, text 3 and text 4 correspond to standard text A. The initial text corresponding to the standard voice segment is text 2, so the standard text corresponding to the standard voice segment is standard text A.

[0068] By converting standard speech segments into standard text through text mapping relationships, the matching degree between text and speech segments can be improved, thereby improving the accuracy of sign language animation.

[0069] S104: Convert each standard text according to a preset image database and a text-to-sign language image model to obtain a sign language image corresponding to each standard speech segment.

[0070] In this step, after the computer obtains the standard text, in order to generate the sign language animation, it is necessary to convert each standard text according to the preset image database and the text-to-sign language image model to obtain the sign language image corresponding to each standard speech segment.

[0071] Among them, the text-to-sign language image model is a pre-trained neural network model that obtains images based on text.

[0072] Specifically, for each standard text corresponding to a standard voice segment, if an image corresponding to the standard text exists in the preset image database, the image is used as the sign language image corresponding to the standard voice segment. Since the preset image database stores sign language images and uses text as an index, it can be determined whether there is an image corresponding to the standard text in the preset image database based on the standard text.

[0073] It should be noted that when using a preset image database, the data in the preset database can be loaded into the memory to improve efficiency.

[0074] For each standard text corresponding to a standard speech segment, if there is no sign language image corresponding to the standard text in the preset image database, the standard text is input into the text-to-sign language image model to obtain the sign language image corresponding to the standard speech segment.

[0075] It should be noted that after converting the standard text into a sign language image through the text-to-sign language image model, the standard text can be used as an index of the sign language image and the sign language image can be stored in a preset image database.

[0076] By converting standard text into sign language images through a preset image database and a text-to-sign language image model, the matching degree between sign language images and speech segments is improved, thereby improving the accuracy of sign language animation.

[0077] S105: Generate a sign language animation according to the sign language image corresponding to each standard speech segment.

[0078] In this step, after the computer obtains the sign language image, it generates a sign language animation according to the sign language image corresponding to each standard speech segment.

[0079] The computer can play sign language animations, and the hearing-impaired people can handle business after watching the sign language animations.

[0080] The speech-to-sign language animation method provided in this embodiment obtains a plurality of standard speech segments by preprocessing, segmenting and correcting the acquired initial speech. The standard speech segments are then processed by speech-to-text to obtain standard text; the standard text is then processed by text-to-image to obtain a sign language image, and finally a sign language animation is generated based on the sign language image. This solution achieves the conversion of speech into sign language animation by segmenting speech to obtain speech segments, and then performing speech-to-text and text-to-image processing to obtain a sign language image, and then generating a sign language animation based on the sign language image, which can assist the hearing-impaired in completing business transactions.

[0081] Figure 2 This is a flow chart of the second embodiment of the speech-to-sign language animation method provided by the present application. Based on the above embodiment, the present embodiment of the present application describes the situation where a computer pre-processes and segments the initial speech to obtain multiple initial speech segments. Figure 2 As shown, the speech-to-sign language animation method specifically includes the following steps:

[0082] S201: Perform accent optimization and noise removal processing on the initial speech to obtain the target speech.

[0083] In this step, after the computer obtains the initial speech, in order to improve the accuracy of the sign language animation, it is necessary to perform accent optimization and noise removal on the initial speech to obtain the target speech.

[0084] It should be noted that the deep learning model can be used to process the initial speech to achieve accent optimization and noise removal. Accent optimization includes converting non-Mandarin into Mandarin.

[0085] S202: Using a plurality of segmentation strategies to segment the target speech respectively, and obtaining speech segments to be selected corresponding to each segmentation strategy.

[0086] In this step, after the computer obtains the target speech, since the target speech is long, the accuracy of the converted text and image is low, which will also reduce the accuracy of the sign language animation. Therefore, it is necessary to adopt multiple segmentation strategies to segment the target speech separately and obtain the candidate speech segments corresponding to each segmentation strategy.

[0087] Specifically, the plurality of segmentation strategies include at least two of a first strategy, a second strategy, and a third strategy.

[0088] The first strategy is: according to the speech energy at each moment in the target speech, the first moment is determined, and the target speech is segmented according to each first moment. Since the speech energy of the paused part of the speech is low, the moment when the speech energy is less than the preset energy can be taken as the first moment, and then segmented at each first moment to obtain the selected speech segment corresponding to the first strategy.

[0089] It should be noted that the preset energy may be 5 milliwatts, 7 milliwatts, 1 watt, etc. The embodiment of the present application does not limit the preset energy and may be determined according to actual conditions.

[0090] The second strategy is: determine the second moment according to the speech frequency at each moment in the target speech, and segment the target speech according to each second moment; since the speech frequency of the paused part in the speech is relatively low, the moment when the speech energy is less than the preset frequency can be used as the second moment, and then segment at each second moment to obtain the selected speech segment corresponding to the second strategy.

[0091] For example, the text corresponding to the target speech is "Please enter information A, information B, information C, and click the Done button after entering", where a space indicates a pause. The target speech is segmented using speech frequency to obtain the selected speech segments "Please enter information A", "Information B", "Information C", and "Click the Done button after entering".

[0092] It should be noted that the preset frequency may be 20 Hz, 50 Hz, 70 Hz, etc. The embodiment of the present application does not limit the preset frequency and may be determined according to actual conditions.

[0093] It should be noted that the third moment may also be determined according to acoustic features such as the pitch and sound pressure level at each moment in the target speech, and then the target speech may be segmented according to each third moment.

[0094] The third strategy is: use the speech segmentation model to segment the target speech; input the target speech into the speech segmentation model to obtain the selected speech segments corresponding to the third strategy, and the speech segmentation model is a pre-trained neural network model that segments speech into speech segments. The speech segmentation model analyzes the semantics in the target speech and segments it according to the semantics.

[0095] For example, the text corresponding to the target speech is "Please enter information A, information B, information C, and click the Done button after entering", where a space indicates a pause. The target speech is segmented using the speech segmentation model to obtain the candidate speech segments "Please enter information A, information B, information C" and "Click the Done button after entering".

[0096] By segmenting the target speech according to a variety of segmentation strategies such as speech energy, speech frequency and speech segmentation model, the accuracy of the selected speech segments can be improved, thereby improving the accuracy of the speech animation.

[0097] S203: Determine a plurality of initial speech segments from the candidate speech segments corresponding to all segmentation strategies.

[0098] In this step, after the computer obtains the speech segments to be selected, since the speech segments to be selected corresponding to some segmentation strategies are not accurate, it is necessary to determine multiple initial speech segments from the speech segments to be selected corresponding to all segmentation strategies.

[0099] Specifically, the hash algorithm is used to process the selected speech segments corresponding to each segmentation strategy to obtain the hash value corresponding to each segmentation strategy. In order to determine whether the selected speech segments corresponding to different segmentation strategies are the same, it is necessary to use the hash algorithm to process each selected speech segment corresponding to each segmentation strategy, and the hash value of each selected speech segment is the hash value corresponding to the segmentation strategy.

[0100] It should be noted that the hash algorithm can be MD5 (Message-Digest Algorithm 5) algorithm, SHA-256 (Secure Hash Algorithm 256) algorithm, BLAKE2 algorithm, etc. The embodiment of the present application does not limit the hash algorithm and can be determined according to actual conditions.

[0101] If there is a first target segmentation strategy with the same hash value among all segmentation strategies, it means that the selected speech segment obtained by segmenting using the first target segmentation strategy is more accurate, and the selected speech segment corresponding to the first target segmentation strategy is used as the initial speech segment.

[0102] It should be noted that the same hash value means that for each candidate voice segment in one first target segmentation strategy, there is a candidate voice segment with the same hash value in each other first target segmentation strategy.

[0103] If there is no first target segmentation strategy with the same hash value among all segmentation strategies, that is, the hash values ​​of the selected speech segments corresponding to any two segmentation strategies are different, it means that the selected speech segments obtained by different segmentation strategies are different. Because standard speech is generally the speech corresponding to words, and the speech length is shorter, the more speech segments obtained by segmentation, the more standard the speech segments. The selected speech segment corresponding to the second target segmentation strategy with the most selected speech segments among all segmentation strategies is used as the initial speech segment.

[0104] It should be noted that the different hash values ​​of the candidate voice segments corresponding to the two segmentation strategies means that: among the candidate voice segments corresponding to the two segmentation strategies, there is at least one candidate voice segment with a different hash value.

[0105] Exemplarily, the first segmentation strategy corresponds to the selected voice segments A1, A2, A3, and A4; the second segmentation strategy corresponds to the selected voice segments A1, A2, and A5; the third segmentation strategy corresponds to the selected voice segments A1, A2, A3, A6, and A7; there is no segmentation strategy with the same hash value among the three segmentation strategies; the third segmentation strategy has the most selected voice segments, so A1, A2, A3, A6, and A7 are the initial voice segments.

[0106] By determining the initial voice segment according to the hash value and the number of voice segments to be selected, the accuracy of the initial voice segment is improved.

[0107] The speech-to-sign language animation method provided in this embodiment obtains a target speech by performing accent optimization and noise removal processing on the initial speech, and then adopts a plurality of segmentation strategies to respectively segment the target speech, and then determines a plurality of initial speech segments from the candidate speech segments obtained by each segmentation strategy, thereby effectively improving the accuracy of the initial speech segments and thus improving the accuracy of the sign language animation.

[0108] Figure 3 This is a flow chart of the third embodiment of the method for converting speech to sign language animation provided by this application. Based on the above embodiments, this embodiment of the application describes the situation where a computer generates a sign language animation based on a sign language image. Figure 3 As shown, the speech-to-sign language animation method specifically includes the following steps:

[0109] S301: Sort the sign language images corresponding to each standard speech segment in the order of the time period of the speech segment from early to late, to obtain an initial sign language image sequence.

[0110] In this step, after the computer obtains the sign language images, in order to generate the sign language animation, the sign language images need to be sorted. The sign language images corresponding to each standard speech segment are sorted according to the time period of the speech segment from early to late to obtain the initial sign language image sequence.

[0111] S302: Perform frame interpolation processing on the initial sign language image sequence to obtain a target sign language image sequence.

[0112] In this step, after the computer obtains the initial sign language image sequence, if the initial sign language image sequence is directly used to render the sign language animation, the sign language animation will not be smooth. The initial sign language image sequence needs to be interpolated to obtain the target sign language image sequence.

[0113] Specifically, for every two adjacent sign language images in the initial sign language image sequence, the pixel difference of each pixel at the same position in the two sign language images is calculated; and then the number of pixels whose pixel difference is greater than a preset pixel threshold is determined. If the number of pixels is greater than the preset number threshold, a new sign language image is generated based on the two sign language images, and the new sign language image is inserted between the two sign language images. A new target sign language image sequence is obtained, and this process is repeated until the number of pixels is less than or equal to the preset number threshold, and the new target sign language image sequence is used as the target sign language image sequence.

[0114] It should be noted that the preset pixel threshold can be 10, 20, 50, etc., and the preset quantity threshold can be 20, 30, 100, 1000, etc. The embodiment of the present application does not limit the preset pixel threshold and the preset quantity threshold, which can be determined according to actual conditions.

[0115] It should be noted that the method of generating a new sign language image based on two sign language images is: for each two pixel points at the same position in the two sign language images, the average pixel value of the two pixel points is calculated to obtain the pixel average value, and then the pixel value of the pixel point at the position in the new sign language image is determined as the pixel average value.

[0116] S303: Rendering the target sign language image sequence to obtain a sign language animation.

[0117] In this step, after obtaining the target sign language image sequence, the computer renders the target sign language image sequence to obtain a sign language animation.

[0118] The speech-to-sign language animation method provided in this embodiment obtains a target sign language image sequence by performing frame interpolation processing on an initial sign language image sequence; and then renders the target sign language image sequence to obtain a sign language animation, thereby effectively improving the fluency of the sign language animation.

[0119] Figure 4 The flowchart of the fourth embodiment of the speech-to-sign language animation method provided in this application is based on the above embodiment. This embodiment describes the situation that a computer trains an initial neural network model according to a training data set to obtain a speech-to-text model. Figure 4 As shown, the speech-to-sign language animation method specifically includes the following steps:

[0120] S401: Preprocess and segment each training speech in the training data set to obtain a plurality of first speech segments.

[0121] It should be noted that the training voice can be the voice spoken by the staff when assisting the user in handling business, or it can be the voice related to business handling collected from other channels.

[0122] It should be noted that this step is similar to the implementation process of Example 2 and will not be described in detail here.

[0123] S402: Perform conversion processing on each first voice segment to obtain a second voice segment.

[0124] In this step, after the computer obtains the first voice segment, in order to increase the amount of training data and increase the applicability of the speech-to-text model, it is necessary to perform conversion processing on each first voice segment to obtain a second voice segment.

[0125] It should be noted that the way of converting each first voice segment can be: adding noise to the first voice segment, converting the first voice segment into a dialect, lowering the volume of the first voice segment, etc. The embodiment of the present application does not limit the way of converting the first voice segment, which can be determined according to actual conditions.

[0126] S403: All the first voice segments and all the second voice segments form a training voice segment set, and the real text of each first voice segment and each second voice segment is obtained.

[0127] In this step, after the computer obtains the second voice segment, in order to train the speech-to-text image, all the first voice segments and all the second voice segments are combined into a training voice segment set, and the real text of each first voice segment and each second voice segment is obtained.

[0128] It should be noted that the method of obtaining the real text of each first voice segment and each second voice segment can be: the staff inputs the real text of each first voice segment and each second voice segment into the computer, and the computer can obtain the real text of each first voice segment and each second voice segment. It can also be: the staff uses the terminal device to send the real text of each first voice segment and each second voice segment to the computer, and the computer can obtain the real text of each first voice segment and each second voice segment.

[0129] It should be noted that, since the second voice segment is converted from the first voice segment, and there may be voice segments with the same meaning in the first voice segment, there may be a situation where the real texts of multiple voice segments are the same.

[0130] S404: Acquire a speech segment from the training speech segment set.

[0131] S405: Input the speech segment into the initial neural network model to obtain training text.

[0132] S406: Calculate the loss value based on the training text and the real text of the speech segment.

[0133] In the above steps, after the computer obtains the training speech segment set, it obtains a speech segment from the training speech segment set, inputs the speech segment into the initial neural network model, and obtains the training text. Then, based on the training text and the real text of the speech segment, the loss value is calculated using the loss function.

[0134] It should be noted that the loss function used to calculate the loss value can be an L1 norm loss function, a mean square error loss function, a cross entropy loss function, etc. The embodiment of the present application does not limit the loss function used to calculate the loss value, and it can be determined according to actual conditions.

[0135] S407: Update the initial neural network model according to the loss value to obtain a trained neural network model.

[0136] In this step, after the computer obtains the loss value, it updates the initial neural network model according to the loss value to obtain a trained neural network model.

[0137] S408: Update the number of training times.

[0138] S409: Determine whether the updated training number is equal to the preset number; if the updated training number is equal to the preset number, execute step S410; if the updated training number is less than the preset number, execute step S411.

[0139] In the above steps, after the computer obtains the trained neural network model, in order to determine whether training needs to continue, it is necessary to update the number of training times, that is, add one to the number of training times, and then determine whether the updated number of training times is equal to the preset number of times.

[0140] It should be noted that the preset number of times may be 500, 1000, 10000, 100000, etc. The embodiment of the present application does not limit the preset number of times and may be determined according to actual conditions.

[0141] S410: Using the trained neural network model as a speech-to-text model.

[0142] In this step, if the computer determines that the updated number of training times is equal to the preset number of times, it means that the training is completed and there is no need to continue training. The trained neural network model is used as the speech-to-text model.

[0143] S411: Use the trained neural network model as a new initial neural network model and execute step S404.

[0144] In this step, if the computer determines that the updated training times are less than the preset times, it means that the training is not completed and needs to be continued. The trained neural network model is used as the new initial neural network model and step S404 is executed.

[0145] That is, a new speech segment is obtained from the set of training speech segments, and the new speech segment is input into the new initial neural network model to obtain a new training text; a new loss value is calculated based on the new training text and the real text of the new speech segment; the new initial neural network model is updated according to the new loss value to obtain a new trained neural network model; the number of training times is updated again, and if the updated number of training times is less than the preset number, this process is repeated until the updated number of training times is equal to the preset number, and the new trained neural network model is used as the speech-to-text model.

[0146] The speech-to-sign language animation method provided in this embodiment can improve the accuracy and diversity of the training data and the accuracy of the speech-to-text model by first preprocessing and segmenting each training speech in the training data set to obtain a first speech segment, converting the first speech segment to obtain a second speech segment, and combining all the first speech segments and all the second speech segments into a training speech segment set.

[0147] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0148] Figure 5 A schematic diagram of the structure of an embodiment of a speech-to-sign language animation device provided in this application; Figure 5 As shown, the speech-to-sign language animation device 50 comprises:

[0149] The processing module 51 is used for:

[0150] Preprocessing and segmenting the acquired initial speech to obtain multiple initial speech segments;

[0151] Processing the multiple initial speech segments according to a preset speech database and a speech correction model to obtain multiple standard speech segments, wherein the speech correction model is a pre-trained neural network model for correcting speech segments;

[0152] The conversion module 52 is used for:

[0153] According to a speech-to-text model, the plurality of standard speech segments are converted to obtain a standard text corresponding to each of the standard speech segments, wherein the speech-to-text model is a pre-trained neural network model for obtaining text according to speech;

[0154] According to a preset image database and a text-to-sign language image model, each of the standard texts is converted to obtain a sign language image corresponding to each of the standard speech segments, wherein the text-to-sign language image model is a pre-trained neural network model that obtains images from texts;

[0155] The generating module 53 is used to generate a sign language animation according to the sign language image corresponding to each of the standard speech segments.

[0156] Furthermore, the processing module 51 is specifically used for:

[0157] Performing accent optimization processing and noise removal processing on the initial speech to obtain a target speech;

[0158] Using a plurality of segmentation strategies to segment the target speech respectively, and obtaining speech segments to be selected corresponding to each segmentation strategy;

[0159] The multiple initial speech segments are determined from the candidate speech segments corresponding to all the segmentation strategies.

[0160] Further, the plurality of segmentation strategies include at least two of a first strategy, a second strategy, and a third strategy;

[0161] The first strategy is: determining a first moment according to the speech energy at each moment in the target speech, and segmenting the target speech according to each of the first moments;

[0162] The second strategy is: determining a second moment according to the speech frequency at each moment in the target speech, and segmenting the target speech according to each second moment;

[0163] The third strategy is: segmenting the target speech using a speech segmentation model;

[0164] The speech segmentation model is a pre-trained neural network model that segments speech into speech segments.

[0165] Furthermore, the processing module 51 is further configured to:

[0166] Using a hash algorithm to process the selected speech segments corresponding to each segmentation strategy respectively, to obtain a hash value corresponding to each segmentation strategy;

[0167] If there is a first target segmentation strategy with the same hash value among all the segmentation strategies, taking the selected voice segment corresponding to the first target segmentation strategy as the initial voice segment;

[0168] If there is no first target segmentation strategy with the same hash value among all the segmentation strategies, the speech segments to be selected corresponding to the second target segmentation strategy with the most speech segments to be selected among all the segmentation strategies are taken as the initial speech segments.

[0169] Furthermore, the processing module 51 is further configured to:

[0170] For each of the initial voice segments, if the initial voice segment exists in the preset voice database, the initial voice segment is used as a standard voice segment;

[0171] For each of the initial speech segments, if the initial speech segment does not exist in the preset speech database, the initial speech segment is input into the speech correction model to obtain a standard speech segment.

[0172] Furthermore, the conversion module 52 is specifically used for:

[0173] For each of the standard speech segments, inputting the standard speech segment into the speech-to-text model to obtain an initial text corresponding to the standard speech segment;

[0174] According to a preset text mapping relationship and an initial text corresponding to each of the standard voice segments, a standard text corresponding to each of the standard voice segments is determined.

[0175] Furthermore, the conversion module 52 is further configured to:

[0176] For each standard text corresponding to the standard voice segment, if there is an image corresponding to the standard text in the preset image database, the image is used as the sign language image corresponding to the standard voice segment;

[0177] For each standard text corresponding to the standard speech segment, if the sign language image corresponding to the standard text does not exist in the preset image database, the standard text is input into the text-to-sign language image model to obtain the sign language image corresponding to the standard speech segment.

[0178] Furthermore, the generating module 53 is specifically used for:

[0179] According to the time period of the speech segment from early to late, the sign language images corresponding to each standard speech segment are sorted to obtain an initial sign language image sequence;

[0180] Performing frame interpolation processing on the initial sign language image sequence to obtain a target sign language image sequence;

[0181] The target sign language image sequence is rendered to obtain the sign language animation.

[0182] The speech-to-sign language animation device provided in this embodiment is used to execute the technical solution in any of the aforementioned method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0183] Figure 6 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 6 As shown, the electronic device 60 includes:

[0184] Processor 61, memory 62, and communication interface 63;

[0185] The memory 62 is used to store executable instructions of the processor 61;

[0186] The processor 61 is configured to execute the technical solution in any of the aforementioned method embodiments by executing the executable instructions.

[0187] Optionally, the memory 62 may be independent or integrated with the processor 61 .

[0188] Optionally, when the memory 62 is a device independent of the processor 61, the electronic device 60 may further include:

[0189] The bus 64 , the memory 62 and the communication interface 63 are connected to the processor 61 via the bus 64 and communicate with each other. The communication interface 63 is used to communicate with other devices.

[0190] Optionally, the communication interface 63 may be implemented by a transceiver. The communication interface is used to implement communication between the database access device and other devices (such as a client, a read-write library, and a read-only library). The memory may include a random access memory (RAM) and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0191] The bus 64 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0192] The above-mentioned processor can be a general-purpose processor, including a central processing unit CPU, a network processor (NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0193] The electronic device is used to execute the technical solution in any of the aforementioned method embodiments, and its implementation principle and technical effect are similar and will not be repeated here.

[0194] An embodiment of the present application also provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the technical solution provided in any of the aforementioned embodiments is implemented.

[0195] An embodiment of the present application also provides a computer program product, including a computer program, which is used to implement the technical solution provided by any of the aforementioned method embodiments when executed by a processor.

[0196] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for converting speech into sign language animation, characterized in that: include: Preprocessing and segmenting the acquired initial speech to obtain multiple initial speech segments; Processing the multiple initial speech segments according to a preset speech database and a speech correction model to obtain multiple standard speech segments, wherein the speech correction model is a pre-trained neural network model for correcting speech segments; According to a speech-to-text model, the plurality of standard speech segments are converted to obtain a standard text corresponding to each of the standard speech segments, wherein the speech-to-text model is a pre-trained neural network model for obtaining text according to speech; According to a preset image database and a text-to-sign language image model, each of the standard texts is converted to obtain a sign language image corresponding to each of the standard speech segments, wherein the text-to-sign language image model is a pre-trained neural network model that obtains images from texts; A sign language animation is generated according to the sign language image corresponding to each of the standard speech segments.

2. The method according to claim 1, characterized in that The obtained initial speech is preprocessed and segmented to obtain a plurality of initial speech segments, including: Performing accent optimization processing and noise removal processing on the initial speech to obtain a target speech; Using a plurality of segmentation strategies to segment the target speech respectively, and obtaining speech segments to be selected corresponding to each segmentation strategy; The multiple initial speech segments are determined from the candidate speech segments corresponding to all the segmentation strategies.

3. The method according to claim 2, characterized in that The plurality of segmentation strategies include at least two of a first strategy, a second strategy, and a third strategy; The first strategy is: determining a first moment according to the speech energy at each moment in the target speech, and segmenting the target speech according to each of the first moments; The second strategy is: determining a second moment according to the speech frequency at each moment in the target speech, and segmenting the target speech according to each second moment; The third strategy is: segmenting the target speech using a speech segmentation model; The speech segmentation model is a pre-trained neural network model that segments speech into speech segments.

4. The method according to claim 2, characterized in that: The step of determining the plurality of initial speech segments from the selected speech segments corresponding to all the segmentation strategies comprises: Using a hash algorithm to process the selected speech segments corresponding to each segmentation strategy respectively, to obtain a hash value corresponding to each segmentation strategy; If there is a first target segmentation strategy with the same hash value among all the segmentation strategies, taking the selected voice segment corresponding to the first target segmentation strategy as the initial voice segment; If there is no first target segmentation strategy with the same hash value among all the segmentation strategies, the speech segments to be selected corresponding to the second target segmentation strategy with the most speech segments to be selected among all the segmentation strategies are taken as the initial speech segments.

5. The method according to any one of claims 1 to 4, characterized in that: The method of processing the plurality of initial speech segments according to the preset speech database and the speech correction model to obtain a plurality of standard speech segments includes: For each of the initial voice segments, if the initial voice segment exists in the preset voice database, the initial voice segment is used as a standard voice segment; For each of the initial speech segments, if the initial speech segment does not exist in the preset speech database, the initial speech segment is input into the speech correction model to obtain a standard speech segment.

6. The method according to any one of claims 1 to 4, characterized in that: The converting process of the plurality of standard speech segments according to the speech-to-text model to obtain a standard text corresponding to each of the standard speech segments includes: For each of the standard speech segments, inputting the standard speech segment into the speech-to-text model to obtain an initial text corresponding to the standard speech segment; According to a preset text mapping relationship and an initial text corresponding to each of the standard voice segments, a standard text corresponding to each of the standard voice segments is determined.

7. The method according to any one of claims 1 to 4, characterized in that: The converting process is performed on each of the standard texts according to the preset image database and the text-to-sign language image model to obtain a sign language image corresponding to each of the standard speech segments, including: For each standard text corresponding to the standard voice segment, if there is an image corresponding to the standard text in the preset image database, the image is used as the sign language image corresponding to the standard voice segment; For each standard text corresponding to the standard speech segment, if the sign language image corresponding to the standard text does not exist in the preset image database, the standard text is input into the text-to-sign language image model to obtain the sign language image corresponding to the standard speech segment.

8. The method according to any one of claims 1 to 4, characterized in that: Generating a sign language animation according to the sign language image corresponding to each of the standard speech segments includes: According to the time period of the speech segment from early to late, the sign language images corresponding to each standard speech segment are sorted to obtain an initial sign language image sequence; Performing frame interpolation processing on the initial sign language image sequence to obtain a target sign language image sequence; The target sign language image sequence is rendered to obtain the sign language animation.

9. A speech-to-sign language animation device, characterized in that: include: Processing modules for: Preprocessing and segmenting the acquired initial speech to obtain multiple initial speech segments; Processing the multiple initial speech segments according to a preset speech database and a speech correction model to obtain multiple standard speech segments, wherein the speech correction model is a pre-trained neural network model for correcting speech segments; Conversion modules for: According to a speech-to-text model, the plurality of standard speech segments are converted to obtain a standard text corresponding to each of the standard speech segments, wherein the speech-to-text model is a pre-trained neural network model for obtaining text according to speech; According to a preset image database and a text-to-sign language image model, each of the standard texts is converted to obtain a sign language image corresponding to each of the standard speech segments, wherein the text-to-sign language image model is a pre-trained neural network model that obtains images from texts; A generating module is used to generate a sign language animation according to the sign language image corresponding to each of the standard speech segments.

10. An electronic device, characterized in that: include: Processor, memory, communication interface; The memory is used to store executable instructions of the processor; Wherein, the processor is configured to execute the speech-to-sign language animation method according to any one of claims 1 to 8 by executing the executable instructions.

11. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for converting speech to sign language animation according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that It comprises a computer program, which is used to implement the speech-to-sign language animation method according to any one of claims 1 to 8 when executed by a processor.

Images