Training and speech generation methods and apparatuses for speech generation model, electronic device, computer-readable storage medium, and computer program product

Abstract

The present application provides training and speech generation methods and apparatuses for a speech generation model, an electronic device, a computer-readable storage medium, and a computer program product. The method comprises: obtaining a first speech generation model; obtaining sample data of a plurality of modalities; on the basis of a prompt image sequence and speech text, respectively calling a plurality of encoders to perform encoding, so as to obtain a multi-modal encoding vector sequence; on the basis of the multi-modal encoding vector sequence, calling a decoder to perform decoding, so as to obtain decoded text; determining a probability distribution for the decoded text and the sample data of the plurality of modalities, and determining a target loss on the basis of the probability distribution; and on the basis of the target loss, updating parameters of the decoder and at least one of the encoders, wherein the updated decoder and the plurality of updated encoders are configured to form a second speech generation model, and the second speech generation model is used to generate target speech text corresponding to a prompt image sequence of a target object.

Description

Method for training speech generation model, speech generation method, device, electronic equipment, computer readable storage medium and computer program product

[0001] Cross-reference to related applications

[0002] The present application is based on and claims priority to Chinese Patent Application No. 2024106268722, filed on May 15, 2024, the entire contents of which are hereby incorporated by reference into the present application. TECHNICAL FIELD

[0003] The present application relates to the technical field of artificial intelligence, and in particular to a method for training a speech generation model, a speech generation method, a device, an electronic equipment, a computer readable storage medium and a computer program product. BACKGROUND

[0004] In the process of interpersonal communication or in the scene where human-computer interaction is needed, when the target object cannot directly express its intention in the form of voice (for example, due to the limitation of the current scene, or the user has physiological barriers to make sound), the related technology supports the user to express his intention through body movements and lip shapes, or inputting text in the terminal device. For example, in the related technology, the hand movements and lip movements of the target object can be encoded to generate speech text, but the speech text generated by the related technology is not accurate and convenient enough, resulting in poor user experience. SUMMARY

[0005] Therefore, the embodiments of the present application provide a method for training a speech generation model, a speech generation method, a device, an electronic equipment, a computer readable storage medium and a computer program product, which can improve the efficiency and accuracy of generating speech text from prompt image.

[0006] The technical solutions of the embodiments of the present application are implemented as follows:

[0007] The embodiments of the present application provide a method for training a speech generation model, which is executed by an electronic equipment, and the method comprises:

[0008] Obtaining a first speech generation model, wherein the first speech generation model comprises a decoder and a plurality of encoders corresponding to a plurality of modalities respectively;

[0009] Obtaining sample data of a plurality of modalities, wherein the sample data of the plurality of modalities comprises a sequence of prompt images of a sample object and speech text corresponding to the sequence of prompt images;

[0010] Calling the plurality of encoders to encode based on the sequence of prompt images and the speech text respectively, to obtain a sequence of multi-modal encoding vectors;

[0011] calling the decoder based on the multimodal encoding vector sequence to obtain a decoded text;

[0012] determining a probability distribution of the decoded text and the sample data of the plurality of modalities, and determining a target loss based on the probability distribution;

[0013] updating parameters of the decoder and at least one of the plurality of encoders based on the target loss to obtain an updated decoder and the plurality of encoders, wherein the updated decoder and the plurality of encoders are used to constitute a second speech generation model, and the second speech generation model is used to generate a target speech text corresponding to the prompt image sequence of the target object.

[0014] Embodiments of the present application provide a speech generation method of a speech generation model, the speech generation model being the second speech generation model described above, and the method is executed by an electronic device, and the method comprises:

[0015] obtaining a prompt image sequence of a target object;

[0016] generating a target speech text corresponding to the prompt image sequence of the target object based on the second speech generation model and the prompt image sequence.

[0017] Embodiments of the present application provide a training device of a speech generation model, and the device comprises:

[0018] an obtaining module configured to obtain a first speech generation model, wherein the first speech generation model comprises a decoder and a plurality of encoders corresponding to a plurality of modalities respectively; and obtain sample data of the plurality of modalities, wherein the sample data of the plurality of modalities comprises a prompt image sequence of a sample object and a speech text corresponding to the prompt image sequence;

[0019] an encoding module configured to call the plurality of encoders based on the prompt image sequence and the speech text to obtain a multimodal encoding vector sequence;

[0020] a decoding module configured to call the decoder based on the multimodal encoding vector sequence to obtain a decoded text;

[0021] a determining module configured to determine a probability distribution of the decoded text and the sample data of the plurality of modalities, and determine a target loss based on the probability distribution;

[0022] The generating module is configured to update parameters of the decoder and at least one of the encoders based on the target loss to obtain an updated decoder and the plurality of encoders, wherein the updated decoder and the plurality of encoders are used to constitute a second speech generation model, and the second speech generation model is used to generate a target speech text corresponding to the prompt image sequence of the target object.

[0023] The embodiment of the present application provides a speech generation device of a speech generation model, and the speech generation model is the second speech generation model described above, and the device comprises:

[0024] The obtaining module is configured to obtain a prompt image sequence of a target object.

[0025] The generating module is configured to call the second speech generation model to generate a target speech text corresponding to the prompt image sequence of the target object based on the prompt image sequence.

[0026] The embodiment of the present application provides an electronic device, and the electronic device comprises:

[0027] The memory is used for storing computer executable instructions.

[0028] The processor is used for executing the computer executable instructions stored in the memory, and a training method of a speech generation model or a speech generation method of a speech generation model provided by the embodiment of the present application is implemented.

[0029] The embodiment of the present application provides a computer readable storage medium, which stores computer programs or computer executable instructions, and is used for implementing the training method of the speech generation model or the speech generation method of the speech generation model provided by the embodiment of the present application when the processor executes.

[0030] The embodiment of the present application provides a computer program product, which comprises computer programs or computer executable instructions, and the computer programs or computer executable instructions are executed by the processor to implement the training method of the speech generation model or the speech generation method of the speech generation model provided by the embodiment of the present application.

[0031] The embodiment of the present application has the following beneficial effects:

[0032] By encoding the sample data of multiple modalities including the image sequence of the prompt (corresponding to the visual modality) and the speech text (corresponding to the text modality), the obtained multi-modal encoding vector sequence (that is, the encoding vector sequence of multiple modalities) is calculated with the probability distribution of the decoded text to obtain the target loss, so that the target loss can reflect the difference between the representation spaces of multiple modalities in the first speech generation model. Based on the target loss, the parameters of the first speech generation model are updated through back propagation, so that the first speech generation model can gradually learn the internal relationship between the information of the visual modality and the text modality, improve the performance and generalization ability of the model, and realize the unification of the representation spaces of the visual modality and the text modality in the second speech generation model obtained after training, which means that the second speech generation model can map data of different modalities to a common representation space, so that data of different modalities have similar semantic expressions in the space, laying a foundation for subsequent accurate generation of speech text, and ensuring that the target speech text output by the second speech generation model can accurately match the intention of the prompt image of the target object in expression, thereby ensuring the accuracy of generating speech text from the prompt image. At the same time, compared with the way of generating speech text by manually inputting text in the related art, it is more convenient, more efficient, and better user experience. BRIEF DESCRIPTION OF DRAWINGS

[0033] FIG. 1 is an architecture schematic diagram of a training system 100 of a speech generation model provided by an embodiment of the present application;

[0034] FIG. 2A is a structural schematic diagram of an electronic device 500-1 provided by an embodiment of the present application;

[0035] FIG. 2B is a structural schematic diagram of an electronic device 500-2 provided by an embodiment of the present application;

[0036] FIG. 3A is a flow schematic diagram of a training method of a speech generation model provided by an embodiment of the present application;

[0037] FIG. 3B is a first flow schematic diagram of obtaining a multi-modal encoding vector sequence provided by an embodiment of the present application;

[0038] FIG. 3C is a flow schematic diagram of obtaining a visual encoding vector sequence provided by an embodiment of the present application;

[0039] FIG. 3D is a second flow schematic diagram of obtaining a multi-modal encoding vector sequence provided by an embodiment of the present application;

[0040] FIG. 3E is a third flow schematic diagram of obtaining a multi-modal encoding vector sequence provided by an embodiment of the present application;

[0041] FIG. 3F is a fourth flow schematic diagram of obtaining a multi-modal encoding vector sequence provided by an embodiment of the present application;

[0042] FIG. 3G is a flowchart of a process for aligning lengths of sequences of encoding vectors, according to some embodiments of the present disclosure;

[0043] FIG. 3H is a first flowchart of a process for determining a probability distribution, according to some embodiments of the present disclosure;

[0044] FIG. 31 is a first flowchart of a process for determining a target loss, according to some embodiments of the present disclosure;

[0045] FIG. 3J is a second flowchart of a process for determining a probability distribution, according to some embodiments of the present disclosure;

[0046] FIG. 3K is a second flowchart of a process for determining a target loss, according to some embodiments of the present disclosure;

[0047] FIG. 3L is a third flowchart of a process for determining a probability distribution, according to some embodiments of the present disclosure;

[0048] FIG. 3M is a third flowchart of a process for determining a target loss, according to some embodiments of the present disclosure;

[0049] FIG. 3N is a first flowchart of a process for updating parameters, according to some embodiments of the present disclosure;

[0050] FIG. 30 is a second flowchart of a process for updating parameters, according to some embodiments of the present disclosure;

[0051] FIG. 3P is a third flowchart of a process for updating parameters, according to some embodiments of the present disclosure;

[0052] FIG. 3Q is a flowchart of a process for generating speech using a speech generation model, according to some embodiments of the present disclosure;

[0053] FIG. 4A is a first architecture of a speech generation model, according to some embodiments of the present disclosure;

[0054] FIG. 4B is a second architecture of a speech generation model, according to some embodiments of the present disclosure;

[0055] FIG. 4C is a third architecture of a speech generation model, according to some embodiments of the present disclosure;

[0056] FIG. 5A is a structure of a visual encoder model, according to some embodiments of the present disclosure;

[0057] FIG. 5B is a structure of a machine learning model, according to some embodiments of the present disclosure;

[0058] FIG. 6 is a diagram of aligning sequences of encoding vectors, according to some embodiments of the present disclosure;

[0059] FIG. 7 is a diagram of a prompt speech encoding information, according to some embodiments of the present disclosure;

[0060] FIG. 8 is a diagram of training data, according to some embodiments of the present disclosure;

[0061] FIG. 9 is a diagram of a visual coding model architecture according to an embodiment of the present application.

[0062] It should be noted that the above-mentioned "first", "second" are only used to distinguish different schemes, and do not represent the degree of distinction of the schemes or the priority in the implementation process. DETAILED DESCRIPTION

[0063] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0064] In the following description, "some embodiments" are related to a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0065] In the following description, the terms "first / second / third" are only used to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that "first / second / third" can interchange the specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0066] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the function of the module or unit.

[0067] Unless otherwise specified, at least one described below refers to one or more cases, and "a plurality of" can refer to two or more cases.

[0068] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0069] The related data collection and processing in the embodiments of the present application should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0070] Before the embodiments of the present application are further described in detail, the terms and phrases involved in the embodiments of the present application are explained, which are applicable to the following explanations.

[0071] 1) Speech generation model: including a decoder and multiple encoders corresponding to multiple modalities respectively, trained by sample data of multiple modalities of a sample object, for generating target speech text corresponding to visual expression of a target object.

[0072] 2) Visual modality: through intuitive ways such as hand actions, lip actions and body actions, the other party can perceive and understand the visual expression of the target object's facial expressions, eye contact, body language and other signals, so as to obtain the emotional state and intention of the target object. For example, for a prompt image sequence of the target object, each prompt image in the prompt image sequence is a static image showing the hand or lip posture when the sample object makes a body action or a lip action. In each prompt image, the posture and relative position of the hand and the lip of the target object when making a key action are marked.

[0073] 3) Text modality: a text form of expression for expressing the emotional state and intention of the target object, such as "the weather is nice today".

[0074] 4) Speech modality: a speech signal form of expression for expressing the emotional state and intention of the target object.

[0075] 5) Multi-modal encoding vector sequence: calling the encoders corresponding to multiple modalities in the language generation model, respectively encoding the sample data of multiple modalities, and taking the obtained encoding vector sequence of multiple modalities as the multi-modal encoding vector sequence.

[0076] 6) Representation space: refers to a vector space composed of feature vectors or feature representations, used to describe and represent different attributes, features or information of data. In different modalities, the representation space embodies the features of different types of data and their representation in the vector space. For example, the representation space of the visual modality is a vector space composed of visual feature vectors, and the representation space of the text modality is a vector space composed of text feature vectors.

[0077] 7) Cued-speech (CS): an encoding system used by people who cannot speak or have impaired speech to express spoken language.

[0078] 8) Sign Language (SL): a form of language that uses hand gestures, hand movements, and facial expressions to communicate and convey messages. Words, phrases, and sentences are represented using hand gestures and hand movements, with different shapes, positions, and movements of fingers, palms, wrists, and other parts of the hand, in combination with facial expressions and body movements.

[0079] 9) Impaired-speech (IS): refers to speech signals with reduced quality or difficulty in understanding due to various reasons, such as when noise is mixed with speech signals, resulting in reduced speech quality and difficulty in clearly hearing and understanding the speech content; speech signals are distorted or damaged during transmission, recording, or processing, resulting in spectral distortion, time-domain distortion, distortion impact, and other distortions, making the speech sound unnatural or distorted; people with speech disorders or pronunciation difficulties due to physiological or neurological reasons, making it difficult or impossible to understand the speech due to unclear pronunciation, abnormal speech rate, unstable tone, and other reasons.

[0080] 10) Knowledge Distillation (KD): a model training technique that transfers knowledge from a teacher model to a student model to assist the student model in learning and improving its performance. In knowledge distillation, the teacher model is usually a complex and accurate model with good performance. The student model is a lightweight model that cannot achieve the complexity of the teacher model due to limitations in computing resources or model size. For example, in the present application, the output of the visual encoder is aligned to the feature space of the output of the text encoder through knowledge distillation, i.e., the model of the text encoder is used as the teacher model, and the model of the visual encoder is used as the student model, with the text encoder guiding the visual encoder.

[0081] 11) Prompt image sequence: a sequence of prompt images composed of multiple prompt images, each of which identifies the pose and position of the target object's limbs, lips, and hands when performing limb movements, lip movements, or hand movements.

[0082] 12) Key point sequence: in each prompt image of the prompt image sequence, points that identify the pose and position of the limbs, lips, or hands are identified as key points, and the key points in each prompt image of the prompt image sequence are combined in the order of the occurrence of the movements to obtain a key point sequence.

[0083] In the related art, a speech text is generated by encoding hand actions and lip actions of a target object, but the way of generating a speech text by encoding is only applicable to users who understand the encoding method, has a small scope of application, and the expression of the generated speech text is less accurate and natural than normal communication. The way of completing the interaction process by manually inputting text by the user is not convenient, has low efficiency, and has poor user experience.

[0084] To solve the above problems, the embodiments of the present application provide a speech generation model training method and device, electronic equipment, computer readable storage medium and computer program product, which can improve the efficiency and accuracy of generating a speech text from a prompt image.

[0085] The following describes an exemplary application of the electronic equipment provided by the embodiments of the present application. The electronic equipment provided by the embodiments of the present application can be implemented as a notebook computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart television, a vehicle-mounted terminal, and the like. The following describes an exemplary application when the electronic equipment is implemented as a terminal or a server.

[0086] Referring to FIG. 1, FIG. 1 is an architecture schematic diagram of a speech generation model training system 100 provided by the embodiments of the present application. To realize a speech generation model training application, a terminal 400 (exemplarily showing a graphical interface 410) is connected to a server 200 through a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0087] The method provided by the embodiments of the present application can be applied to the following several different scenarios, which can improve the interaction experience of users, enhance the flexibility and convenience of human-computer interaction, and help to realize more intelligent and personalized human-computer interaction.

[0088] 1. Assist communication: For people with hearing or language barriers, their gestures and lip shapes can be recognized to help them communicate with others. For example, the terminal 400 obtains a prompt image sequence of a person with hearing or language barriers, and sends it to the server 200. The server 200 translates the sign language of the person with hearing or language barriers into text or speech by recognizing the position of the fingers, the shape of the gestures, and the motion trajectory in the prompt image sequence, and sends it to the terminal 400 for display to the target person. In this way, people with limited hearing or language can communicate with others in real time through gestures.

[0089] 2. Gesture control: Body movement and gesture recognition can be applied in smart devices or wearable devices, allowing users to control device functions through gestures, such as adjusting volume, switching songs, etc. For example, terminal 400 detects user body movements and sends them to server 200, which determines the increase (or decrease) in volume amplitude by recognizing the user's arm sliding up (or down) or raising (lowering) the palm height in the body movement, and sends it to terminal 400, which increases (or decreases) the volume through terminal 400.

[0090] 3. Expression recognition: By recognizing facial expressions and mouth shapes, the user's emotional state and intentions can be determined, providing better user experience and personalized services, such as automated sentiment analysis, emotional interaction in games, etc. For example, in a game, terminal 400 can obtain player facial expression change data and send it to server 200, which decodes the prompt image sequence by editing the player's facial expression change data to obtain the player's target intention data and send it to terminal 400, allowing terminal 400 to adjust game difficulty, provide appropriate props, or change the plot according to the target intention data, thus providing a more challenging and personalized gaming experience.

[0091] 4. Virtual reality and augmented reality: By recognizing body movements and mouth shapes, users' movements and expressions can be transmitted in real time to virtual reality or augmented reality environments, providing a more immersive interactive experience, such as virtual character body movement imitation, etc. For example, terminal 400 obtains user prompt image sequences and sends them to terminal 200, which determines the user's target intention by recognizing the user's body movements and mouth shapes and sends it to terminal 400, which maps the user's movements and expressions in real time to virtual characters or virtual scenes according to the user's target intention, allowing actors or artists to control virtual character movements and interactions through their own movements and expressions in performances or shows, making augmented reality experiences more vivid.

[0092] 5. Intelligent driving assistance: By recognizing drivers' body movements and mouth shapes, state abnormalities can be detected in time, improving driving safety. For example, terminal 400 (such as a car-mounted camera) obtains driver prompt image sequences (such as yawning, frequent blinking, hands off the steering wheel, etc.) during driving and sends them to server 200. Server 200 determines whether the driver is tired or distracted by recognizing the prompt image sequences. For example, recognizing frequent blinking as a sign of fatigue, server 200 generates an alert message and sends it to terminal 400. Terminal 400 reminds the driver to rest or takes appropriate safety measures, such as playing an alarm sound, prompting a rest message, etc.

[0093] 6、Medical rehabilitation and adjuvant therapy: According to the patient's limb movement and mouth shape, the patient's rehabilitation progress is detected, and the doctor is assisted in diagnosis and treatment. For example, the terminal 400 obtains the image sequence of the patient's prompt words (such as the amplitude and frequency of limb movement) in the rehabilitation training and sends it to the server 200. The server 200 analyzes the patient's rehabilitation by recognizing the image sequence of the prompt words. For example, it is identified that the amplitude of the patient's limb movement gradually increases, indicating good rehabilitation effect, and an evaluation report is generated and sent to the terminal 400. The terminal 400 displays the evaluation report to the doctor and the patient, and the doctor can adjust the rehabilitation plan according to the report, and the patient can know his own rehabilitation progress.

[0094] 7、Smart home control: In a smart home environment, users can control various devices in the home such as lights, curtains, appliances, etc. through limb movements and mouth shapes, improving the convenience of life. For example, the terminal 400 (such as a smart camera) obtains the image sequence of the user's prompt words (such as waving arms, opening mouths, etc.), and sends it to the server 200. The server 200 determines the user's operation intention by recognizing the user's limb movements and mouth shapes. For example, it is identified that the user's arm waving action means opening the curtains, and the corresponding control instruction is generated and sent to the terminal 400. The terminal 400 operates the smart home device according to the control instruction, such as opening the curtains, adjusting the light brightness, etc.

[0095] The terminal 400 is configured to send the image sequence of the prompt words of the sample object and the sample data of multiple modalities containing the speech text corresponding to the image sequence of the prompt words, and the image sequence of the prompt words of the target object to the server 200 via the network 300. The server 200 calls multiple encoders corresponding to multiple modalities in the first speech generation model to encode the received sample data of multiple modalities of the sample object, to obtain a multi-modal encoding vector sequence, and calls a decoder in the first speech generation model to decode the multi-modal encoding vector sequence, to obtain a decoding text. Then, the probability distribution of the decoding text and the sample data of multiple modalities is determined, and the target loss is determined based on the probability distribution. The parameters of the decoder and at least one encoder are updated based on the target loss to obtain a second speech generation model. Finally, the server 200 generates a target speech text corresponding to the image sequence of the prompt words of the target object by calling the second speech generation model, and sends the target speech text to the terminal 400 via the network 300. The target speech text is displayed through the graphical interface 410, and at the same time, the target speech text is generated through the speech signal generator of the second speech generation model. The corresponding speech information is played back at the terminal.

[0096] In some embodiments, the terminal 400 calls a plurality of encoders corresponding to a plurality of modalities in the first speech generation model to encode the sample data of a plurality of modalities of the received sample object, to obtain a multi-modal encoding vector sequence, and calls a decoder in the first speech generation model to decode the multi-modal encoding vector sequence, to obtain a decoded text; then determines the probability distribution of the decoded text and the sample data of the plurality of modalities, and determines a target loss based on the probability distribution; updates the parameters of the decoder and at least one encoder based on the target loss, to obtain a second speech generation model; finally, generates a target speech text corresponding to the prompt image sequence of the target object by calling the second speech generation model, and displays the target speech text through the graphical interface 410, at the same time, the target speech text passes through the speech signal generator of the second speech generation model to generate corresponding speech information, and is broadcast in the terminal.

[0097] In some embodiments, the server 200 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited in the embodiments of the present application.

[0098] The embodiments of the present application can be implemented through artificial intelligence technology. Artificial intelligence (AI) is a theory, method, technology and application system for using digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology of computer science, which aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0099] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software level technology. Artificial intelligence basic technology generally includes, such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-training model technology, operation / interaction system, mechatronics, etc. Among them, the pre-training model is also called large model, basic model, which can be widely applied to downstream tasks in various directions of artificial intelligence after fine-tuning. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning, etc.

[0100] Natural language processing (NLP) is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers using natural language. Natural language processing involves natural language, i.e. the language used in daily life, and is closely related to linguistic research; at the same time, it involves important technologies for model training in the fields of computer science and mathematics, artificial intelligence. Pre-training model is developed from the large language model (LLM) in the field of NLP. After fine-tuning, the large language model can be widely applied to downstream tasks. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question and answer, knowledge graph, etc.

[0101] The electronic device implementing the training method of the speech generation model provided in the embodiments of the present application can be the terminal 400 or the server 200 in FIG. 1. Referring to FIG. 2A, FIG. 2A is a structural schematic diagram of an electronic device 500-1 provided by the embodiments of the present application. The electronic device 500-1 shown in FIG. 2A includes at least one processor 510-1, at least one network interface 520-1, a user interface 530-1 and a memory 550-1. Each component in the electronic device 500-1 is coupled together through a bus system 540-1. It can be understood that the bus system 540-1 is used to realize the connection communication between the components. In addition to the data bus, the bus system 540-1 also includes power bus, control bus and status signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as bus system 540-1 in FIG. 2A.

[0102] The processor 510-1 can be an integrated circuit chip with signal processing capability, such as a general purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general purpose processor can be a microprocessor or any conventional processor.

[0103] The user interface 530-1 includes one or more output devices 531-1 that enable presentation of media content, including at least one of one or more speakers and one or more visual display screens. The user interface 530-1 also includes one or more input devices 532-1 that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0104] The memory 550-1 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, and the like. The memory 550-1 optionally includes one or more storage devices remotely located from the processor 510-1 in a physical location.

[0105] The memory 550-1 includes volatile memory or nonvolatile memory, and can also include both volatile and nonvolatile memory. Nonvolatile memory can be read only memory (ROM), volatile memory can be random access memory (RAM). The memory 550-1 described in embodiments of the present application is intended to include any suitable type of memory.

[0106] In some embodiments, the memory 550-1 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or a subset or superset thereof, which are exemplarily illustrated below.

[0107] The operating system 551-1 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks;

[0108] The network communication module 552-1 is used to communicate with other electronic devices via one or more (wired or wireless) network interfaces 520-1, examples of which include Bluetooth, wireless compatibility certification (WiFi), and universal serial bus (USB), and the like;

[0109] The presentation module 553-1 is used to enable presentation of information via one or more output devices 531-1 associated with the user interface 530-1 (e.g., display screens, speakers, and the like) (e.g., user interfaces for operating peripheral devices and displaying content and information);

[0110] The input processing module 554-1 is used to detect and interpret one or more user inputs or interactions from one or more input devices 532-1.

[0111] In some embodiments, the apparatus provided by the embodiments of the present application can be implemented in a software manner. FIG. 2A shows a training apparatus 555-1 of a language understanding model stored in a memory 550-1, which can be software in the form of programs and plug-ins, etc., including the following software modules: an obtaining module 5551-1, an encoding module 5552-1, a decoding module 5553-1, a determining module 5554-1 and a generating module 5555-1. These modules are logical, and thus can be combined or further split according to the implemented functions. The functions of the various modules will be described below.

[0112] The electronic device implementing the speech generation method of the speech generation model provided by the embodiments of the present application can be the terminal 400 or the server 200 in FIG. 1. Referring to FIG. 2B, FIG. 2B is a structural schematic diagram of an electronic device 500-2 provided by the embodiments of the present application. The electronic device 500-2 shown in FIG. 2B includes at least one processor 510-2, at least one network interface 520-2, a user interface 530-2 and a memory 550-2. The various components in the electronic device 500-2 are coupled together by a bus system 540-2. It can be understood that the bus system 540-2 is used to realize the connection communication between the components. In addition to the data bus, the bus system 540-2 also includes a power bus, a control bus and a status signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 540-2 in FIG. 2B.

[0113] The processor 510-2 can be an integrated circuit chip with signal processing capability, such as a general purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc., wherein the general purpose processor can be a microprocessor or any conventional processor.

[0114] The user interface 530-2 includes one or more output devices 531-2 that enable presentation of media content, including at least one of one or more speakers and one or more visual display screens. The user interface 530-2 also includes one or more input devices 532-2, including user interface components that facilitate input by a user, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons and controls.

[0115] The memory 550-2 can be removable, non-removable or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 550-2 optionally includes one or more storage devices physically located in proximity to the processor 510-2.

[0116] The memory 550-2 includes volatile memory or non-volatile memory, and can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), and the volatile memory can be random access memory (RAM). The memory 550-2 described in the embodiments of the present application is intended to include any suitable type of memory.

[0117] In some embodiments, the memory 550-2 can store data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily illustrated below.

[0118] The operating system 551-2 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks;

[0119] The network communication module 552-2 is configured to communicate with other electronic devices via one or more (wired or wireless) network interfaces 520-2, examples of which include Bluetooth, wireless compatibility authentication (WiFi), and universal serial bus (USB), and the like;

[0120] The presentation module 553-2 is configured to enable presentation of information via one or more output devices 531-2 (e.g., a display screen, a speaker, and the like) associated with the user interface 530-2 (e.g., a user interface for operating a peripheral device and displaying content and information);

[0121] The input processing module 554-2 is configured to detect and interpret one or more user inputs or interactions from one or more input devices 532-2.

[0122] In some embodiments, the apparatus provided by the embodiments of the present application can be implemented in software, and FIG. 2B shows a training apparatus 555-2 of a language understanding model stored in the memory 520-2, which can be software in the form of programs and plug-ins, including the following software modules: an acquisition module 5551-2 and a generation module 5552-2, which are logical, and thus can be combined or further split according to the implemented functions. The functions of each module will be described below.

[0123] In some embodiments, a terminal or a server can implement the training method of the language understanding model provided in the embodiments of the present application by running various computer-executable instructions or computer programs. For example, the computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. The computer program can be a native program in the operating system or a software module; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run. In summary, the above-mentioned computer-executable instructions can be any form of instructions, and the above-mentioned computer program can be any form of application program, module, or plug-in.

[0124] Next, the training method of the speech generation model provided in the embodiments of the present application is described. As described above, the electronic device implementing the training method of the speech generation model of the embodiments of the present application can be a terminal or a server, or a combination of the two. Therefore, the execution subject of each step will not be repeated in the following description.

[0125] Referring to FIG. 3A, which is a flowchart of the training method of the speech generation model provided in the embodiments of the present application, the steps shown in FIG. 3A will be described.

[0126] In step 101, a first speech generation model is obtained, wherein the first speech generation model includes a decoder and a plurality of encoders corresponding to a plurality of modalities respectively.

[0127] In some embodiments, the first speech generation model can be pre-trained locally to form the first speech generation model, or the first speech generation model formed by pre-training by other devices can be obtained from other devices, including a decoder and a plurality of encoders corresponding to a plurality of modalities respectively.

[0128] As an example, referring to FIG. 4A, which is a first architecture diagram of the speech generation model provided in the embodiments of the present application, FIG. 4A shows a text decoder, a visual encoder corresponding to a visual modality, and a text encoder corresponding to a text modality.

[0129] In step 102, sample data of a plurality of modalities is obtained, wherein the sample data of the plurality of modalities includes a sequence of prompt image of a sample object and speech text corresponding to the sequence of prompt image.

[0130] In some embodiments, the sample data of multiple modalities of the sample object includes sample data of a visual modality, such as a sequence of prompt image of the sample object, and sample data of a text modality, such as speech text corresponding to the sequence of prompt image. Each prompt image in the sequence of prompt image is a static image of a hand or lip posture when the sample object makes a limb action or a lip action, which can capture the visual features of the sample object in a specific scene and provide rich visual information for the model to better understand the behavior and state of the sample object.

[0131] Here, the sample data of the visual modality refers to a data form that describes the sample object through visual information. Through intuitive ways such as hand actions, lip actions, and limb actions, the other party can perceive and understand the visual expressions of the target object's facial expressions, eye contact, body language, and other signals to obtain the emotional state and intention of the target object. The sample data of the text modality refers to data that describes the sample object in the form of text. These text data can be an explanation, description, or other information related to the image content, providing a semantic supplement for the model. By combining the data of the visual modality and the text modality, the model can better understand the characteristics and behavior of the sample object.

[0132] As an example, the sample data of the visual modality of the sample object can be collected by using a camera, an image database, or a network source; the speech text related to the sample data of the visual modality of the sample object can be recorded by a recording device, or extracted from an existing speech database or corpus that matches the sample data of the visual modality of the sample object.

[0133] In step 103, multiple encoders are called based on the sequence of prompt images and the speech text for encoding, respectively, to obtain a sequence of multimodal encoding vectors.

[0134] In some embodiments, referring to FIG. 3B, which is a first flowchart of obtaining a sequence of multimodal encoding vectors provided by the embodiments of the present application, when the multiple modalities include a visual modality and a text modality, step 103 of FIG. 3A can be implemented by steps 1031A to 1033A of FIG. 3B, which are described in detail below.

[0135] In step 1031A, a visual encoder is called to encode the sequence of prompt images to obtain a sequence of visual encoding vectors of the visual modality, wherein the visual encoder is an encoder corresponding to the visual modality in the multiple encoders.

[0136] In some embodiments, referring to FIG. 3C, which is a flowchart of obtaining a visual coding vector sequence according to an embodiment of the present application, step 1031A of FIG. 3B can be implemented by steps 10311A to 10314A of FIG. 3C, which are described in detail as follows.

[0137] In step 10311A, the plurality of prompt image in the prompt image sequence is respectively identified to obtain the position of the key point in each prompt image.

[0138] In some embodiments, the plurality of prompt image in the prompt image sequence can be respectively subjected to human pose estimation (HPE) by a deep learning model to identify the key point position of the human body as the position of the key point in each prompt image. Here, human pose estimation is a computer vision task aimed at detecting and locating key points (such as head, shoulder, elbow, knee, etc.) of the human body from images or videos. The position information of these key points can be used to estimate the posture of the whole human body.

[0139] As an example, the deep learning model used to identify the plurality of prompt image in the prompt image sequence can be OpenPose, DeepPose, PoseNet, or DensePose.

[0140] In step 10312A, a key point sequence is generated based on the position of the key point in each prompt image.

[0141] As an example, the position of the key point in each prompt image can be the position of the lips, fingers, etc. Based on the position of the key point in all prompt images in the prompt image sequence corresponding to the video, a key point sequence can be generated. Referring to FIG. 9, the points marked in each prompt image in the prompt image sequence shown in FIG. 9 as the position of the hand and the lip are the key points. Based on the position of the key point in each prompt image, the key point sequence shown in FIG. 9 is obtained.

[0142] In step 10313A, an embedding vector of the key point sequence is obtained.

[0143] In some embodiments, each key point in the key point sequence can be subjected to pose embedding processing by a machine learning model to obtain an embedding vector of each key point in the key point sequence. The embedding vectors are spliced to obtain an embedding vector corresponding to the key point sequence.

[0144] As an example, the pose embedding of the keypoint sequence can be performed by a recurrent neural network (RNN) or a convolutional neural network (CNN) to obtain an embedding vector of the keypoint sequence.

[0145] In step 10314A, a visual encoder is invoked to encode the embedding vector to obtain a visual encoding vector of the visual modality.

[0146] In some embodiments, a pre-trained machine learning model can be used as the visual encoder to encode the embedding vector to obtain the visual encoding vector of the visual modality.

[0147] As an example, the model of the visual encoder can be a convolutional neural network model, as shown in FIG. 5A, which is a structural schematic diagram of a visual encoder model according to an embodiment of the present application. The convolutional neural network shown in FIG. 5A includes an input layer, a convolutional layer, a pooling layer, and an output layer (a fully connected layer and a softmax layer). After the embedding vector of the keypoint sequence of the visual modality is input into the convolutional neural network, the embedding vector is sequentially processed by multiple convolutional layers to obtain a feature map corresponding to the embedding vector, and then the feature map is sequentially processed by multiple max-pooling layers to obtain a feature vector of the visual modality, and finally the output layer outputs a visual encoding vector sequence of the visual modality.

[0148] The embodiments of the present application convert the human body action in the video into the keypoint sequence of the visual modality, and extract the key information in the prompt image in the form of the keypoint sequence. These keypoints can capture the information of the body action or the lip action in the image, so as to retain the key information in the image in the form of the keypoint sequence, and clearly describe the change of the human body posture and action. Compared with directly processing the original image or video data, the keypoint sequence greatly reduces the complexity of the data. In the above manner, the most important information in the image is extracted, the redundant data is reduced, the computing efficiency is improved, the subsequent analysis and recognition of the prompt image feature of the visual modality are facilitated, and the characteristic information in the prompt image can be effectively utilized.

[0149] Continuing to refer to FIG. 3B, the subsequent step 1031A is described.

[0150] In step 1032A, a text encoder is invoked to encode the multiple phonemes in the speech text respectively to obtain a sequence of text encoding vectors of the text modality, where the sequence of text encoding vectors includes text encoding vectors corresponding to the multiple phonemes respectively, and the text encoder is an encoder corresponding to the text modality in the multiple encoders.

[0151] In some embodiments, the multiple phonemes (e.g., words, phrases, or sentences) are split by language segmentation on the speech text. The text encoder can be a machine learning model, which converts the multiple phonemes in the speech text into text encoding vectors by encoding the multiple phonemes in the speech text respectively, and combines the text encoding vectors corresponding to the phonemes in the speech text to obtain the sequence of text encoding vectors of the text modality.

[0152] As an example, the machine learning model for encoding the speech text can be a recurrent neural network, a convolutional neural network, a Transformer, a Bidirectional Encoder Representations from Transformer (BERT), or the like.

[0153] Referring to FIG. 5B, which is a structural diagram of a machine learning model according to an embodiment of the present application, FIG. 5B shows an input embedding layer, a positional embedding layer, a self-attention mechanism, a feed-forward neural network, and an output layer. First, the phonemes of the speech text are converted into text embedding vectors of a fixed dimension by the input embedding layer, usually using word embedding technology to map the phonemes to vector representations in a continuous space; second, the positional information of each phoneme is embedded into a feature vector by the positional embedding layer to obtain a positional embedding vector; then, the importance of each phoneme in the context is modeled by the self-attention mechanism, and semantic relationships are learned, and the positional embedding vector of each phoneme is nonlinearly transformed and mapped by the feed-forward neural network layer to extract higher-level semantic and contextual information; finally, the entire sequence of text encoding vectors is generated based on the positional embedding vector of each phoneme.

[0154] In step 1033A, the sequence of visual encoding vectors and the sequence of text encoding vectors are taken as a first multi-modal encoding vector sequence, where the first multi-modal encoding vector sequence is used for decoding by the decoder.

[0155] In some embodiments, when the plurality of modalities includes a visual modality and a text modality, the multimodal encoding vector sequence includes a visual encoding vector sequence corresponding to the visual modality and a text encoding vector sequence corresponding to the text modality.

[0156] In the embodiments of the present application, a plurality of morphemes in a speech text are respectively encoded into a text encoding vector sequence of a text modality, so that the encoding vector of each morpheme can capture its semantic information and contextual relationship, thereby better expressing the meaning of the entire text and improving the accuracy of subsequent speech generation.

[0157] In some embodiments, referring to FIG. 3D, which is a second flowchart of obtaining a multimodal encoding vector sequence provided by the embodiments of the present application, when the plurality of modalities further includes a speech modality, the sample data of the plurality of modalities further includes speech signal features of the speech modality; the step 103 of FIG. 3A can be implemented by the steps 1031B to 1032B of FIG. 3D, which are described below in detail.

[0158] In step 1031B, a speech encoder is called to encode the speech signal features to obtain a speech encoding vector sequence of the speech modality, wherein the speech encoder is an encoder corresponding to the speech modality in the plurality of encoders.

[0159] As an example, referring to FIG. 4B, which is a second architecture diagram of a speech generation model provided by the embodiments of the present application, FIG. 4B shows a text decoder, a visual encoder corresponding to a visual modality, a text encoder corresponding to a text modality, and a speech encoder corresponding to a speech modality.

[0160] In some embodiments, the calling of the speech encoder to encode the speech signal features to obtain a speech encoding vector sequence of the speech modality can be implemented in the following manner: first, pre-processing the original speech signal, such as removing noise (eliminating background noise and retaining cleaned speech signal), filtering (removing noise in a specific frequency range or enhancing speech signal in a specific frequency range), standardizing volume, etc.; second, dividing the pre-processed speech signal into short-time frames, and the commonly used frame length is 20-40 milliseconds, usually with 50% or 75% overlap; third, multiplying each frame of speech signal by a window function, such as a Hamming window or a rectangular window, to reduce boundary artifacts between frames; fourth, applying Fast Fourier Transform (FFT) to each frame to convert the time-domain signal into a frequency-domain representation; and finally, extracting speech encoding vectors from the frequency spectrum and concatenating the speech encoding vectors to obtain a speech encoding vector sequence.

[0161] As an example, the model of the speech encoder used to encode the speech signal features can be a Mel-filterbank extractor.

[0162] In step 1032B, the visual encoding vector sequence, the text encoding vector sequence and the speech encoding vector sequence are taken as a second multi-modal encoding vector sequence, wherein the second multi-modal encoding vector sequence is used to replace the first multi-modal encoding vector sequence for decoding by the decoder.

[0163] In some embodiments, when the multiple modalities include a visual modality, a text modality and a speech modality, the multi-modal encoding vector sequence includes a visual encoding vector sequence corresponding to the visual modality, a text encoding vector sequence corresponding to the text modality and a speech encoding vector sequence corresponding to the speech modality.

[0164] By introducing the encoding vector sequence of the speech modality, the speech generation model can more comprehensively capture the association between multi-modal information, and the addition of speech signal features enables the model to better understand the semantic alignment relationship between speech content and visual and text information. The fusion of the speech encoding vector sequence with the encoding vector sequences of other modalities forms a unified multi-modal representation, enabling the speech generation model to more efficiently process multi-modal data, especially in tasks that require simultaneous processing of multiple modalities, which can significantly improve the performance of the model in multi-modal tasks, enhance its semantic understanding ability, improve the generation quality, and improve the robustness and adaptability of the speech generation model.

[0165] In some embodiments, referring to FIG. 3E, which is a third flowchart for obtaining a multi-modal encoding vector sequence according to an embodiment of the present application, when the multiple modalities further include a speech modality, step 103 of FIG. 3A can also be implemented by steps 1031C to 1034C of FIG. 3E, which are described in detail below.

[0166] In step 1031C, based on the prompt image sequence and the speech text, multiple encoders are called for encoding to obtain a visual encoding vector sequence of the visual modality, a text encoding vector sequence of the text modality and a speech encoding vector sequence of the speech modality.

[0167] In some embodiments, as previously described, when the multiple modalities include a visual modality, a text modality and a speech modality, a visual encoder can be called to encode the sample data of the visual modality to obtain a visual encoding vector sequence of the visual modality; a text encoder can be called to encode the sample data of the text modality to obtain a text encoding vector sequence of the text modality; and a language encoder can be called to encode the sample data of the speech modality to obtain a speech encoding vector sequence of the speech modality.

[0168] In some embodiments, the sample data of the plurality of modalities further comprises speech signal features of a speech modality; referring to FIG. 3F, which is a fourth flowchart for obtaining a sequence of multimodal encoding vectors according to an embodiment of the present application, step 1031C of FIG. 3E can be implemented by steps 10311C to 10313C of FIG. 3F, which are described in detail as follows.

[0169] In step 10311C, a visual encoder is invoked to encode the sequence of prompt image to obtain a sequence of visual encoding vectors of the visual modality, wherein the visual encoder is an encoder corresponding to the visual modality among the plurality of encoders.

[0170] In some embodiments, step 10311C described above can be implemented by performing the following processing:

[0171] First, the plurality of prompt images in the sequence of prompt images are respectively recognized to obtain the positions of the key points in each prompt image.

[0172] In some embodiments, the plurality of prompt images in the sequence of prompt images can be respectively subjected to human pose estimation (HPE) by a deep learning model to recognize the positions of the key points of the human body as the positions of the key points in each prompt image. Here, human pose estimation is a computer vision task aimed at detecting and locating key points of the human body (such as head, shoulder, elbow, knee, etc.) from images or videos. The position information of these key points can be used to estimate the posture of the whole human body.

[0173] As an example, the deep learning model used to recognize the plurality of prompt images in the sequence of prompt images can be Open Pose, Deep Pose, PoseNet, or Dense Pose.

[0174] Second, based on the positions of the key points in each prompt image, a sequence of key points is generated.

[0175] As an example, the positions of the key points in each prompt image can be the positions of the lips, fingers, etc., and based on the positions of the key points in all prompt images in the sequence of prompt images corresponding to the video, a sequence of key points can be generated. Referring to FIG. 9, the points marked in each prompt image in the sequence of prompt images shown in FIG. 9 as the positions of the hands and lips are the key points, and based on the positions of the key points in each prompt image, the sequence of key points shown in FIG. 9 is obtained.

[0176] Then, an embedding vector of the sequence of key points is obtained.

[0177] In some embodiments, pose embedding processing can be performed on each key point in the key point sequence by a machine learning model to obtain an embedding vector of each key point in the key point sequence, and the embedding vectors are spliced to obtain an embedding vector corresponding to the key point sequence.

[0178] As an example, the pose embedding of the key point sequence can be performed by a recurrent neural network or a convolutional neural network to obtain the embedding vector of the key point sequence.

[0179] Finally, the embedding vector is encoded by calling a visual encoder to obtain a visual encoding vector of the visual modality.

[0180] In some embodiments, a pre-trained machine learning model can be used as the visual encoder to encode the embedding vector to obtain the visual encoding vector of the visual modality.

[0181] As an example, the model of the visual encoder can be a convolutional neural network model, as shown in FIG. 5A, which is a structural schematic diagram of a visual encoder model according to an embodiment of the present application. The convolutional neural network shown in FIG. 5A includes an input layer, a convolutional layer, a pooling layer, and an output layer. After the embedding vector of the key point sequence of the visual modality is input into the convolutional neural network, the embedding vector is sequentially processed by multiple convolutional layers to obtain a feature map corresponding to the embedding vector, and then the feature map is sequentially processed by multiple max-pooling layers to obtain a feature vector of the visual modality, and finally the output layer outputs a visual encoding vector sequence of the visual modality.

[0182] The embodiments of the present application convert the human body action in the video into the key point sequence of the visual modality, and extract the key information in the prompt image in the form of the key point sequence. These key points can capture the information of the body action or the lip action in the image, thereby retaining the key information in the image in the form of the key point sequence, clearly describing the changes of the body posture and action, and greatly reducing the complexity of the data compared to directly processing the original image or video data. In the above manner, the most important information in the image is extracted, the redundant data is reduced, the computing efficiency is improved, the subsequent analysis and recognition of the prompt image feature of the visual modality are facilitated, and the characteristic information in the prompt image can be effectively utilized.

[0183] In step 10312C, a text encoder is called to encode each morpheme in the speech text to obtain a text encoding vector sequence of the text modality, wherein the text encoding vector sequence includes a text encoding vector corresponding to each morpheme, and the text encoder is an encoder corresponding to the text modality in the plurality of encoders.

[0184] In some embodiments, the step 10312C described above can be implemented by performing the following process: by performing language segmentation on the speech text, a plurality of morphemes (e.g., words, phrases, or sentences) are split. The text encoder can be a machine learning model, which encodes each of the plurality of morphemes in the speech text through the machine learning model, converts the morphemes into a text encoding vector, and combines the text encoding vectors corresponding to the morphemes of the speech text to obtain a text encoding vector sequence of the text modality.

[0185] As an example, the machine learning model used to encode the speech text can be a recurrent neural network, a convolutional neural network, a Transformer, a bidirectional encoder representation based on Transformer, etc.

[0186] Referring to FIG. 5B, which is a structural diagram of a machine learning model according to an embodiment of the present application, FIG. 5B shows an input embedding layer, a position embedding, a self-attention mechanism, a feed-forward neural network, and an output layer. First, the morphemes of the speech text are converted into text embedding vectors of a fixed dimension through the input embedding layer, usually using word embedding technology to map the morphemes to vector representations in a continuous space; second, the position information of each morpheme is embedded into a feature vector through the position embedding layer to obtain a position embedding vector; then, the importance of each morpheme in the context is modeled through the self-attention mechanism, and semantic relationships are learned, and the position embedding vector of each morpheme is nonlinearly transformed and mapped through the feed-forward neural network layer to extract higher-level semantic and contextual information; finally, the entire text encoding vector sequence is generated based on the position embedding vector of each morpheme.

[0187] In step 10313C, a speech encoder is called to encode the speech signal features to obtain a speech encoding vector sequence of the speech modality, wherein the speech encoder is an encoder corresponding to the speech modality among the plurality of encoders.

[0188] In some embodiments, calling the speech encoder to encode the speech signal features to obtain a speech encoding vector sequence of the speech modality can be implemented in the following manner: first, the original speech signal is preprocessed, such as removing noise, filtering, standardizing volume, etc.; second, the preprocessed speech signal is cut into short-time frames, with a commonly used frame length of 20-40 milliseconds, usually with an overlap of 50% or 75%; then, the speech signal of each frame is multiplied by a window function, such as a Hamming window or a rectangular window, to reduce boundary artifacts between frames; a fast Fourier transform is applied to each frame to convert the time-domain signal into a frequency-domain representation; finally, the speech encoding vector is extracted from the frequency spectrum, and the speech encoding vectors are spliced to obtain a speech encoding vector sequence.

[0189] Continuing to refer to FIG. 3E, the step 1031C described above is explained.

[0190] In step 1032C, the visual encoding vector sequence and the speech encoding vector sequence are feature spliced to obtain a spliced encoding vector sequence.

[0191] In some embodiments, the feature vectors in the visual encoding vector sequence and the feature vectors in the speech encoding vector sequence can be feature spliced to obtain a spliced encoding vector, and the spliced encoding vectors corresponding to each position are combined to obtain a spliced encoding vector sequence.

[0192] As an example, if the visual encoding vector in the visual encoding vector sequence output by the visual encoder is the speech encoding vector in the speech encoding vector sequence output by the speech encoder is where f represents the length of the encoding vector sequence, and D represents the dimension of the encoding vector in the encoding vector sequence, then the spliced encoding vector obtained by feature splicing the visual encoding vector sequence and the speech encoding vector sequence is

[0193] In step 1033C, the spliced encoding vector sequence is feature mapped to obtain a fusion encoding vector sequence.

[0194] As an example, in the above example, the dimension of the spliced encoding vector in the spliced encoding vector sequence is 2D, and by feature mapping the spliced encoding vector sequence, the dimension of the spliced encoding vector is converted from 2D to D to obtain a fusion encoding vector The fusion encoding vectors of each position are combined to obtain a fusion encoding vector sequence.

[0195] In step 1034C, the fusion encoding vector sequence and the text encoding vector sequence are taken as a third multi-modal encoding vector sequence, wherein the third multi-modal encoding vector sequence is used to replace the second multi-modal encoding vector sequence for decoding by the decoder.

[0196] As an example, referring to FIG. 4C, which is a third architecture schematic diagram of a speech generation model provided by an embodiment of the present application, FIG. 4C shows a text decoder, a visual encoder corresponding to a visual modality, a text encoder corresponding to a text modality, and a speech encoder corresponding to a speech modality, as shown in FIG. 4C, the visual encoding vector sequence output by the visual encoder and the speech encoding vector sequence output by the speech encoder are feature spliced and mapped to obtain a fusion encoding vector sequence, which is then taken as a third multi-modal encoding vector sequence together with the output of the text encoder.

[0197] The embodiment of the application encodes sample data of multiple different modalities by calling the encoders corresponding to the multiple modalities in the first speech generation model respectively, obtains a multi-modal encoding vector sequence, and further provides three different modal combination encoding modes, so that the first speech generation model fully learns the features of different modalities of the visual modality, the text modality and the speech modality, can improve the ability of the speech generation model to perform a speech generation task, improve the accuracy of speech generation, and at the same time, the combination mode of different modalities can ensure that the execution of the speech generation task can adapt to different application scenarios, and improve user experience.

[0198] With reference to FIG. 3A, the step 103 is explained.

[0199] In step 104, a decoder is called based on the multi-modal encoding vector sequence to perform decoding, and decoded text is obtained.

[0200] In some embodiments, the first speech generation model is a pre-trained speech recognition model, and after pre-training, the representation space of the text modality and the representation space of the speech modality of the first speech generation model have been unified into the same representation space.

[0201] Here, the representation space refers to a vector space composed of feature vectors or feature representations, used to describe and represent different attributes, features or information of data. In different modalities, the representation space embodies the features of different types of data and their representation in the vector space. For example, the representation space of the visual modality is a vector space composed of visual feature vectors, and the representation space of the text modality is a vector space composed of text feature vectors.

[0202] Referring to FIG. 3G, FIG. 3G is a flowchart of the alignment of the length of the encoding vector sequence provided by the embodiment of the application. Before step 104 of FIG. 3A is performed, the length of the multi-modal encoding vector sequence can be aligned through steps 201 to 202 of FIG. 3G, which are explained in detail as follows.

[0203] In step 201, the visual encoding vector sequence and the text encoding vector sequence are mapped to obtain vector sequence representations of the visual encoding vector sequence and the text encoding vector sequence in the representation space.

[0204] In some embodiments, the visual encoding vector sequence and the text encoding vector sequence are mapped by a linear mapping function to obtain vector sequence representations of the visual encoding vector sequence and the text encoding vector sequence in the same representation space.

[0205] As an example, referring to FIG. 6, FIG. 6 is a schematic diagram of alignment of a sequence of visual encoding vectors, a sequence of vectors in a representation space of visual encoding vectors is represented as "E1 E2 E3 E3 E4 E5 E5 E6", and a sequence of vectors in a representation space of text encoding vectors is represented as "T1 T2 T3 T4 T5 T6".

[0206] In step 202, the length of the vector sequence representation of the sequence of visual encoding vectors is adjusted to be the same as the length of the vector sequence representation of the sequence of text encoding vectors.

[0207] As an example, the length of the vector sequence representation of the sequence of visual encoding vectors in the above example is 8, and the length of the vector sequence representation of the sequence of text encoding vectors is 6. Since there is redundant information "E3" and "E5" in the vector sequence representation of the sequence of visual encoding vectors, the redundant information in the vector sequence representation of the sequence of visual encoding vectors is deleted to obtain the vector sequence representation of the sequence of visual encoding vectors as "E1 E2 E3 E4 E5 E6". At this time, the length of the vector sequence representation of the sequence of visual encoding vectors is the same as the length of the vector sequence representation of the sequence of text encoding vectors.

[0208] The training process of the first speech generation model according to the embodiments of the present application is performed on the basis of the pre-trained speech recognition model. On the basis that the representation space of the text modality and the representation space of the speech modality have been unified to the same representation space, the conversion function from the prompt image sequence to the speech text can be realized on the basis of the original function of the pre-trained speech recognition model through fine-tuning. Meanwhile, the multiple functions of the pre-trained speech recognition model itself can be reused, such as recognizing the prompt image sequence of the target object, text translation, and speech generation, thereby avoiding repeated development and improving development efficiency.

[0209] By aligning the lengths of the feature sequences of the two modalities, they can be more consistent in the feature space, thereby facilitating the speech generation model to perform the speech generation task. This processing process can improve the overall efficiency and performance of the speech generation model when processing multi-modal data.

[0210] Continuing to refer to FIG. 3A, the step 104 is explained as follows.

[0211] In step 105, the probability distribution of the decoded text and the sample data of the multiple modalities is determined, and the target loss is determined based on the probability distribution.

[0212] In some embodiments, the determining, in step 105, the probability distribution of the decoded text and the sample data of the plurality of modalities can be implemented by: determining a probability distribution of the sequence of encoding vectors of the sample data of at least two modalities with respect to preset conditions, wherein different modalities correspond to different preset conditions; determining a probability distribution of the speech text with respect to the sample data of at least one modality; and determining a probability distribution of the decoded text with respect to the sample data of at least one modality.

[0213] Here, the determining a probability distribution of the sequence of encoding vectors of the sample data of at least two modalities with respect to preset conditions, wherein different modalities correspond to different preset conditions, can depend on the data involved in the encoding structure of the model; the determining a probability distribution of the speech text with respect to the sample data of at least one modality, that is, a conditional probability distribution of the speech text conditioned on the sample data of at least one modality; and the determining a probability distribution of the decoded text with respect to the sample data of at least one modality, that is, a conditional probability distribution of the decoded text conditioned on the sample data of at least one modality. Different modalities correspond to different conditions, which can depend on the encoding structure of the model.

[0214] For example, in the architecture of the speech generation model shown in FIG. 4A, a text decoder, a visual encoder corresponding to a visual modality, and a text encoder corresponding to a text modality are shown. The speech generation model of FIG. 4A only includes two modalities, the visual modality and the text modality, so the probability distribution of the decoded text and the sample data of the plurality of modalities is determined based on the visual modality and the text modality, and the target loss is determined based on the probability distribution. For the architecture of FIG. 4A, the preset condition corresponding to the text modality can be the decoded text and the speech text, and the preset condition corresponding to the visual modality can be the decoded text and the sequence of prompt image. The condition corresponding to the speech text is the sequence of prompt image, and the condition corresponding to the decoded text is the sequence of prompt image.

[0215] For example, in the architecture of the speech generation model shown in FIG. 4B, a text decoder, a visual encoder corresponding to a visual modality, a text encoder corresponding to a text modality, and a speech encoder corresponding to a speech modality are shown. The speech generation model of FIG. 4B is further trained in combination with a language modality compared to FIG. 4A, so the probability distribution of the decoded text and the sample data of the plurality of modalities is determined based on the speech modality and the text modality, and the target loss is determined based on the probability distribution. For the architecture of FIG. 4B, the preset condition corresponding to the text modality can be the decoded text and the speech text, and the preset condition corresponding to the speech modality can be the decoded text and the speech signal feature. The condition corresponding to the speech text is the speech signal feature, and the condition corresponding to the decoded text is the speech signal feature.

[0216] For example, in the architecture of the speech generation model shown in FIG. 4C, the visual encoding vector sequence output by the visual encoder and the speech encoding vector sequence output by the speech encoder are mapped by feature concatenation to obtain a fusion encoding vector sequence, which is then combined with the output of the text encoder as a third multi-modal encoding vector sequence. Thus, the probability distribution of the decoded text and the sample data of the multiple modalities is determined based on the visual modality, the speech modality, and the text modality, and then the target loss is determined based on the probability distribution. For the architecture of FIG. 4C, the preset condition corresponding to the text modality can be the decoded text and the speech text, the fusion encoding vector sequence corresponds to the preset condition, which can be the decoded text, the prompt image sequence, and the speech signal feature; the condition corresponding to the speech text can be the prompt image sequence and the speech signal feature; and the condition corresponding to the decoded text can be the prompt image sequence and the speech signal feature.

[0217] The embodiments of the present application determine the probability distribution of the encoding vector sequence of the sample data of at least two modalities with respect to the preset condition. By setting different preset conditions for different modalities and calculating the probability distribution of the encoding vector sequence, the characteristics and internal relations of the data of each modality can be better understood. For example, in cross-modal retrieval of images and texts, by constructing semantic spaces of different modalities and learning the semantic distribution of samples, the sample semantics belonging to different modalities can be more accurately aligned. Different modalities correspond to different preset conditions, which enables the model to better adapt to the characteristics of various modalities of data, thereby improving the generalization ability of the model in multi-modal tasks. By calculating the probability distribution of the speech text and the modal sample data, the relationship between speech and modalities such as vision and text can be more accurately modeled, thereby improving the accuracy of speech recognition and generation. By calculating the probability distribution of the decoded text and the modal sample data, the consistency of the decoded text and the input modal data can be ensured, and the decoding quality can be improved. For example, in a text generation task, by modeling the conditional probability distribution, text that better meets the context and semantic requirements can be generated. Through the above-mentioned manner, more intelligent and flexible multi-modal interaction can be achieved, and the convenience and user experience of the interaction can be improved.

[0218] In some embodiments, the determination of the target loss based on the probability distribution in step 105 can be achieved by the following manner:

[0219] For the encoding vector sequence of the sample data of at least two modalities, the difference between the probability distributions of the encoding vector sequences of different modalities is determined based on the probability distribution of the encoding vector sequence of the sample data of at least two modalities with respect to the preset condition, as a sub-target loss of the encoding vector sequence; the sub-target loss of the speech text negatively related to the probability distribution is determined based on the probability distribution of the speech text with respect to the sample data of at least one modality; and the sub-target loss of the decoded text negatively related to the probability distribution is determined based on the probability distribution of the decoded text with respect to the sample data of at least one modality.

[0220] For example, in the architecture shown in FIG. 4A, the difference between the probability distribution of the encoding vector sequence of the sample data of the visual modality and the text modality and the probability distribution of the preset condition is determined as the sub-target loss of the encoding vector sequence based on the probability distribution; the sub-target loss of the speech text is determined based on the probability distribution of the sample data of the visual modality and the probability distribution of the speech text is negatively related to the probability distribution; the sub-target loss of the decoding text is determined based on the probability distribution of the sample data of the visual modality and the probability distribution of the speech text is negatively related to the probability distribution; the sub-target loss of the encoding vector sequence, the sub-target loss of the speech text and the sub-target loss of the decoding text are fused to obtain the target loss.

[0221] For example, in the architecture shown in FIG. 4B, the difference between the probability distribution of the encoding vector sequence of the sample data of the speech modality and the text modality and the probability distribution of the preset condition is determined as the sub-target loss of the encoding vector sequence based on the probability distribution; the sub-target loss of the speech text is determined based on the probability distribution of the sample data of the speech modality and the probability distribution of the speech text is negatively related to the probability distribution; the sub-target loss of the decoding text is determined based on the probability distribution of the sample data of the speech modality and the probability distribution of the speech text is negatively related to the probability distribution; the sub-target loss of the encoding vector sequence, the sub-target loss of the speech text and the sub-target loss of the decoding text are fused to obtain the target loss.

[0222] For example, in the architecture shown in FIG. 4C, the difference between the probability distribution of the encoding vector sequence of the sample data of the visual modality, the speech modality and the text modality and the probability distribution of the preset condition is determined as the sub-target loss of the encoding vector sequence based on the probability distribution; the sub-target loss of the speech text is determined based on the probability distribution of the sample data of the visual modality and the speech modality and the probability distribution of the speech text is negatively related to the probability distribution; the sub-target loss of the decoding text is determined based on the probability distribution of the sample data of the visual modality and the speech modality and the probability distribution of the speech text is negatively related to the probability distribution; the sub-target loss of the encoding vector sequence, the sub-target loss of the speech text and the sub-target loss of the decoding text are fused to obtain the target loss.

[0223] Next, the three different architectures of the speech generation model shown in FIGS. 4A to 4C are described in detail.

[0224] For the architecture of the speech generation model shown in FIG. 4A, referring to FIG. 3H, which is a first flow diagram for determining a probability distribution provided by an embodiment of the present application, the “determining a probability distribution of the decoded text and the sample data of the plurality of modalities” in step 105 of FIG. 3A can be implemented by determining a probability distribution of the encoding vector sequence of the sample data of the visual modality and the text modality with respect to a preset condition, determining a probability distribution of the speech text with respect to the prompt image sequence, and determining a probability distribution of the decoded text with respect to the prompt image sequence based on the encoding vector sequence of the sample data of the visual modality and the text modality described above. The following will be specifically described in combination with steps 1051A to 1053A of FIG. 3H.

[0225] In step 1051A, a first probability distribution of the text encoding vector sequence with respect to the decoded text and the speech text, and a second probability distribution of the visual encoding vector sequence with respect to the decoded text and the prompt image sequence are determined.

[0226] In some embodiments, the above step 1051A can be implemented in the following manner: a pre-trained first probability distribution model is called to determine the first probability distribution of the text encoding vector sequence with respect to the decoded text and the speech text, wherein the first probability distribution model is trained in the following manner: first sample data is obtained, wherein the first sample data includes sample decoded text and sample speech text, and a reference probability distribution of the sample text encoding vector sequence with respect to the sample decoded text and the sample speech text; the sample decoded text, the sample speech text, and the sample text encoding vector sequence are taken as an input of an initialized first probability distribution model, and a predicted probability distribution of the text encoding vector sequence with respect to the sample decoded text and the sample speech text is output; based on a difference between the predicted probability distribution and the reference probability distribution, a loss value is determined; and according to the loss value, parameters of the first probability distribution model are updated.

[0227] By calculating the probability distribution of the text encoding vector sequence with respect to the decoded text and the speech text, the semantic information of different modalities can be more accurately aligned. This alignment can ensure consistency and correlation of the text, speech, and visual information in the generation process, and the probability distribution model can capture the complex dependency relationship between the text encoding vector and the decoded text and the speech text, thereby improving the semantic understanding ability of the model for multi-modal data.

[0228] As an example, the first probability distribution of the text encoding vector sequence with respect to the decoded text and the speech text can be represented as The first probability distribution model can be a Conditional Random Field (CRF), a Hidden Markov Model (HMM).

[0229] As an example, a difference operation can be used to calculate the difference between the predicted probability distribution of the text encoding vector sequence with respect to the decoded text and the speech text and the reference probability distribution, and the square or absolute value of the difference is taken as the loss value; or an exponential operation is used to calculate the cross-entropy between the predicted probability distribution of the text encoding vector sequence with respect to the decoded text and the speech text and the reference probability distribution as the loss value. For example, the predicted probability distribution of the text encoding vector sequence with respect to the decoded text and the speech text is p1, and the reference probability distribution is p2, then the difference between the predicted probability distribution of the text encoding vector sequence with respect to the decoded text and the speech text and the reference probability distribution is p1-p2, and the loss value can be represented as |p1-p2|.

[0230] In some embodiments, the second probability distribution of the visual encoding vector sequence with respect to the decoded text and the sequence of prompt images can be determined by a pre-trained second probability distribution model, i.e., the conditional probability distribution of the visual encoding vector sequence with respect to the decoded text and the sequence of prompt images. The second probability distribution model can be trained in the following way: first, a second sample data set is obtained, the sample data including decoded text, a sequence of prompt images, and a reference probability distribution of a visual encoding vector sequence with respect to the decoded text and the sequence of prompt images; then, the decoded text and the sequence of prompt images in the sample data are input into the initialized second probability distribution model, and the predicted probability distribution of the visual encoding vector sequence with respect to the decoded text and the sequence of prompt images is output; finally, the loss value is determined based on the difference between the predicted probability distribution of the visual encoding vector sequence with respect to the decoded text and the sequence of prompt images and the reference probability distribution, and the parameters of the second probability distribution model are updated based on the loss value by a back propagation algorithm.

[0231] As an example, the second probability distribution of the visual encoding vector sequence with respect to the decoded text and the sequence of prompt images can be represented as The second probability distribution model can be a conditional random field, a hidden Markov model.

[0232] As an example, a difference operation can be used to calculate the difference between the predicted probability distribution of the visual encoding vector sequence with respect to the decoded text and the sequence of prompt images and the reference probability distribution, and the square or absolute value of the difference is taken as the loss value; or an exponential operation is used to calculate the cross-entropy between the predicted probability distribution of the visual encoding vector sequence with respect to the decoded text and the sequence of prompt images and the reference probability distribution as the loss value.

[0233] In step 1052A, the third probability distribution of the speech text with respect to the sequence of prompt images is determined.

[0234] In some embodiments, the third probability distribution of the speech text with respect to the sequence of prompt images, i.e., the conditional probability distribution of the speech text with respect to the sequence of prompt images, can be determined by a pre-trained third probability distribution model. The third probability distribution model can be trained in the following manner: first, a third sample data set is obtained, the sample data including a sequence of prompt images and a reference probability distribution of the speech text with respect to the sequence of prompt images; then, the sequence of prompt images in the sample data is taken as the input of the initialized third probability distribution model, and the predicted probability distribution of the speech text with respect to the sequence of prompt images is output; finally, a loss value is determined based on the difference between the predicted probability distribution of the speech text with respect to the sequence of prompt images and the reference probability distribution, and the parameters of the third probability distribution model are updated based on the loss value by a back propagation algorithm.

[0235] As an example, the third probability distribution of the speech text with respect to the sequence of prompt images can be represented as p(x text |xcued-speech), and the third probability distribution model can be a conditional random field, a hidden Markov model. A difference operation can be used to calculate the difference between the predicted probability distribution of the speech text with respect to the sequence of prompt images and the reference probability distribution, and the square or absolute value of the difference is taken as the loss value; for example, the predicted probability distribution of the speech text with respect to the sequence of prompt images is p3, and the reference probability distribution is p4, then the difference between the predicted probability distribution of the speech text with respect to the sequence of prompt images and the reference probability distribution is p3-p4, and the loss value can be represented as |p3-p4|. Or an exponential operation is used to calculate the cross-entropy of the predicted probability distribution of the speech text with respect to the sequence of prompt images and the reference probability distribution as the loss value.

[0236] As an example, the speech text includes 3, A, B, and C, respectively, and the reference probability distribution of the speech text with respect to the sequence of prompt images is represented as p1=[0.5, 0.3, 0.2], and the predicted probability distribution is represented as p2=[0.4, 0.4, 0.2], and the formula of the cross-entropy is represented as formula (1):

[0237] Wherein, H(p1, p2) is the cross-entropy of the predicted probability distribution of the speech text with respect to the sequence of prompt images and the reference probability distribution, i.e., the loss value, n is the number of speech texts, p 1,i is the probability of the i-th speech text with respect to the sequence of prompt images in the reference probability distribution, and p 2,i is the probability of the i-th speech text with respect to the sequence of prompt images in the predicted probability distribution.

[0238] Then the loss value H(p1, p2) =-(0.5log(0.4)+0.3log(0.4)+0.2log(0.2))=0.6388.

[0239] In step 1053A, in the case where the target language of the speech generation task of the first speech generation model is different from the source language, the fourth probability distribution of the decoded text with respect to the prompt image sequence is determined.

[0240] In some embodiments, when the target language of the speech generation task of the first speech generation model is different from the source language, for example, the target language of the speech generation task of the first speech generation model is English and the source language is Chinese, the Chinese of the source language needs to be translated into English first.

[0241] In some embodiments, the fourth probability distribution of the decoded text with respect to the prompt image sequence, i.e., the conditional probability distribution of the decoded text with respect to the prompt image sequence, can be determined by a pre-trained fourth probability distribution model. The fourth probability distribution model can be trained in the following way: first, a fourth sample data set is obtained, the sample data including the prompt image sequence, the source speech text and the reference probability distribution of the decoded text with respect to the prompt image sequence; then, the prompt image sequence and the source speech text in the sample data are taken as the input of the initialized fourth probability distribution model, and the predicted probability distribution of the decoded text with respect to the prompt image sequence is output; finally, the loss value is determined based on the difference between the predicted probability distribution of the decoded text with respect to the prompt image sequence and the reference probability distribution, and the parameters of the fourth probability distribution model are updated based on the loss value by the back propagation algorithm.

[0242] As an example, the fourth probability distribution of the decoded text with respect to the prompt image sequence can be represented as The fourth probability distribution model can be a conditional random field, a hidden Markov model. The difference between the predicted probability distribution of the decoded text with respect to the prompt image sequence and the reference probability distribution can be calculated using difference operation, and the square or absolute value of the difference is taken as the loss value; or the cross-entropy of the predicted probability distribution of the decoded text with respect to the prompt image sequence and the reference probability distribution is calculated using exponential operation as the loss value.

[0243] The embodiments of the present application can ensure consistency and alignment of information between different modalities (text, speech, and vision) in the generation process by calculating the first probability distribution between the text encoding vector sequence and the decoded text, and the second probability distribution between the visual encoding vector sequence and the decoded text and the prompt image sequence. Determining the third probability distribution of the speech text relative to the prompt image sequence can guide the speech generation model to generate speech text that matches the prompt image better. In the case where the target language and the source language are different, calculating the fourth probability distribution of the decoded text relative to the prompt image sequence helps to optimize the effect of cross-lingual speech generation, which enables the speech generation model to better handle cross-lingual speech synthesis tasks and generate speech that better meets the characteristics and semantics of the target language. By considering the probability distribution of multiple modalities, the model can better adapt to different types of input data, including text, speech, and visual information. This multi-modal processing approach enables the model to generate high-quality outputs more flexibly when faced with complex input scenarios.

[0244] In some embodiments, referring to FIG. 3I, which is a first flowchart of determining a target loss provided by the embodiments of the present application, the "determining a target loss based on the probability distribution" in step 105 of FIG. 3A can be achieved by determining the difference between the probability distributions of the encoding vector sequences of the visual and text modalities based on the probability distribution of the encoding vector sequences of the sample data of the visual and text modalities relative to the preset conditions as the sub-target loss of the encoding vector sequences, determining the sub-target loss of the speech text negatively related to the probability distribution based on the probability distribution of the speech text relative to the sample data of the visual modality, and determining the sub-target loss of the speech text negatively related to the probability distribution based on the probability distribution of the decoded text relative to the sample data of the visual modality. The following will be described in detail in combination with steps 1054A to 1057A of step 105 of FIG. 3I.

[0245] In step 1054A, the difference between the first probability distribution and the second probability distribution is determined as a first sub-target loss.

[0246] In some embodiments, the difference between the first probability distribution and the second probability distribution can be determined by divergence calculation. For example, Kullback-Leibler divergence (KL divergence), cross entropy.

[0247] As an example, the first sub-target loss can be determined by formula (2):

[0248] wherein, is the first probability distribution, is the second probability distribution, y textis the decoded text in the target language corresponding to the target speech, xc text are the prompt image sequence and the corresponding source speech text in the source language, respectively, θcs E is the parameter of the visual encoder.

[0249] In step 1055A, a second sub-target loss negatively related to the third probability distribution is determined.

[0250] As an example, the second sub-target loss negatively related to the third probability distribution can be determined by formula (3) shown below.

[0251] where θt D is the parameter of the decoder.

[0252] In step 1056A, a third sub-target loss negatively related to the fourth probability distribution is determined.

[0253] As an example, the third sub-target loss negatively related to the fourth probability distribution can be determined by formula (4) shown below.

[0254] In step 1057A, the first sub-target loss, the second sub-target loss and the third sub-target loss are fused to obtain the first target loss.

[0255] As an example, the first sub-target loss, the second sub-target loss and the third sub-target loss are weighted and summed by formula (5) shown below to obtain the first target loss.

[0256] where α is the weight parameter of the third sub-target loss.

[0257] The embodiments of the present application comprehensively consider the information of the three modalities of text, speech and vision, and fully utilize the complementarity between the three modalities. The text provides semantic information, the speech provides prosody and emotional information, and the visual information provides intuitive description of the scene and context, in this way, the model can generate more natural and accurate speech. Through explicit probability distribution calculation and loss function design, the model can more efficiently learn the relationship between the text, speech and visual information, so as to converge faster in the training process, which helps to reduce the consumption of training time and computing resources, and improves the development efficiency of the model.

[0258] The architecture of the speech generation modality shown in FIG. 4B is described below. Referring to FIG. 3J, which is a second flow diagram for determining a probability distribution, the “determining a probability distribution of the decoded text and the sample data of the plurality of modalities” in step 105 of FIG. 3A can also be implemented by determining, for the sequence of encoding vectors of the sample data of the speech modality and the text modality, a probability distribution of the sequence of encoding vectors of the sample data of the speech modality and the text modality with respect to the preset condition, determining a probability distribution of the speech text with respect to the speech signal feature, and determining a probability distribution of the decoded text with respect to the speech signal feature. This is implemented in steps 1051B to 1053B of FIG. 3J, which are described below in detail.

[0259] In step 1051B, a fifth probability distribution of the sequence of text encoding vectors with respect to the decoded text and the speech text, and a sixth probability distribution of the sequence of speech encoding vectors with respect to the decoded text and the speech signal feature are determined.

[0260] In some embodiments, the fifth probability distribution of the sequence of text encoding vectors with respect to the decoded text and the speech text, i.e., the conditional probability distribution of the sequence of text encoding vectors with respect to the decoded text and the speech text, can be determined by a pre-trained fifth probability distribution model. Here, the fifth probability distribution is merely for the purpose of distinguishing from the first probability distribution described above.

[0261] In some embodiments, the sixth probability distribution of the sequence of speech encoding vectors with respect to the decoded text and the speech signal feature, i.e., the conditional probability distribution of the sequence of speech encoding vectors with respect to the decoded text and the speech signal feature, can be determined by a pre-trained sixth probability distribution model. The sixth probability distribution model can be trained in the following manner: first, a sixth sample data set is obtained, the sample data including the decoded text, the speech signal feature, and a reference probability distribution of the sequence of speech encoding vectors with respect to the decoded text and the speech signal feature; then, the decoded text and the speech signal feature in the sample data are taken as the input of the initialized sixth probability distribution model, and the predicted probability distribution of the sequence of speech encoding vectors with respect to the decoded text and the speech signal feature is output; finally, a loss value is determined based on the difference between the predicted probability distribution and the reference probability distribution of the sequence of speech encoding vectors with respect to the decoded text and the speech signal feature, and the parameters of the sixth probability distribution model are updated based on the loss value by a back propagation algorithm.

[0262] As an example, the sixth probability distribution of the sequence of speech encoding vectors with respect to the decoded text and the speech signal feature can be represented as The sixth probability distribution model can be a conditional random field or a hidden Markov model.

[0263] As an example, a difference operation can be used to calculate the difference between the predicted probability distribution of the speech coding vector sequence with respect to the decoded text and the speech signal features and the reference probability distribution, and the square or absolute value of the difference is taken as the loss value; or an exponential operation is used to calculate the cross-entropy between the predicted probability distribution of the speech coding vector sequence with respect to the decoded text and the speech signal features and the reference probability distribution as the loss value.

[0264] In step 1052B, the seventh probability distribution of the speech text with respect to the speech signal features is determined.

[0265] In some embodiments, the seventh probability distribution of the speech text with respect to the speech signal features, i.e., the conditional probability distribution of the speech text with respect to the speech signal features, can be determined by a pre-trained seventh probability distribution model. The seventh probability distribution model can be trained in the following way: first, a seventh sample data set is obtained, the sample data including the speech signal features and the reference probability distribution of the speech text with respect to the speech signal features; then, the speech signal features in the sample data are taken as the input of the initialized seventh probability distribution model, and the predicted probability distribution of the speech text with respect to the speech signal features is output; finally, the loss value is determined based on the difference between the predicted probability distribution of the speech text with respect to the speech signal features and the reference probability distribution, and the parameters of the seventh probability distribution model are updated based on the loss value by a back propagation algorithm.

[0266] As an example, the seventh probability distribution of the speech text with respect to the speech signal features can be represented as The seventh probability distribution model can be a conditional random field, a hidden Markov model. A difference operation can be used to calculate the difference between the predicted probability distribution of the speech text with respect to the speech signal features and the reference probability distribution, and the square or absolute value of the difference is taken as the loss value; or an exponential operation is used to calculate the cross-entropy between the predicted probability distribution of the speech text with respect to the speech signal features and the reference probability distribution as the loss value.

[0267] In step 1053B, in the case where the target language of the speech generation task of the first speech generation model is different from the source language, the eighth probability distribution of the decoded text with respect to the speech signal features is determined.

[0268] In some embodiments, when the target language of the speech generation task of the first speech generation model is different from the source language, the eighth probability distribution of the decoded text with respect to the speech signal feature, i.e., the conditional probability distribution of the decoded text with respect to the speech signal feature, can be determined by a pre-trained eighth probability distribution model. The eighth probability distribution model can be trained in the following way: first, an eighth sample data set is obtained, and the sample data includes the speech signal feature, the source speech text, and the reference probability distribution of the decoded text with respect to the speech signal feature; then, the speech signal feature and the source speech text in the sample data are taken as the input of the initialized eighth probability distribution model, and the predicted probability distribution of the decoded text with respect to the speech signal feature is output; finally, the loss value is determined based on the difference between the predicted probability distribution of the decoded text with respect to the speech signal feature and the reference probability distribution, and the parameters of the eighth probability distribution model are updated based on the loss value by a back propagation algorithm.

[0269] As an example, the eighth probability distribution of the decoded text with respect to the speech signal feature can be represented as The eighth probability distribution model can be a conditional random field, a hidden Markov model.

[0270] As an example, the difference between the predicted probability distribution of the decoded text with respect to the speech signal feature and the reference probability distribution can be calculated using a difference operation, and the square or absolute value of the difference is taken as the loss value; or the cross-entropy of the predicted probability distribution of the decoded text with respect to the speech signal feature and the reference probability distribution is calculated using an exponential operation as the loss value.

[0271] By using the pre-trained probability distribution model, the conditional probability relationship between the decoded text and the speech signal feature can be more accurately modeled, and the complex dependency relationship between the speech signal feature and the decoded text can be captured, thereby improving the accuracy of cross-lingual speech generation, enabling the speech generation model to generate text content that better meets the user's needs according to the user's speech signal feature, and providing personalized services.

[0272] In some embodiments, referring to FIG. 3K, which is a second flowchart of determining a target loss according to an embodiment of the present application, the “determining a target loss based on the probability distribution” in step 105 of FIG. 3A can also be implemented by determining a difference between the probability distribution of the encoding vector sequence of the sample data of the speech modality and the text modality with respect to the preset condition as a sub-target loss of the probability distribution of the encoding vector sequence of the speech modality and the text modality; determining a sub-target loss of the speech text negatively related to the probability distribution based on the probability distribution of the sample data of the speech modality with respect to the speech modality; and determining a sub-target loss of the speech text negatively related to the probability distribution based on the probability distribution of the sample data of the speech modality with respect to the speech modality. The implementation is described below in combination with steps 1054B to 1057B of FIG. 3K, which are described in detail as follows.

[0273] In step 1054B, a difference between the fifth probability distribution and the sixth probability distribution is determined as a fourth sub-target loss.

[0274] In some embodiments, the difference between the fifth probability distribution and the sixth probability distribution can be determined by divergence calculation, for example, KL divergence, cross-entropy.

[0275] As an example, the fourth sub-target loss can be determined by formula (6) as follows:

[0276] where ximpaired-speech is a speech signal feature in the sample data, θ isE is a parameter of the speech encoder, y text is the decoding text corresponding to the target language, x text is the source speech text, is a fifth probability distribution of the text encoding vector sequence with respect to the decoding text and the speech text, is a sixth probability distribution of the speech encoding vector sequence with respect to the decoding text and the speech signal feature.

[0277] In step 1055B, a fifth sub-target loss negatively related to a seventh probability distribution is determined.

[0278] As an example, the fifth sub-target loss negatively related to the seventh probability distribution can be determined by formula (7) as shown below.

[0279] In step 1056B, a sixth sub-target loss negatively related to an eighth probability distribution is determined.

[0280] As an example, the sixth sub-target loss negatively related to the eighth probability distribution can be determined by formula (8) as shown below.

[0281] In step 1057B, the fourth sub-target loss, the fifth sub-target loss and the sixth sub-target loss are fused to obtain the second target loss.

[0282] As an example, the fourth sub-target loss, the fifth sub-target loss and the sixth sub-target loss are weighted and summed by the formula (9) shown below to obtain the second target loss.

[0283] Wherein, a is a weight parameter of the sixth sub-target loss.

[0284] By calculating the probability distribution difference between different modalities (such as the difference between the fifth probability distribution and the sixth probability distribution) and optimizing it as a sub-target loss, the embodiments of the present application can generate more natural and contextually consistent speech content. Through explicit probability distribution calculation and loss function design, the model can consider multiple factors during the training process, avoiding overfitting problems that may be caused by single target optimization. The speech generation model can more efficiently learn the relationship between different modalities, thus converging faster during the training process. This efficient training method can reduce the consumption of training time and computing resources. In summary, by calculating the probability distribution between different modalities and optimizing the target loss based on these distributions, the performance of the multi-modal model in consistency, generation quality, robustness and generalization ability can be significantly improved, making it perform better in complex multi-modal tasks.

[0285] The architecture of the speech generation modality shown in FIG. 4C will be described below. Referring to FIG. 3L, which is a third flowchart for determining probability distribution provided by the embodiments of the present application, the “determining the probability distribution of the decoded text and the sample data of the multiple modalities” in step 105 of FIG. 3A can also be implemented by determining the probability distribution of the encoding vector sequence of the sample data of the visual modality, the speech modality and the text modality with respect to the preset condition based on the encoding vector sequence of the sample data of the visual modality, the speech modality and the text modality described above; determining the probability distribution of the speech text with respect to the prompt image sequence and the speech signal feature; and determining the probability distribution of the decoded text with respect to the prompt image sequence and the speech signal feature. The implementation of the above will be described below in combination with steps 1051C to 1053C of FIG. 3L, which will be described in detail as follows.

[0286] In step 1051C, the ninth probability distribution of the text encoding vector sequence with respect to the decoded text and the speech text is determined, and the tenth probability distribution of the encoding vector sequence with respect to the decoded text, the prompt image sequence and the speech signal feature is fused.

[0287] In some embodiments, the ninth probability distribution model can be pre-trained to determine a ninth probability distribution of the text encoding vector sequence with respect to the decoded text and the speech text, i.e., a conditional probability distribution of the text encoding vector sequence with respect to the decoded text and the speech text. Here, the ninth probability distribution is merely for the purpose of distinguishing from the first probability distribution described above, and the training process of the ninth probability distribution model is the same as that of the first probability distribution model described above.

[0288] In some embodiments, the tenth probability distribution model can be pre-trained to determine a tenth probability distribution of the fusion encoding vector sequence with respect to the decoded text, the prompt image sequence, and the speech signal feature, i.e., a conditional probability distribution of the fusion encoding vector sequence with respect to the decoded text, the prompt image sequence, and the speech signal feature. The tenth probability distribution model can be trained in the following way: first, obtain a tenth sample data set, the sample data including the decoded text, the prompt image sequence, the speech signal feature, and a reference probability distribution of the fusion encoding vector sequence with respect to the decoded text, the prompt image sequence, and the speech signal feature; then, input the decoded text, the prompt image sequence, and the speech signal feature in the sample data into the initialized tenth probability distribution model as input, and output a predicted probability distribution of the fusion encoding vector sequence with respect to the decoded text, the prompt image sequence, and the speech signal feature; finally, determine a loss value based on the difference between the predicted probability distribution of the fusion encoding vector sequence with respect to the decoded text, the prompt image sequence, and the speech signal feature and the reference probability distribution, and update the parameters of the tenth probability distribution model based on the loss value through a back propagation algorithm.

[0289] As an example, the tenth probability distribution of the fusion encoding vector sequence with respect to the decoded text, the prompt image sequence, and the speech signal feature can be represented as The tenth probability distribution model can be a conditional random field or a hidden Markov model.

[0290] As an example, the difference between the predicted probability distribution of the fusion encoding vector sequence with respect to the decoded text, the prompt image sequence, and the speech signal feature and the reference probability distribution can be calculated using a difference operation, and the square or absolute value of the difference can be taken as the loss value; or the cross-entropy of the predicted probability distribution of the fusion encoding vector sequence with respect to the decoded text, the prompt image sequence, and the speech signal feature and the reference probability distribution can be calculated using an exponential operation as the loss value.

[0291] In step 1052C, a eleventh probability distribution of the speech text with respect to the prompt image sequence and the speech signal feature is determined.

[0292] In some embodiments, the eleventh probability distribution model can be used to determine the eleventh probability distribution of the speech text with respect to the prompt image sequence and the speech signal feature, i.e., the conditional probability distribution of the speech text with respect to the prompt image sequence and the speech signal feature. The eleventh probability distribution model can be trained in the following manner: first, a tenth sample data set is obtained, the sample data including the prompt image sequence, the speech signal feature, and the reference probability distribution of the speech text with respect to the prompt image sequence and the speech signal feature; then, the prompt image sequence and the speech signal feature in the sample data are taken as the input of the initialized eleventh probability distribution model, and the predicted probability distribution of the speech text with respect to the prompt image sequence and the speech signal feature is output; finally, a loss value is determined based on the difference between the predicted probability distribution and the reference probability distribution of the speech text with respect to the prompt image sequence and the speech signal feature, and the parameters of the eleventh probability distribution model are updated based on the loss value through a back propagation algorithm.

[0293] As an example, the eleventh probability distribution of the speech text with respect to the prompt image sequence and the speech signal feature can be represented as p(x text |xcued-speech,ximpaired-speech), and the eleventh probability distribution model can be a conditional random field or a hidden Markov model.

[0294] As an example, the difference between the predicted probability distribution and the reference probability distribution of the speech text with respect to the prompt image sequence and the speech signal feature can be calculated using a difference operation, and the square or absolute value of the difference can be taken as the loss value; or the cross-entropy of the predicted probability distribution and the reference probability distribution of the speech text with respect to the prompt image sequence and the speech signal feature can be calculated using an exponential operation as the loss value. For example, the predicted probability distribution of the speech text with respect to the prompt image sequence and the speech signal feature is p5, and the reference probability distribution is p6, then the difference between the predicted probability distribution and the reference probability distribution of the speech text with respect to the prompt image sequence and the speech signal feature is represented as p5-p6, and the loss value can be represented as |p5-p6|.

[0295] In step 1053C, in the case where the target language of the speech generation task of the first speech generation model is different from the source language, the twelfth probability distribution of the decoded text with respect to the prompt image sequence and the speech signal feature is determined.

[0296] In some embodiments, when the target language of the speech generation task of the first speech generation model is different from the source language, the twelfth probability distribution model pre-trained can be used to determine the twelfth probability distribution of the decoded text with respect to the prompt image sequence and the speech signal feature, i.e., the conditional probability distribution of the decoded text with respect to the prompt image sequence and the speech signal feature. The twelfth probability distribution model can be trained in the following way: first, a twelfth sample data set is obtained, which includes the prompt image sequence and the speech signal feature, the source speech text, and the reference probability distribution of the decoded text with respect to the prompt image sequence and the speech signal feature; then, the fusion features of the prompt image sequence and the speech signal feature in the sample data, and the source speech text are used as the input of the initialized twelfth probability distribution model, and the predicted probability distribution of the decoded text with respect to the prompt image sequence and the speech signal feature is output; finally, the loss value is determined based on the difference between the predicted probability distribution of the decoded text with respect to the prompt image sequence and the speech signal feature and the reference probability distribution, and the parameters of the twelfth probability distribution model are updated based on the loss value through the back propagation algorithm.

[0297] As an example, the twelfth probability distribution of the decoded text with respect to the prompt image sequence and the speech signal feature can be represented as The twelfth probability distribution model can be a conditional random field or a hidden Markov model.

[0298] As an example, the difference between the predicted probability distribution of the decoded text with respect to the prompt image sequence and the speech signal feature and the reference probability distribution can be calculated using a difference operation, and the square or absolute value of the difference can be taken as the loss value; or the cross-entropy of the predicted probability distribution of the decoded text with respect to the prompt image sequence and the speech signal feature and the reference probability distribution can be calculated using an exponential operation as the loss value.

[0299] In some embodiments, referring to FIG. 3M, which is a third flowchart for determining the target loss provided by the embodiments of the present application, the step 105 of FIG. 3A “determining the target loss based on the probability distribution” can also be implemented by determining the difference between the probability distribution of the fusion encoding vector sequence and the encoding vector sequence of the text modality based on the encoding vector sequence of the sample data of the visual modality, the speech modality, and the text modality with respect to the preset condition, taking the difference as the sub-target loss of the encoding vector sequence; determining the sub-target loss of the speech text negatively related to the probability distribution based on the probability distribution of the speech text with respect to the sample data of the visual modality and the speech modality; and determining the sub-target loss of the speech text negatively related to the probability distribution based on the probability distribution of the decoded text with respect to the sample data of the visual modality and the speech modality. The implementation of the above is described below in combination with the step 1054C to the step 1057C of FIG. 3M.

[0300] In step 1054C, a difference between the ninth probability distribution and the tenth probability distribution is determined as a seventh sub-target loss.

[0301] In some embodiments, the difference between the ninth probability distribution and the tenth probability distribution can be determined by a divergence calculation. For example, KL divergence, cross-entropy.

[0302] As an example, the difference between the ninth probability distribution and the tenth probability distribution can be determined as the seventh sub-target loss by formula (10) shown as follows.

[0303] In step 1055C, an eighth sub-target loss negatively related to the eleventh probability distribution is determined.

[0304] As an example, the eighth sub-target loss negatively related to the eleventh probability distribution can be determined by formula (11) shown as follows.

[0305] In step 1056C, a ninth sub-target loss negatively related to the twelfth probability distribution is determined.

[0306] As an example, the ninth sub-target loss negatively related to the twelfth probability distribution can be determined by formula (12) shown as follows.

[0307] In step 1057C, the seventh sub-target loss, the eighth sub-target loss and the ninth sub-target loss are fused to obtain a third target loss.

[0308] As an example, the seventh sub-target loss, the eighth sub-target loss and the ninth sub-target loss are weighted and summed by formula (13) shown as follows to obtain the third target loss.

[0309] Wherein, a is a weight parameter of the ninth sub-target loss.

[0310] Continuing to refer to FIG. 3A, the step 105 is explained in continuation.

[0311] In step 106, parameters of the decoder and the at least one encoder are updated based on the target loss to obtain an updated decoder and the plurality of encoders, wherein the updated decoder and the plurality of encoders are used to constitute a second speech generation model, and the second speech generation model is used to generate a target speech text corresponding to the prompt image sequence of the target object.

[0312] In some embodiments, referring to FIG. 3N, which is a first flow diagram of updating parameters according to an embodiment of the present application, the “updating parameters of the decoder and the at least one encoder based on the target loss” in step 106 of FIG. 3A can be implemented by steps 1061A-1062A of FIG. 3N, which are explained in detail as follows.

[0313] In step 1061A, the parameters of the visual encoder are updated based on the first sub-target loss.

[0314] As an example, the parameters of the visual encoder are updated based on the first sub-target loss as shown in equation (2) above. csE .

[0315] In step 1062A, the parameters of the visual encoder and the decoder are updated based on the second sub-target loss and the third sub-target loss.

[0316] As an example, when the target language of the speech generation task of the first speech generation model is the same as the source language, the parameters of the visual encoder are updated based on the second sub-target loss in the manner shown in equation (3) above. csE and the parameters of the decoder are updated based on the second sub-target loss in the manner shown in equation (4) above. tD When the target language of the speech generation task of the first speech generation model is different from the source language, the parameters of the visual encoder are updated based on the third sub-target loss in the manner shown in equation (5) above. csE and the parameters of the decoder are updated based on the third sub-target loss in the manner shown in equation (6) above. tD .

[0317] The first sub-target loss reflects the difference between the text encoding vector sequence and the visual encoding vector sequence. By updating the parameters of the visual encoder based on this loss, it can ensure that the vectors generated by the visual encoder can better align with the text encoding vectors, thereby improving the consistency between multi-modal information. The second sub-target loss is related to the matching degree of the speech text and the prompt image sequence, and the third sub-target loss is related to the negative correlation of the speech text and the prompt image sequence. By updating the parameters of the visual encoder by combining these two losses, the visual encoder can better capture the features of the prompt image sequence and generate encoding vectors that better match the speech text and the decoded text, further improving the fusion effect of visual information and other modal information. This updating method enables the decoder to generate speech text that better matches the semantics of the prompt image, while avoiding generating content that does not match the prompt image, thereby improving the quality and accuracy of speech generation. By updating the parameters for different sub-target losses, the model can better coordinate the relationship between text, speech, and visual information. The update of the visual encoder enables the visual information to better align with the text and speech information, and the update of the decoder ensures that the generated speech text better reflects the content of the visual information. This collaborative optimization method can significantly improve the effect of multi-modal information fusion and generate more natural and consistent output.

[0318] In some embodiments, referring to FIG. 3O, which is a second flow diagram of updating parameters provided in the embodiments of the present application, the "updating parameters of the decoder and at least one encoder based on the target loss" in step 106 of FIG. 3A can be implemented by steps 1061B to 1062B of FIG. 3O, which are explained in detail as follows.

[0319] In step 1061B, the parameters of the speech encoder are updated based on the fourth sub-target loss.

[0320] As an example, as shown in formula (6) above, the parameters of the speech encoder are updated based on the fourth sub-target loss isE .

[0321] In step 1062B, the parameters of the speech encoder and the decoder are updated based on the fifth sub-target loss and the sixth sub-target loss.

[0322] As an example, when the target language of the speech generation task of the first speech generation model is the same as the source language, the parameters of the speech encoder are updated based on the fifth sub-target loss in the manner shown in formula (7) above isE and the parameters of the decoder are updated based on the fifth sub-target loss in the manner shown in formula (7) above tD When the target language of the speech generation task of the first speech generation model is different from the source language, the parameters of the speech encoder are updated based on the sixth sub-target loss in the manner shown in formula (8) above isE and the parameters of the decoder are updated based on the sixth sub-target loss in the manner shown in formula (8) above tD.

[0323] In some embodiments, referring to FIG3P, FIG3P is a schematic diagram of the third process of updating parameters provided in the embodiments of this application. The "updating the parameters of the decoder and at least one encoder based on the target loss" in step 106 of FIG3A can be implemented through steps 1061C to 1062C of FIG3P, which will be described in detail below.

[0324] In step 1061C, the parameters of the visual encoder and the speech encoder are updated based on the seventh sub-target loss.

[0325] As an example, as shown in formula (10) above, the parameters θ of the visual encoder are updated based on the loss of the seventh sub-target. csE and the parameters θ of the speech encoder isE .

[0326] In step 1062C, the parameters of the visual encoder, speech encoder, and decoder are updated based on the eighth sub-target loss and the ninth sub-target loss.

[0327] As an example, when the target language of the first speech generation model's speech generation task is the same as the source language, the parameters θ of the visual encoder are updated based on the eighth sub-target loss, as shown in formula (11) above. csE The parameters θ of the voice encoder isE and the decoder parameter θ tD When the target language of the first speech generation model's speech generation task is different from the source language, as shown in formula (12) above, the parameters θ of the visual encoder are updated based on the ninth sub-target loss. csE The parameters θ of the voice encoder isE and the decoder parameter θ tD .

[0328] In some embodiments, the first speech generation model further includes a speech signal generator, wherein the speech signal generator is used to form a second speech generation model with the decoder and the updated plurality of encoders, the second speech generation model being used to generate speech signals.

[0329] As an example, continuing to refer to Figure 4A, the speech signal generator shown in 4A includes a text-to-unit module and a unit-to-speech module, which together with the decoder and multiple updated encoders form a second speech generation model, which is used to generate speech signals.

[0330] By encoding the sample data of multiple modalities including the image sequence of the prompt speech (corresponding to the visual modality) and the speech text (corresponding to the text modality), the obtained multi-modal encoding vector sequence (that is, the encoding vector sequence of multiple modalities) is calculated with the probability distribution of the decoded text to obtain the target loss, so that the target loss can reflect the difference between the representation spaces of multiple modalities in the first speech generation model. Based on the target loss, the parameters of the first speech generation model are updated through back propagation, so that the first speech generation model can gradually learn the internal relationship between the information of the visual modality and the text modality, improve the performance and generalization ability of the model, and realize the unification of the representation spaces of the visual modality and the text modality in the second speech generation model obtained after training, which means that the second speech generation model can map data of different modalities to a common representation space, so that data of different modalities have similar semantic expressions in the space, laying a foundation for subsequent accurate generation of speech text, and ensuring that the target speech text output by the second speech generation model can accurately match the intention of the prompt speech image of the target object in expression, thereby ensuring the accuracy of generating speech text from the prompt speech image. At the same time, compared with the way of generating speech text by manually inputting text in related technologies, it is more convenient, more efficient, and better user experience.

[0331] Next, the speech generation method of the speech generation model provided by the embodiments of the present application will be described. As described above, the electronic device implementing the speech generation method of the speech generation model of the embodiments of the present application can be a terminal or a server, or a combination of the two. Therefore, the execution subject of each step will not be repeated in the following description.

[0332] Referring to FIG. 3Q, FIG. 3Q is a flowchart of a speech generation method of a speech generation model provided by an embodiment of the present application, wherein the speech generation model is the second speech generation model described above. Next, the steps 301 to 302 shown in FIG. 3Q will be described.

[0333] In step 301, a prompt speech image sequence of a target object is obtained.

[0334] In some embodiments, in each prompt speech image in the prompt speech image sequence, a static image of a hand or lip pose corresponding to a video frame moment when the sample object makes a limb action or a lip action is displayed.

[0335] As an example, the prompt speech image sequence of the target object can be obtained by using a camera to take pictures, an image database, or a network source, etc.

[0336] In step 302, the second speech generation model is called based on the prompt speech image sequence to generate target speech text corresponding to the prompt speech image sequence of the target object.

[0337] In some embodiments, the second speech generation model described above can be invoked based on the prompt image sequence to generate target speech text corresponding to the prompt image sequence of the target object.

[0338] In some embodiments, the target speech text generated based on the speech generation model can be further generated into corresponding target speech by a speech signal generator in the speech generation model.

[0339] In the embodiments of the present application, based on the sample data of multiple modalities of the sample object, the first speech generation model is trained in three parallel ways: 1) encoding based on the sample data of the visual modality and the text modality, and invoking the text decoder to decode the encoding results of the visual encoder and the text encoder respectively to update the parameters of the visual encoder; 2) encoding based on the sample data of the visual modality, the text modality and the speech modality, and invoking the text decoder to decode the encoding results of the visual encoder, the text encoder and the speech encoder respectively to update the parameters of the speech encoder; 3) encoding based on the sample data of the visual modality, the text modality and the speech modality, fusing the encoding results corresponding to the visual modality and the text modality, and then invoking the text decoder to decode the fused encoding results and the encoding results of the text encoder respectively to update the parameters of the visual encoder and the speech encoder, so that the speech generation model fully learns the features of different modalities of the visual modality, the text modality and the speech modality, which can improve the ability of the speech generation model to perform the speech generation task and improve the accuracy of speech generation. At the same time, the combination of different modalities can ensure that the execution of the speech generation task can adapt to different application scenarios and improve user experience.

[0340] In the following, an exemplary application of the embodiments of the present application applied to the scene of providing fluent speech for the target object with hearing impairment or speech impairment will be described.

[0341] The visual expression of the target object from the prompt speech (equivalent to the prompt image sequence described above) is collected by the terminal, and the visual expression of the target object is uploaded to the server. The server identifies the received visual expression of the target object through the trained second speech generation model, generates target speech text corresponding to the visual expression, and sends it to the terminal. Then the terminal outputs the speech corresponding to the target speech text; or the server identifies the visual expression of the target object based on the visual expression of the target object, generates target speech text corresponding to the visual expression, and generates corresponding speech based on the target speech text, and then sends the speech to the terminal for broadcast. The training process of the speech generation model will be described in detail below.

[0342] For people with hearing or speech disorders, lip reading or sign language is the main way of communication, but the way of lip reading is difficult to distinguish the pronunciation with similar lip shapes, such as [u] and [y]; the way of sign language needs to spend a long time to learn to communicate with others; in the related technology, the hand is encoded by the prompt speech system, and the hand shape and hand position are combined with the way of lip reading to provide clear visual representation of all phonemes in spoken language for users with hearing or speech disorders.

[0343] Referring to FIG. 7, FIG. 7 is a schematic diagram of prompt speech coding information provided by the embodiment of the present application, as shown in FIG. 7, five different hand positions are used to encode the vowel group in Mandarin, and eight different hand gestures are used to encode the consonant group in Mandarin. Through prompt speech, people with hearing impairment can distinguish the pronunciation that cannot be distinguished by lip reading by combining hand information. However, the prompt speech system encodes the visual expression provided by the prompt speech user based on the hand movements and lip movements of the prompt speech user, on the one hand, it cannot output smooth and natural universal speech expression, on the other hand, it is not suitable for non-prompt speech users.

[0344] The speech engine technology in the related technology can clone a person's voice with only 15 seconds of speech samples, support cross-language, and users can choose a dedicated real voice tone instead of a synthetic voice with obvious mechanical feeling, and the voice tone can be consistent among various languages, which can provide support for Augmentative & Alternative Communication (AAC) devices, provide personalized voices across multiple languages for users who cannot speak, and thus help people who cannot speak, such as providing therapeutic applications for users with speech-affected diseases, and providing education enhancement services for users with learning needs. However, the core of the speech engine technology in the related technology is a voice synthesis and cloning method, which requires users to input text in the application, and the text can be converted into speech, and this way of communication expression cannot support streaming real-time expression.

[0345] The method provided by the embodiment of the present application can directly generate corresponding fluent speech based on the prompt speech of the target object with hearing or speech disorders by encoding the visual expression from the prompt speech and mapping it to a unified representation space with the mapping space of the text mode and the speech mode in the speech recognition translation synthesis engine (equivalent to the speech recognition model), and thus providing the target object with cross-language, customizable and natural fluent voice, thereby helping the target object to communicate with others without barriers.

[0346] Referring to FIG. 8, FIG. 8 is a schematic diagram of training data provided by an embodiment of the present application. As shown in FIG. 8, the training data includes prompt speech of a visual modality, phonemes and audio of a speech modality, and text of a text modality. In order to construct a hidden representation space of a visual modality shared with a speech modality and a text modality, the training data needs to be preprocessed and encoded first.

[0347] Referring to FIG. 9, FIG. 9 is a schematic diagram of a visual encoding model architecture provided by an embodiment of the present application. As shown in FIG. 9, first, the video of the target object is preprocessed to extract the keypoints of each frame of the video to obtain a sequence of keypoints. The keypoint (CS keypoint) of the visual modality includes the coordinate positions of a total of J keypoints such as lips, fingers, and hands.

[0348] Secondly, the sequence of keypoints of each frame of the video is input into a visual encoder. The keypoint information input into the visual encoder (CS Encoder) is represented as wherein F represents the total number of video frames, and C represents the coordinate dimension of the keypoint. For example, when the keypoint coordinate is 2D dimension, C = 2, and when the keypoint coordinate is 3D dimension, C = 3. Then, the sequence of keypoints is pose-embedded and encoded through a Transformer block to obtain a sequence of visual encoding vectors.

[0349] The State-of-the-Art (SOTA) pose detection base model can be used as the pre-training base model of the visual encoder. Based on the pre-trained visual encoder trained by high-resource human pose data, the parameters of the pre-trained visual encoder model are fine-tuned by using low-resource prompt speech training data (for example, the training data shown in FIG. 8).

[0350] In order to reduce the redundancy of the video while retaining the diversity and integrity of the semantic information, the clustering pruning technology is used in the related art to cluster the input F-frame Token sequence according to the similarity by using the k-nearest neighbor peak density clustering algorithm (Density Peaks Clustering Based on Knearest Neighbors, DPC-KNN). Based on the two criteria of having a higher density relative to the nearest neighbor and having a farther distance relative to other high-density Tokens, the f frame with a higher score is taken as the clustering center, that is, the key Token reserved after pruning, and f is much smaller than the original sequence length F (f << F).

[0351] Unlike the cluster pruning technique of the related art, the embodiment of the present application aligns the visual encoding vector sequence obtained after the visual encoder is encoded to the output feature space of the text encoder when fine-tuning the pre-trained visual encoding model, thereby ensuring the semantic integrity of the former, including the following two aspects of processing, referring to FIG. 4A, which is a first architecture schematic diagram of a speech generation model (CSeamless) provided by the embodiment of the present application.

[0352] First, sequence length alignment processing. The visual encoding vector sequence (sequence length F) obtained by the visual encoder is aligned with the output sequence length f of the text encoder through a length adaptor. The length adaptor can adopt the length adaptor (Length Adaptor) based on the Transformer structure in SEAMLESSM4T V2.

[0353] By aligning the lengths of the feature vector sequences of the visual and text modalities, the length of the visual encoding vector sequence is compressed from F to f (f << F), at this time, the output of the visual encoder is where D represents the dimension of the features input by the text encoder to the text decoder, which improves the processing efficiency of the visual encoding model.

[0354] Then, feature space alignment processing. The output of the visual encoder and the output of the text encoder are processed in the feature space by using the knowledge distillation as shown in the above formula (2) as the objective function, to extract the knowledge in the powerful text-to-text machine translation base model (such as SEAMLESSM4T-NLLB) and guide the cued-speech to text (and text translation), i.e., the task from the visual modality to the text modality, thereby ensuring that the semantic integrity is maintained while the output sequence of the visual encoder is compressed.

[0355] where xcued-speech and x text are the source language cued-speech and the corresponding source text respectively, y text is the corresponding target language text. In the embodiment of the present application, the source language is Chinese, and the target language includes Chinese and English. When the target language is English, the source text x text is text translated from Chinese to English to obtain the pseudo label y text When the target language is the same as the source language, it is equivalent to the cued-speech to text recognition task (CS2T).

[0356] By fixing the pre-trained text encoder, the parameters θ of the visual encoder (CS Encoder) are fine-tuned csE and the parameters θ of the text decoder (Text Decoder) shown in FIG. 4A tD The loss function of the corresponding recognition task and the loss function of the translation task are taken as additional fine-tuning training objective functions, the loss function of the recognition task and the loss function of the translation task are respectively shown in the above formula (3) and formula (4).

[0357] In summary, the loss function of the overall fine-tuning training of the first architecture of the speech generation model provided by the embodiments of the present application is shown in the above formula (5), where α is a weight parameter determined by the proportion of the fine-tuning training set in the full training set.

[0358] As an example, the text encoder and text decoder shown in FIG. 4A can adopt the text encoding & decoding (Text Encoder & Decoder) model and pre-training weight of SEAMLESSM4T-NLLB based on Transformer, and fix the weight of the text encoder, and fine-tune the parameters of the text decoder as described above; the text-to-unit (Text2Unit) module can adopt the non-autoregressive (NAR) T2U model and weight in SEAMLESSM4T V2; and the unit-to-speech (Unit Vocoder) module can adopt the unit vocoder model based on High-Fidelity Generative Adversarial Network (HiFi-GAN) in SEAMLESSM4T V2.

[0359] In the actual application process of the prompting speech system, the prompting sound of the hearing-impaired person is accompanied by impaired voice or unvoiced sound while expressing hand movements and lip movements. The actual impaired voice varies depending on the situation of each person, and some have a slight impairment and can produce certain distinguishable sounds, while some have a serious impairment and the sound produced is difficult to distinguish. Such impaired speech has an auxiliary effect on improving the accuracy and intelligibility of communication expression in practice, and therefore the impaired speech coding model can also be combined for training in the training process of the speech generation model.

[0360] Referring to FIG. 4B, FIG. 4B is a second architecture schematic diagram of the speech generation model provided in the embodiments of the present application. As shown in FIG. 4B, compared with the training architecture of the speech generation model shown in FIG. 4A, an impaired-speech encoder (IS Encoder) is added, which can take the speech encoder (Speech Encoder) in SEAMLESSM4T V2 as an initialized impaired-speech encoder, and align the speech encoding vector sequence of the speech modality with the text vector sequence of the text modality through a length adapter of the speech modality. Then, the impaired-speech encoder is fine-tuned by using a low-resource impaired-speech (referring to the speech of a speaker who cannot speak normally clear and fluent due to physiology or disease) dataset.

[0361] As shown in FIG. 4B, 80-dimensional spectral features are extracted from the impaired speech by using a mel filter bank extractor where F' represents the total number of speech frames as the input of the impaired-speech encoder. The output of the impaired-speech encoder and the output of the text encoder are processed for feature space alignment by using knowledge distillation as shown in formula (6) above as the objective function.

[0362] where ximpaired-speech and x text are impaired-speech and corresponding source text of the source language respectively, y text is the corresponding text of the target language. When the target language is English, the source text x text is translated from Chinese to English to obtain the pseudo-label y text When the target language is the same as the source language (Chinese), it is equivalent to the recognition task from impaired-speech to text (IS2T).

[0363] By fixing the pre-trained text encoder (Text Encoder), the parameters θ isE of the impaired-speech encoder (IS Encoder) and the parameters θ tD of the text decoder (Text Decoder) shown in FIG. 4B are jointly fine-tuned, and the loss function of recognition and the loss function of translation are taken as additional fine-tuning training objective functions, the loss function of recognition and the loss function of translation are respectively shown in formula (7) and formula (8) above.

[0364] In summary, the loss function of the overall fine-tuning training of the second architecture of the speech generation model provided in the embodiments of the present application is shown in formula (9) above.

[0365] In practical applications, there is a strong correlation between visual modal prompts and impaired speech in the speech modality. Hearing-impaired individuals can improve the accuracy and intelligibility of their communication by jointly expressing these prompts and speech. Therefore, the visual encoder and speech encoder can be jointly trained during the training of the speech generation model.

[0366] Referring to Figure 4C, Figure 4C is a schematic diagram of the third architecture of the speech generation model provided in the embodiments of this application. As shown in Figure 4C, the damaged speech can be processed by the speech encoder to output... and the output of the visual encoder The concatenation projector module first performs concatenation processing to obtain... Then perform mapping processing to obtain Then it is aligned with the feature space of the text encoder output. The objective function of knowledge distillation corresponding to the alignment of feature vector sequences is shown in formula (10) above.

[0367] In formula (10), xcued-speech, ximpaired-speech, and x text These are the cued-speech and impaired-speech in the source language and their corresponding text, respectively, while ytext corresponds to the text in the target language. When the target language is English, the source text x is processed... text Perform a text translation from Chinese to English to obtain pseudo-labels y. text When the target language is the same as the source language (Chinese), it is equivalent to a text recognition task that combines cued-speech and impaired-speech (CS_IS2T).

[0368] By fixing the pre-trained text encoder, the parameters θ of the visual encoder (CS encoder) are adjusted. csE The parameters θ of the speech encoder (IS Encoder) isE The parameters θ of the text decoder shown in Figure 4C tD Perform joint fine-tuning and increase the recognition loss function. and translation loss function As an additional fine-tuning training objective function, the loss function for identification. and translation loss function The above formulas (11) and (12) are shown respectively.

[0369] In summary, the third architecture of the speech generation model provided in the embodiments of the present application has a loss function for overall fine-tuning training as shown in formula (13) above.

[0370] Through the above training method, the corresponding fluent speech can be directly generated based on the prompt speech of the person who cannot speak or has impaired speech, supporting real-time speech output in a streaming manner, supporting the generation of natural and fluent speech across languages, supporting user self-selection of voice timbre, and assisting more hearing-impaired and voice-impaired persons to better and faster integrate into the digital society.

[0371] The following continues to illustrate an exemplary structure of the implementation of the training device 555-1 of the speech generation model provided in the embodiments of the present application as a software module. In some embodiments, as shown in FIG. 2A, the software module stored in the training device 555-1 of the speech generation model in the memory 550-1 can include:

[0372] The acquisition module 5551-1 is configured to acquire a first speech generation model, wherein the first speech generation model includes a decoder and a plurality of encoders corresponding to a plurality of modalities respectively; and acquire sample data of the plurality of modalities, wherein the sample data of the plurality of modalities includes a prompt image sequence of a sample object and speech text corresponding to the prompt image sequence.

[0373] The encoding module 5552-1 is configured to call the plurality of encoders for encoding based on the prompt image sequence and the speech text, to obtain a multimodal encoding vector sequence.

[0374] The decoding module 5553-1 is configured to call the decoder for decoding based on the multimodal encoding vector sequence, to obtain a decoded text.

[0375] The determination module 5554-1 is configured to determine a probability distribution of the decoded text and the sample data of the plurality of modalities, and determine a target loss based on the probability distribution.

[0376] The generation module 5555-1 is configured to update parameters of the decoder and at least one encoder based on the target loss, to obtain an updated decoder and a plurality of encoders, wherein the updated decoder and the plurality of encoders are used to compose a second speech generation model, and the second speech generation model is used to generate target speech text corresponding to a prompt image sequence of a target object.

[0377] In some embodiments, when the plurality of modalities includes a visual modality and a text modality, the encoding module 5552-1 is further configured to invoke a visual encoder to encode the sequence of prompt image to obtain a sequence of visual encoding vectors of the visual modality, where the visual encoder is an encoder corresponding to the visual modality in the plurality of encoders; invoke a text encoder to encode the plurality of morphemes in the speech text respectively to obtain a sequence of text encoding vectors of the text modality, where the sequence of text encoding vectors includes a text encoding vector corresponding to each of the plurality of morphemes, and the text encoder is an encoder corresponding to the text modality in the plurality of encoders; and take the sequence of visual encoding vectors and the sequence of text encoding vectors as a first sequence of multi-modal encoding vectors, where the first sequence of multi-modal encoding vectors is for decoding by the decoder.

[0378] In some embodiments, the encoding module 5552-1 is further configured to identify each of the plurality of prompt images in the sequence of prompt images to obtain a position of a key point in each of the prompt images; generate a sequence of key points based on the position of the key point in each of the prompt images; obtain an embedding vector of the sequence of key points; and invoke the visual encoder to encode the embedding vector to obtain the sequence of visual encoding vectors of the visual modality.

[0379] In some embodiments, the determining module 5554-1 is further configured to determine a probability distribution of the sequence of encoding vectors of the sample data of at least two modalities with respect to a preset condition, where different modalities correspond to different preset conditions; determine a probability distribution of the speech text with respect to the sample data of at least one modality; and determine a probability distribution of the decoding text with respect to the sample data of at least one modality.

[0380] In some embodiments, the determining module 5554-1 is further configured to determine a first probability distribution of the sequence of text encoding vectors with respect to the decoding text and the speech text, and a second probability distribution of the sequence of visual encoding vectors with respect to the decoding text and the sequence of prompt images; determine a third probability distribution of the speech text with respect to the sequence of prompt images; in a case where the target language and the source language of the speech generation task of the first speech generation model are different, determine a fourth probability distribution of the decoding text with respect to the sequence of prompt images; determine a difference between the first probability distribution and the second probability distribution as a first sub-target loss; determine a second sub-target loss negatively related to the third probability distribution; determine a third sub-target loss negatively related to the fourth probability distribution; and fuse the first sub-target loss, the second sub-target loss, and the third sub-target loss to obtain the first target loss.

[0381] In some embodiments, the determining module 5554-1 is further configured to call a pre-trained first probability distribution model to determine the first probability distribution of the sequence of text encoding vectors with respect to the decoded text and the speech text, wherein the first probability distribution model is trained by: obtaining first sample data, wherein the first sample data includes sample decoded text and sample speech text, and a reference probability distribution of a sequence of sample text encoding vectors with respect to the sample decoded text and the sample speech text; taking the sample decoded text, the sample speech text, and the sequence of sample text encoding vectors as input of an initialized first probability distribution model, and outputting a predicted probability distribution of the sequence of text encoding vectors with respect to the sample decoded text and the sample speech text; determining a loss value based on a difference between the predicted probability distribution and the reference probability distribution; and updating parameters of the first probability distribution model according to the loss value.

[0382] In some embodiments, the generating module 5555-1 is further configured to update parameters of the visual encoder based on the first sub-target loss; and update parameters of the visual encoder and the decoder based on the second sub-target loss and the third sub-target loss.

[0383] In some embodiments, the first speech generation model is a pre-trained speech recognition model, and after being pre-trained, the representation space of the text modality and the representation space of the speech modality of the first speech generation model have been unified into the same representation space; the decoding module 5553-1 is further configured to map the sequence of visual encoding vectors and the sequence of text encoding vectors to obtain vector sequence representations of the sequence of visual encoding vectors and the sequence of text encoding vectors in the representation space; and adjust the length of the vector sequence representation of the sequence of visual encoding vectors to be the same as the length of the vector sequence representation of the sequence of text encoding vectors.

[0384] In some embodiments, when the multiple modalities further include a speech modality, the sample data of the multiple modalities further includes speech signal features of the speech modality; the encoding module 5552-1 is further configured to call a speech encoder to encode the speech signal features to obtain a sequence of speech encoding vectors of the speech modality, wherein the speech encoder is an encoder corresponding to the speech modality among the multiple encoders; and take the sequence of visual encoding vectors, the sequence of text encoding vectors, and the sequence of speech encoding vectors as a second sequence of multi-modal encoding vectors, wherein the second sequence of multi-modal encoding vectors is used to replace the first sequence of multi-modal encoding vectors for decoding by the decoder.

[0385] In some embodiments, the determining module 5554-1 is further configured to determine a fifth probability distribution of the text encoding vector sequence with respect to the decoded text and the speech text, and a sixth probability distribution of the speech encoding vector sequence with respect to the decoded text and the speech signal feature; determine a seventh probability distribution of the speech text with respect to the speech signal feature; in a case where the target language of the speech generation task of the first speech generation model is different from the source language, determine an eighth probability distribution of the decoded text with respect to the speech signal feature; determine a difference between the fifth probability distribution and the sixth probability distribution as a fourth sub-target loss; determine a fifth sub-target loss negatively related to the seventh probability distribution; determine a sixth sub-target loss negatively related to the eighth probability distribution; and fuse the fourth sub-target loss, the fifth sub-target loss, and the sixth sub-target loss to obtain the second target loss.

[0386] In some embodiments, the generating module 5555-1 is further configured to update parameters of the speech encoder based on the fourth sub-target loss; and update parameters of the speech encoder and the decoder based on the fifth sub-target loss and the sixth sub-target loss.

[0387] In some embodiments, when the multiple modalities further include a speech modality, the encoding module 5552-1 is further configured to call multiple encoders based on the prompt image sequence and the speech text respectively to obtain a visual encoding vector sequence of a visual modality, a text encoding vector sequence of a text modality, and a speech encoding vector sequence of a speech modality; perform feature splicing on the visual encoding vector sequence and the speech encoding vector sequence to obtain a spliced encoding vector sequence; perform feature mapping on the spliced encoding vector sequence to obtain a fused encoding vector sequence; and take the fused encoding vector sequence and the text encoding vector sequence as a third multi-modal encoding vector sequence, wherein the third multi-modal encoding vector sequence is used to replace the second multi-modal encoding vector sequence for decoding by the decoder.

[0388] In some embodiments, the sample data of the multiple modalities further includes speech signal features of a speech modality, and the encoding module 5552-1 is further configured to call a visual encoder to encode the prompt image sequence to obtain a visual encoding vector sequence of a visual modality, wherein the visual encoder is an encoder corresponding to the visual modality in the multiple encoders; call a text encoder to encode multiple morphemes in the speech text respectively to obtain a text encoding vector sequence of a text modality, wherein the text encoding vector sequence includes text encoding vectors corresponding to the multiple morphemes respectively, and the text encoder is an encoder corresponding to the text modality in the multiple encoders; and call a speech encoder to encode the speech signal features to obtain a speech encoding vector sequence of a speech modality, wherein the speech encoder is an encoder corresponding to the speech modality in the multiple encoders.

[0389] In some embodiments, the determining module 5554-1 is further configured to determine a ninth probability distribution of the text encoding vector sequence with respect to the decoded text and the speech text, and fuse a tenth probability distribution of the encoding vector sequence with respect to the decoded text, the prompt image sequence, and the speech signal feature; determine an eleventh probability distribution of the speech text with respect to the prompt image sequence and the speech signal feature; in a case where a target language of the speech generation task of the first speech generation model is different from a source language, determine a twelfth probability distribution of the decoded text with respect to the prompt image sequence and the speech signal feature; determine a difference between the ninth probability distribution and the tenth probability distribution as a seventh sub-target loss; determine an eighth sub-target loss negatively related to the eleventh probability distribution; determine a ninth sub-target loss negatively related to the twelfth probability distribution; and fuse the seventh sub-target loss, the eighth sub-target loss, and the ninth sub-target loss to obtain the third target loss.

[0390] In some embodiments, the generating module 5555-1 is further configured to update parameters of the visual encoder and the speech encoder based on the seventh sub-target loss; and update parameters of the visual encoder, the speech encoder, and the decoder based on the eighth sub-target loss and the ninth sub-target loss.

[0391] In some embodiments, the first speech generation model further includes a speech signal generator, wherein the speech signal generator is used to constitute a second speech generation model with the decoder and the updated multiple encoders, and the second speech generation model is used to generate the speech signal.

[0392] The following continues to explain an exemplary structure of the speech generation apparatus 555-2 of the speech generation model provided by the embodiments of the present application as a software module. In some embodiments, as shown in FIG. 2B, the software module stored in the training apparatus 555-2 of the speech generation model of the memory 550-2 can include:

[0393] The obtaining module 5551-2 is configured to obtain a prompt image sequence of a target object.

[0394] The generating module 5552-2 is configured to call the second speech generation model to generate target speech text corresponding to the prompt image sequence of the target object based on the prompt image sequence.

[0395] The embodiments of the present application provide a computer program product, which includes a computer program or computer executable instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the training method of the speech generation model provided by the embodiments of the present application, or executes the speech generation method of the speech generation model provided by the embodiments of the present application.

[0396] The embodiments of the present application provide a computer readable storage medium, which stores computer executable instructions or computer programs. When the computer executable instructions or computer programs are executed by a processor, the processor executes a training method of a speech generation model provided by the embodiments of the present application, such as the training method of the speech generation model shown in FIG. 3A, or executes a speech generation method of the speech generation model provided by the embodiments of the present application, such as the speech generation method of the speech generation model shown in FIG. 3Q.

[0397] In some embodiments, the computer readable storage medium can be a RAM, a ROM, a flash memory, a magnetic surface memory, an optical disc, or a CD-ROM, etc. memory; or can be various devices including one or any combination of the above memories.

[0398] In some embodiments, the computer executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or being deployed as modules, components, subroutines or other units suitable for use in a computing environment.

[0399] As an example, the computer executable instructions can but not necessarily correspond to files in a file system, can be stored in a part of a file storing other programs or data, for example, stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program in question, or stored in multiple cooperative files (for example, files storing one or more modules, subroutines or code parts).

[0400] As an example, the computer executable instructions can be deployed to be executed on one electronic device, or on multiple electronic devices located in one place, or on multiple electronic devices distributed in multiple places and interconnected through a communication network.

[0401] In summary, the embodiment of the present application is based on at least the sample data of multiple modalities including the prompt image sequence of the target object (corresponding to the visual modality) and the speech text (corresponding to the text modality), respectively calls multiple encoders for encoding, calculates the target loss by the obtained multi-modal encoding vector sequence (that is, the encoding vector sequence of multiple modalities) and the probability distribution of the decoded text, so that the target loss can reflect the difference between the representation spaces of multiple modalities in the first speech generation model, and the parameters of the first speech generation model are updated by back propagation based on the target loss, so that the first speech generation model can gradually learn the internal relationship between the information of the visual modality and the text modality, improve the performance and generalization ability of the model, and realize the unification of the representation spaces of the visual modality and the text modality in the second speech generation model obtained after training, which means that the second speech generation model can map data of different modalities to a common representation space, so that the data of different modalities have similar semantic expressions in the space, laying a foundation for subsequent accurate generation of speech text, and ensuring that the target speech text output by the second speech generation model can accurately match the intention of the prompt image of the target object in expression, thereby ensuring the accuracy of generating speech text from prompt images. At the same time, compared with the way of generating speech text by manually inputting text in related technologies, it is more convenient, efficient and better user experience. The training process of the first speech generation model is carried out on the basis of the pre-trained speech recognition model, and on the basis that the representation space of the text modality and the representation space of the speech modality have been unified to the same representation space, the conversion function from the prompt image sequence to the speech text can be realized on the basis of the original function of the pre-trained speech recognition model by fine-tuning, and the multiple functions of the pre-trained speech recognition model itself can be reused, such as recognizing the prompt image sequence of the target object, text translation and generating speech, avoiding repeated development and improving development efficiency.

[0402] The above is only an embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement and improvement made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for training a speech generation model, the method being performed by an electronic device, the method comprising: obtaining a first speech generation model, wherein the first speech generation model comprises a decoder and a plurality of encoders respectively corresponding to a plurality of modalities; obtaining sample data of the plurality of modalities, wherein the sample data of the plurality of modalities comprises a sequence of prompt image of a sample object and a speech text corresponding to the sequence of prompt image; encoding, based on the sequence of prompt image and the speech text, the plurality of encoders respectively to obtain a sequence of multi-modality encoding vectors; decoding, based on the sequence of multi-modality encoding vectors, the decoder to obtain a decoded text; determining a probability distribution of the decoded text and the sample data of the plurality of modalities, and determining a target loss based on the probability distribution; updating parameters of the decoder and at least one of the encoders based on the target loss to obtain an updated decoder and the plurality of encoders, wherein the updated decoder and the plurality of encoders are used to compose a second speech generation model, and the second speech generation model is used to generate a target speech text corresponding to a sequence of prompt image of a target object.

2. The method of claim 1, wherein, When the plurality of modalities comprises a visual modality and a text modality, the encoding, based on the sequence of prompt image and the speech text, the plurality of encoders respectively to obtain a sequence of multi-modality encoding vectors comprises: encoding, by a visual encoder, the sequence of prompt image to obtain a sequence of visual encoding vectors of the visual modality, wherein the visual encoder is an encoder corresponding to the visual modality in the plurality of encoders; encoding, by a text encoder, a plurality of morphemes in the speech text respectively to obtain a sequence of text encoding vectors of the text modality, wherein the sequence of text encoding vectors comprises a text encoding vector corresponding to each of the plurality of morphemes, and the text encoder is an encoder corresponding to the text modality in the plurality of encoders; taking the sequence of visual encoding vectors and the sequence of text encoding vectors as a first sequence of multi-modality encoding vectors, wherein the first sequence of multi-modality encoding vectors is used for decoding by the decoder.

3. The method of claim 2, wherein, The encoding, by a visual encoder, the sequence of prompt image to obtain a sequence of visual encoding vectors of the visual modality comprises: identifying a plurality of prompt images in the sequence of prompt image respectively to obtain a position of a key point in each of the prompt images; generating a sequence of key points based on the position of the key point in each of the prompt images; obtaining an embedding vector of the sequence of key points; encoding, by the visual encoder, the embedding vector to obtain the sequence of visual encoding vectors of the visual modality.

4. The method of claim 2 or 3, wherein, The determining a probability distribution of the decoded text and the sample data of the plurality of modalities comprises: determining a probability distribution of an encoding vector sequence of sample data of at least two modalities with respect to a preset condition, wherein different modalities correspond to different preset conditions; determining a probability distribution of the speech text with respect to sample data of at least one of the modalities. determine a probability distribution of the decoded text relative to sample data of at least one of the modalities.

5. The method of any one of claims 2-4, wherein, the determining the probability distribution of the decoded text relative to the sample data of the plurality of modalities comprises: determining a first probability distribution of the sequence of text encoding vectors relative to the decoded text and the speech text, and a second probability distribution of the sequence of visual encoding vectors relative to the decoded text and the sequence of prompt images; determining a third probability distribution of the speech text relative to the sequence of prompt images; in a case where a target language and a source language of a speech generation task of the first speech generation model are different, determining a fourth probability distribution of the decoded text relative to the sequence of prompt images; the determining the target loss based on the probability distributions comprises: determining a difference between the first probability distribution and the second probability distribution as a first sub-target loss; determining a second sub-target loss negatively related to the third probability distribution; determining a third sub-target loss negatively related to the fourth probability distribution; fusing the first sub-target loss, the second sub-target loss, and the third sub-target loss to obtain a first target loss.

6. The method of claim 5, wherein, the determining the first probability distribution of the sequence of text encoding vectors relative to the decoded text and the speech text comprises: calling a pre-trained first probability distribution model to determine the first probability distribution of the sequence of text encoding vectors relative to the decoded text and the speech text, wherein the first probability distribution model is trained in the following manner: obtaining first sample data, wherein the first sample data includes sample decoded text and sample speech text, and a reference probability distribution of a sequence of sample text encoding vectors relative to the sample decoded text and the sample speech text; inputting the sample decoded text, the sample speech text, and the sequence of sample text encoding vectors as an initialized first probability distribution model, and outputting a predicted probability distribution of the sequence of text encoding vectors relative to the sample decoded text and the sample speech text; based on a difference between the predicted probability distribution and the reference probability distribution, determining a loss value; updating parameters of the first probability distribution model according to the loss value.

7. The method of claim 5, wherein, the updating parameters of the decoder and at least one of the encoders based on the target loss comprises: updating parameters of the visual encoder based on the first sub-target loss; updating parameters of the visual encoder and the decoder based on the second sub-target loss and the third sub-target loss.

8. The method of any one of claims 2-4, wherein, the first speech generation model is a pre-trained speech recognition model, and after the pre-training, a representation space of the text modality and a representation space of the speech modality of the first speech generation model have been unified into a same representation space; before the calling the decoder based on the sequence of multi-modal encoding vectors to perform decoding to obtain decoded text, the method further comprises: mapping the visual encoding vector sequence and the text encoding vector sequence to obtain vector sequence representations of the visual encoding vector sequence and the text encoding vector sequence in the representation space; adjusting a length of the vector sequence representation of the visual encoding vector sequence to be the same as a length of the vector sequence representation of the text encoding vector sequence.

9. The method according to any one of claims 2 to 4, wherein, When the multiple modalities further include a speech modality, the sample data of the multiple modalities further includes speech signal features of the speech modality; The calling the multiple encoders to encode, based on the prompt image sequence and the speech text respectively, to obtain the multi-modal encoding vector sequence further includes: calling a speech encoder to encode the speech signal features to obtain a speech encoding vector sequence of the speech modality, wherein the speech encoder is an encoder corresponding to the speech modality in the multiple encoders; taking the visual encoding vector sequence, the text encoding vector sequence, and the speech encoding vector sequence as a second multi-modal encoding vector sequence, wherein the second multi-modal encoding vector sequence is used to replace the first multi-modal encoding vector sequence for decoding by the decoder.

10. The method of claim 9, wherein, The determining the probability distribution of the decoded text and the sample data of the multiple modalities includes: determining a fifth probability distribution of the text encoding vector sequence with respect to the decoded text and the speech text, and a sixth probability distribution of the speech encoding vector sequence with respect to the decoded text and the speech signal features; determining a seventh probability distribution of the speech text with respect to the speech signal features; in a case where a target language and a source language of a speech generation task of the first speech generation model are different, determining an eighth probability distribution of the decoded text with respect to the speech signal features; The determining the target loss based on the probability distribution includes: determining a difference between the fifth probability distribution and the sixth probability distribution as a fourth sub-target loss; determining a fifth sub-target loss negatively related to the seventh probability distribution; determining a sixth sub-target loss negatively related to the eighth probability distribution; fusing the fourth sub-target loss, the fifth sub-target loss, and the sixth sub-target loss to obtain a second target loss.

11. The method of claim 10, wherein, The updating the parameters of the decoder and at least one of the encoders based on the target loss includes: updating the parameters of the speech encoder based on the fourth sub-target loss; updating the parameters of the speech encoder and the decoder based on the fifth sub-target loss and the sixth sub-target loss.

12. The method according to any one of claims 1 to 11, wherein, When the multiple modalities further include a speech modality, the calling the multiple encoders to encode, based on the prompt image sequence and the speech text respectively, to obtain the multi-modal encoding vector sequence includes: calling the multiple encoders to encode, based on the prompt image sequence and the speech text respectively, to obtain a visual encoding vector sequence of a visual modality, a text encoding vector sequence of a text modality, and a speech encoding vector sequence of the speech modality; performing feature splicing on the visual encoding vector sequence and the speech encoding vector sequence to obtain a spliced encoding vector sequence; map the spliced coding vector sequence to obtain a fused coding vector sequence; the fused coding vector sequence and the text coding vector sequence are used as a third multi-modal coding vector sequence, wherein the third multi-modal coding vector sequence is used to replace the second multi-modal coding vector sequence for decoding by the decoder.

13. The method of claim 12, wherein, The sample data of the plurality of modalities further includes speech signal features of a speech modality; The calling of the plurality of encoders based on the prompt image sequence and the speech text respectively to obtain the visual coding vector sequence of the visual modality, the text coding vector sequence of the text modality, and the speech coding vector sequence of the speech modality includes: calling a visual encoder to encode the prompt image sequence to obtain the visual coding vector sequence of the visual modality, wherein the visual encoder is an encoder corresponding to the visual modality in the plurality of encoders; calling a text encoder to encode a plurality of morphemes in the speech text respectively to obtain the text coding vector sequence of the text modality, wherein the text coding vector sequence includes text coding vectors corresponding to the plurality of morphemes respectively, and the text encoder is an encoder corresponding to the text modality in the plurality of encoders; calling a speech encoder to encode the speech signal features to obtain the speech coding vector sequence of the speech modality, wherein the speech encoder is an encoder corresponding to the speech modality in the plurality of encoders.

14. The method of claim 12, wherein, The determining of the probability distribution of the decoded text and the sample data of the plurality of modalities includes: determining a ninth probability distribution of the text coding vector sequence with respect to the decoded text and the speech text, and a tenth probability distribution of the fused coding vector sequence with respect to the decoded text, the prompt image sequence, and the speech signal features; determining an eleventh probability distribution of the speech text with respect to the prompt image sequence and the speech signal features; in a case where a target language and a source language of a speech generation task of the first speech generation model are different, determining a twelfth probability distribution of the decoded text with respect to the prompt image sequence and the speech signal features; The determining of the target loss based on the probability distributions includes: determining a difference between the ninth probability distribution and the tenth probability distribution as a seventh sub-target loss; determining an eighth sub-target loss negatively related to the eleventh probability distribution; determining a ninth sub-target loss negatively related to the twelfth probability distribution; fusing the seventh sub-target loss, the eighth sub-target loss, and the ninth sub-target loss to obtain a third target loss.

15. The method of claim 14, wherein, The updating of the parameters of the decoder and at least one of the encoders based on the target loss includes: updating the parameters of the visual encoder and the speech encoder based on the seventh sub-target loss; updating the parameters of the visual encoder, the speech encoder, and the decoder based on the eighth sub-target loss and the ninth sub-target loss.

16. The method of any one of claims 1 to 14, wherein, The first speech generation model further comprises a speech signal generator, wherein the speech signal generator is configured to form a second speech generation model with the decoder and the plurality of updated encoders, and the second speech generation model is configured to generate a speech signal. 17.A speech generation method of a speech generation model, the method being performed by an electronic device, the speech generation model being the second speech generation model of any one of claims 1 to 16, and the method comprising: obtaining a sequence of prompt image of a target object; calling the second speech generation model to generate a target speech text corresponding to the sequence of prompt image of the target object based on the sequence of prompt image. 18.A training apparatus of a speech generation model, the apparatus comprising: an obtaining module configured to obtain a first speech generation model, wherein the first speech generation model comprises a decoder and a plurality of encoders corresponding to a plurality of modalities respectively; and obtain sample data of the plurality of modalities, wherein the sample data of the plurality of modalities comprises a sequence of prompt image of a sample object and a speech text corresponding to the sequence of prompt image; an encoding module configured to call the plurality of encoders to encode based on the sequence of prompt image and the speech text respectively, and obtain a sequence of multi-modal encoding vectors; a decoding module configured to call the decoder to decode based on the sequence of multi-modal encoding vectors, and obtain a decoded text; a determining module configured to determine a probability distribution of the decoded text and the sample data of the plurality of modalities, and determine a target loss based on the probability distribution; a generating module configured to update parameters of the decoder and at least one of the encoders based on the target loss, and obtain an updated decoder and the plurality of encoders, wherein the updated decoder and the plurality of encoders are configured to form a second speech generation model, and the second speech generation model is configured to generate a target speech text corresponding to a sequence of prompt image of a target object. 19.A speech generation apparatus, the apparatus comprising: an obtaining module configured to obtain a sequence of prompt image of a target object; a generating module configured to call a speech generation model to generate a target speech text corresponding to the sequence of prompt image of the target object based on the sequence of prompt image, wherein the speech generation model is the second speech generation model of any one of claims 1 to 16. 20.An electronic device, the electronic device comprising: a memory configured to store computer executable instructions; a processor configured to execute the computer executable instructions stored in the memory to implement the training method of the speech generation model of any one of claims 1 to 16, or the speech generation method of the speech generation model of claim 17. 21.A computer readable storage medium storing computer executable instructions or computer programs, wherein the computer executable instructions or computer programs are executed by a processor to implement the training method of the speech generation model of any one of claims 1 to 16, or the speech generation method of the speech generation model of claim 17.

22. A computer program product comprising computer-executable instructions or a computer program which, when executed by a processor, implement the training method of the speech generation model of any one of claims 1 to 16, or the speech generation method of the speech generation model of claim 17.

Images