Method and device for generating picture through voice, processor and storage medium

Abstract

The invention provides a method and device for generating a picture through voice, a processor and a storage medium, and belongs to the field of artificial intelligence and computer vision, and the method comprises the steps: responding to the received input voice of a user, and extracting text information and voiceprint feature information from the input voice; based on the voiceprint feature information, determining a target text graph model, and predicting an image style preference of a user; fusing the text information with the image style preference to generate a text instruction; and inputting the text instruction into the target text graph model to generate a target image matched with the user intention. By means of the method, personalized model selection and style fusion can be achieved, image creation is more humanized, and the user experience degree is improved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and computer vision technology, specifically to a method for generating images from speech, a device for generating images from speech, a processor, a machine-readable storage medium, and a computer program product. Background Technology

[0002] With the rapid development of artificial intelligence technology, image generation technology, especially text-to-image generation technology, has made groundbreaking progress. Currently, mainstream text-to-image applications allow users to generate corresponding images by inputting text descriptions, greatly lowering the barrier to content creation. However, these applications generally rely on keyboard or screen for text input, resulting in relatively simple, unnatural, and inefficient interaction methods.

[0003] Meanwhile, as the most natural form of human interaction, voice technology, such as speech recognition, has matured significantly. Introducing voice interaction into image generation is considered an important direction for improving user experience and expanding the application scenarios of the technology. Theoretically, users can drive image generation through more intuitive and convenient voice descriptions, achieving true "what you think is what you get."

[0004] In existing technologies, voice-to-image generation technology converts input speech signals into text information using automatic speech recognition technology, and then uses a text-to-image generation model to create corresponding images based on the text information. However, this approach ignores personalized information in voiceprints, resulting in uniform generated results. Furthermore, the fixed text-to-image generation model cannot dynamically adapt to user input or commands, leading to discrepancies between the generated results and the user's potential expectations. Therefore, the level of intelligence in this technical solution needs improvement. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, processor, storage medium, and machine-readable storage medium for generating images from speech.

[0006] To achieve the above objectives, a first aspect of this application provides a method for generating images from speech, the method comprising: in response to receiving input speech from a user, extracting text information and voiceprint feature information from the input speech; determining a target text-to-image model based on the voiceprint feature information, and predicting the user's image style preference; fusing the text information and the image style preference to generate a text instruction; and inputting the text instruction into the target text-to-image model to generate a target image that matches the user's intent.

[0007] In this embodiment of the application, the step of determining the target text-to-image model based on the voiceprint feature information and predicting the user's image style preference includes: constructing a user profile based on the voiceprint feature information; wherein, the user profile includes, but is not limited to, age, gender, emotion, and occupational orientation; determining the target text-to-image model from a preset text-to-image model library according to the user profile, and predicting the user's image style preference according to the user profile.

[0008] In this embodiment of the application, the method further includes: determining an emotion tag corresponding to the user's current emotional state based on the input speech; and fusing the emotion tag, the image style preference, and the text information to generate a text instruction.

[0009] In this embodiment of the application, determining the emotion label corresponding to the user's current emotional state based on the input speech includes: determining prosodic feature parameters based on the input speech; wherein, the prosodic feature parameters include, but are not limited to, speech rate value, pitch variation value, and volume value; inputting the prosodic feature parameters into a pre-constructed emotion classification model to obtain the emotion category corresponding to the prosodic feature parameters, which is used as the emotion label.

[0010] In this embodiment, fusing the emotion tag, the image style preference, and the text information to generate a text instruction includes: determining the emotion intensity value corresponding to the emotion tag; dynamically assigning fusion weights to the emotion tag, the image style preference, and the text information based on the emotion intensity value; wherein, when the emotion intensity value exceeds a preset threshold, the fusion weight of the emotion tag is higher than the fusion weight of the image style preference; and weighting the emotion tag, the image style preference, and the text information based on the assigned fusion weights to generate the text instruction.

[0011] In this embodiment, determining the emotional intensity value corresponding to the emotional label includes: extracting the acoustic features of the input speech and determining a first emotional intensity value using a first emotional intensity prediction model; wherein the first emotional intensity prediction model is configured to be trained based on a first dataset containing speech samples and their corresponding emotional intensity annotations; inputting the text information into a second emotional intensity prediction model to obtain a second emotional intensity value; wherein the second emotional intensity prediction model is trained based on a second dataset containing text samples and their corresponding emotional intensity annotations; and performing a weighted fusion of the first emotional intensity value and the second emotional intensity value to generate an emotional intensity value corresponding to the emotional label.

[0012] A second aspect of this application provides a speech-to-image device, comprising: an extraction module for extracting text information and voiceprint feature information from a user's input speech in response to receiving such input; a voiceprint feature information processing module for determining a target text-to-image model based on the voiceprint feature information and predicting the user's image style preference; a fusion module for fusing the text information and the image style preference to generate a text instruction; and an image generation module for inputting the text instruction into the target text-to-image model to generate a target image that matches the user's intent.

[0013] In this embodiment of the application, the step of determining the target text-to-image model based on the voiceprint feature information and predicting the user's image style preference includes: constructing a user profile based on the voiceprint feature information; wherein, the user profile includes, but is not limited to, age, gender, emotion, and occupational orientation; determining the target text-to-image model from a preset text-to-image model library according to the user profile, and predicting the user's image style preference according to the user profile.

[0014] In this embodiment of the application, the device further includes an emotion determination module, which is used to determine an emotion tag corresponding to the user's current emotional state based on the input voice; the fusion module is also used to fuse the emotion tag, the image style preference and the text information to generate a text instruction.

[0015] In this embodiment of the application, determining the emotion label corresponding to the user's current emotional state based on the input speech includes: determining prosodic feature parameters based on the input speech; wherein, the prosodic feature parameters include, but are not limited to, speech rate value, pitch variation value, and volume value; inputting the prosodic feature parameters into a pre-constructed emotion classification model to obtain the emotion category corresponding to the prosodic feature parameters, which is used as the emotion label.

[0016] In this embodiment, fusing the emotion tag, the image style preference, and the text information to generate a text instruction includes: determining the emotion intensity value corresponding to the emotion tag; dynamically assigning fusion weights to the emotion tag, the image style preference, and the text information based on the emotion intensity value; wherein, when the emotion intensity value exceeds a preset threshold, the fusion weight of the emotion tag is higher than the fusion weight of the image style preference; and weighting the emotion tag, the image style preference, and the text information based on the assigned fusion weights to generate the text instruction.

[0017] In this embodiment, determining the emotional intensity value corresponding to the emotional label includes: extracting the acoustic features of the input speech and determining a first emotional intensity value using a first emotional intensity prediction model; wherein the first emotional intensity prediction model is configured to be trained based on a first dataset containing speech samples and their corresponding emotional intensity annotations; inputting the text information into a second emotional intensity prediction model to obtain a second emotional intensity value; wherein the second emotional intensity prediction model is trained based on a second dataset containing text samples and their corresponding emotional intensity annotations; and performing a weighted fusion of the first emotional intensity value and the second emotional intensity value to generate an emotional intensity value corresponding to the emotional label.

[0018] A third aspect of this application provides a processor configured to perform the speech-to-image method described above.

[0019] A fourth aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the speech-to-image method described above.

[0020] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the speech-to-image method described above.

[0021] The technical solution provided in this application has at least the following technical effects: The speech-to-image generation method of this application first extracts text information and voiceprint feature information from the user's speech input. Then, based on the voiceprint feature information, a target text-to-image model is determined, and the user's image style preference is predicted. The text information and image style preference are then fused to generate a text command. This text command is then input into the target text-to-image model to generate a target image that matches the user's intent. This speech-to-image generation method enables personalized model selection and style fusion, seamlessly transforming speech input into a customized image that highly matches the user's intent, making image creation more human-centered and improving the user experience.

[0022] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0023] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 A flowchart illustrating a method for generating images from speech according to an embodiment of this application is shown schematically. Figure 2 A schematic diagram of a speech-to-image device according to an embodiment of this application is shown. Figure 3 The diagram illustrates the internal structure of a computer device according to an embodiment of this application.

[0024] Explanation of reference numerals in the attached figures 200 - Voice-to-image generation device; 201 - Extraction module; 202 - Voiceprint feature information processing module; 203 - Fusion module; 204 - Image generation module; A01 - Processor; A02 - Network interface; A03 - Internal memory; A04 - Non-volatile storage medium; B01 - Operating system; B02 - Computer program. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0026] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0027] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0028] The acquisition, transmission, storage, use, and processing of data in this application comply with relevant national laws and regulations. Furthermore, it should be noted that existing industry solutions such as software, components, and models may be mentioned in the embodiments of this application. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0029] Figure 1 A schematic flowchart illustrating a method for generating images from speech according to an embodiment of this application is shown. Figure 1 As shown, in one embodiment of this application, a method for generating images from speech is provided. The method includes: S101: in response to receiving input speech from a user, extracting text information and voiceprint feature information from the input speech; S102: based on the voiceprint feature information, determining a target text-to-image model and predicting the user's image style preference; S103: fusing the text information and the image style preference to generate a text instruction; S104: inputting the text instruction into the target text-to-image model to generate a target image that matches the user's intent.

[0030] Specifically, in the embodiments of this application, the voice-to-image generation method provided by the present invention can be implemented through an application deployed on a cloud server or user terminal (such as a smartphone or personal computer). When a user wants to generate an image via voice, for example, the user speaks the command "draw a peaceful sunset" into the phone, the device's microphone captures the user's input voice. Then, the input voice is processed in parallel: on the one hand, automatic speech recognition technology is used to convert the input voice "draw a peaceful sunset" into corresponding text information; on the other hand, a voiceprint extraction algorithm (such as Mel-frequency cepstral coefficients (MFCC) or a deep learning model) is used to extract voiceprint feature information from the input voice that can uniquely identify the user.

[0031] Next, the extracted voiceprint features are analyzed to determine the user's personalized characteristics. Based on these features, a target image model that best matches the user is selected from a library of image models containing various styles (such as anime, realistic, watercolor, and oil painting styles). For example, a model that excels at generating aesthetically pleasing and soft styles is chosen. Simultaneously, based on the voiceprint features, the user's image style preferences are predicted, such as dreamy or highly saturated images.

[0032] Next, the original text information "Draw a peaceful sunset" is intelligently fused with the predicted image style preference "dreamy, high saturation" to generate a richer and more personalized text instruction, such as "A dreamy, high-saturation peaceful sunset, aesthetic style".

[0033] Finally, the fused text command is input into the selected target text-to-image model. The model generates an image based on the command, ultimately outputting a target image that conforms to both the user's original semantics and their personalized style preferences, which is then displayed on the user interface. The entire process achieves a seamless transition from voice input to personalized image output.

[0034] The speech-to-image generation method provided in this application can achieve a high degree of personalization and customization by deeply mining the voiceprint features in speech. It can not only identify the user's identity to dynamically select the most suitable generation model, but also intelligently integrate predicted image style preferences with text commands, thereby accurately matching the user's deep expectations in terms of content and style, significantly improving the matching degree between the generated results and the user's intent. Furthermore, this method integrates speech recognition, model selection, and image generation into a coherent and automated process. Users only need one natural voice input to obtain personalized works with their own characteristic style, truly achieving a seamless human-computer interaction experience where what you want is what you get, greatly enhancing the uniqueness of creation and the user experience.

[0035] In one embodiment, determining the target text-to-image model based on the voiceprint feature information and predicting the user's image style preference includes: constructing a user profile based on the voiceprint feature information; wherein the user profile includes, but is not limited to, age, gender, emotion, and occupational orientation; determining the target text-to-image model from a preset text-to-image model library based on the user profile, and predicting the user's image style preference based on the user profile.

[0036] Specifically, in this embodiment, after extracting the user's voiceprint feature information through the voiceprint recognition module, a deep learning model trained on massive amounts of voiceprint-user attribute data is invoked. For example, this model can analyze parameters such as accent, fundamental frequency, formants, speech rate, and prosody to infer the user's geographical location (South, North, Central, Southwest, or Northwest), age range (e.g., youth, middle-aged), gender (male, female), current emotional state (e.g., calm, excited, depressed), and even occupational orientation (e.g., leaning towards artistic creativity or business seriousness). These inferred tags collectively constitute the user's dynamic profile.

[0037] Subsequently, based on the user profile, the system selects the most suitable target model from a pre-defined library of text-based image models containing various style optimization models (such as ink painting style, cyberpunk style, and realistic photography style). For example, if the profile indicates the user is a young, creative woman, the system might choose a pop art style model as the target text-based image model; conversely, if the profile indicates a middle-aged, business-oriented man, it might choose a calm, classic realistic photography style model. While selecting the target model, the system further predicts more subtle visual element preferences based on the user profile. In one specific embodiment, this prediction process uses a collaborative filtering algorithm: a user-visual element preference matrix is ​​constructed, where users whose behavior is identified by voiceprint features are listed as visual elements such as color saturation, compositional dynamism, and lighting atmosphere. The matrix values ​​are calculated based on the user's historical behavior data. After a new user's profile is generated, its similarity to other users in the database is calculated, the K most similar users are identified, and the visual element preferences of these users are aggregated. A weighted average is then used to generate a set of specific preference parameters for color, composition, and lighting for the current user. In this way, not only is the most suitable target model selected for subsequent image generation, but the image style preferences that users may like are also initially outlined.

[0038] The voice-to-image generation method provided in this application can proactively understand and predict users' potential aesthetic preferences. This makes model selection no longer random or fixed, but an intelligent decision based on user characteristics, greatly improving the relevance of generated content to the user's personality. It provides a solid data foundation and decision-making basis for subsequent personalized image generation, ensuring that the final output not only conforms to the instructions in content but also matches the user's preferences in style. This significantly enhances the user's sense of exclusivity and satisfaction, forming the technological barrier that constitutes the core competitiveness of this solution.

[0039] In one embodiment, the method further includes: determining an emotion tag corresponding to the user's current emotional state based on the input speech; and fusing the emotion tag, the image style preference, and the text information to generate a text instruction.

[0040] Specifically, in this embodiment, the acoustic features of the input speech (such as speech rate, pitch, and volume) are analyzed to determine the user's emotional state at the moment of speaking, and a specific emotional label is output, such as excitement, calmness, or sadness. In the subsequent instruction fusion stage, this emotional label is used as a key modifier dimension and integrated with the image style preferences derived from the user profile (such as oil painting style or anime style) and the core text information identified from the speech (such as "a cat"). For example, if the emotional label is excitement, the generated text instruction might be "a vibrant, brightly colored cartoon cat, full of dynamism"; if the emotional label is sadness, the instruction might become "a quiet, curled-up, cool-toned realistic cat, with a melancholy atmosphere." In this way, the user's real-time emotions are directly encoded into the final generated instruction.

[0041] The voice-to-image generation method provided in this application can integrate the user's real-time emotional state into the image generation command, keenly capturing the user's instantaneous and dynamic emotional changes. This makes the generated image not only conform to the user's long-term aesthetics, but also accurately respond to the user's current mood and emotions, enhancing the warmth and depth of human-computer interaction, making the user feel truly understood, thereby bringing a stronger sense of immersion and emotional satisfaction, and significantly improving the user experience and the product's unique charm.

[0042] In one embodiment, determining the emotion label corresponding to the user's current emotional state based on the input speech includes: determining prosodic feature parameters based on the input speech; wherein the prosodic feature parameters include, but are not limited to, speech rate value, pitch variation value, and volume value; inputting the prosodic feature parameters into a pre-built emotion classification model to obtain the emotion category corresponding to the prosodic feature parameters, which is used as the emotion label.

[0043] Specifically, in this embodiment, acoustic analysis is performed on the input speech to extract prosodic feature parameters that reflect emotional state. These parameters include, but are not limited to: speech rate (measuring speaking speed), pitch variation (reflecting pitch fluctuations), and volume (characterizing sound strength). Next, this set of extracted prosodic feature parameters is input into a pre-built and trained emotion classification model. By learning the mapping relationship between prosodic features and emotion categories in a large amount of speech data, this model can perform pattern recognition and classification on the input parameters, ultimately outputting a specific emotion category, such as joy, sadness, or anger. This emotion category serves as the emotion label corresponding to the user's current emotional state.

[0044] The speech-to-image generation method provided in this application can directly extract emotional information from the prosodic features of speech, enabling accurate judgment of the user's emotional state and capturing the user's true emotional expression beyond the literal meaning. This avoids misjudgments of emotion caused by linguistic ambiguity, irony, or cultural differences, significantly improving the accuracy and reliability of emotional labeling. It provides a reliable data foundation for subsequently generating images that accurately reflect the user's true state of mind.

[0045] In one embodiment, fusing the emotion tag, the image style preference, and the text information to generate a text instruction includes: determining an emotion intensity value corresponding to the emotion tag; dynamically assigning fusion weights to the emotion tag, the image style preference, and the text information based on the emotion intensity value; wherein, if the emotion intensity value exceeds a preset threshold, the fusion weight of the emotion tag is higher than the fusion weight of the image style preference; and weighting the emotion tag, the image style preference, and the text information based on the assigned fusion weights to generate the text instruction.

[0046] Specifically, in this embodiment, an emotional intensity value corresponding to the emotional tag is first determined, which quantifies the intensity of the user's current emotion. Then, based on this emotional intensity value, fusion weights are dynamically assigned to the emotional tag, image style preference, and text information to be fused. When the emotional intensity value exceeds a preset threshold, it means the user's current emotion is very strong. In this case, the fusion weight of the emotional tag is configured to be higher than the fusion weight of the image style preference, ensuring that the emotion is given priority and prominence in the final image. Conversely, if the emotional intensity value is low, the weights of the image style preference and text information will be relatively higher. Finally, based on these dynamically assigned fusion weights, the emotional tag, image style preference, and text information are weighted and combined to form a balanced text instruction that accurately reflects the user's current intention and mood. In this embodiment, the preset threshold is derived from a large amount of user interaction data and is used to distinguish between the user's normal emotional state and strong emotional state. For example, this threshold can be set to 70% of the maximum emotional intensity value. When the emotional intensity value exceeds this threshold, the fusion weight of the emotional tag is dynamically elevated to a primary position, ensuring that the emotion is given priority and prominence in the final image.

[0047] The voice-to-image generation method provided in this application can intelligently determine the priority of a user's current needs. When a user is emotionally agitated, the method prioritizes meeting the need for emotional expression, generating an image that resonates with their strong emotions. When a user is emotionally calm, the method focuses more on meeting their long-term preferences and specific content descriptions. This ensures that the final generated image meets the user's core needs at a specific moment, making image creation more human-centered, significantly enhancing user satisfaction in each interaction, and achieving personalized and emotional creation.

[0048] In one embodiment, determining the emotional intensity value corresponding to the emotional label includes: extracting acoustic features of the input speech and determining a first emotional intensity value using a first emotional intensity prediction model; wherein the first emotional intensity prediction model is configured to be trained based on a first dataset containing speech samples and their corresponding emotional intensity annotations; inputting the text information into a second emotional intensity prediction model to obtain a second emotional intensity value; wherein the second emotional intensity prediction model is trained based on a second dataset containing text samples and their corresponding emotional intensity annotations; and performing a weighted fusion of the first emotional intensity value and the second emotional intensity value to generate an emotional intensity value corresponding to the emotional label.

[0049] Specifically, in this embodiment, acoustic features of the input speech are first extracted and fed into a first sentiment intensity prediction model. This first sentiment intensity prediction model is trained on a first dataset containing a large number of speech samples and their corresponding sentiment intensity annotations, thereby obtaining a first sentiment intensity value based on the physical features of the sound. Simultaneously, the text information previously extracted from the speech is input into a second sentiment intensity prediction model. This second sentiment intensity prediction model is trained on a second dataset containing a large number of text samples and their corresponding sentiment intensity annotations, thereby outputting a second sentiment intensity value based on semantic content. Finally, these two sentiment intensity values, derived from speech and text respectively, are weighted and fused to generate a more accurate sentiment intensity value that integrates acoustic features and semantic connotations.

[0050] The speech-to-image generation method provided in this application can overcome the limitations of a single information source, achieve complementary advantages and cross-validation, and provide reliable data support for subsequent dynamic weight allocation and personalized image generation.

[0051] In one embodiment, after generating the target image, the method further includes: in response to user feedback voice, extracting correction instruction information and reference style information from the feedback voice; and generating an updated image based on the correction instruction information, the reference style information, and the target image.

[0052] Specifically, in this embodiment, the interaction process does not terminate after the target image is generated; it continues to listen for and receive user feedback voice. For example, after seeing a landscape image, a user might say, "Something doesn't feel right. I want my smile to be brighter, and the overall style to resemble a Miyazaki animation." Upon receiving this feedback voice, it undergoes secondary analysis to precisely extract "correction instruction information" describing specific modification needs, such as "brighter smile," and "reference style information" providing aesthetic references, such as "Miyazaki animation style." Subsequently, these two types of information, along with the target image generated in the previous round, are used as common input to generate a completely new, updated image that better matches the user's immediate intent through image editing technology. This updated image is then presented to the user again, laying the foundation for possible feedback in the next round.

[0053] Please refer to Figure 2 In one embodiment of this application, a speech-to-image device 200 is provided. The speech-to-image device 200 includes: an extraction module 201, configured to extract text information and voiceprint feature information from the input speech in response to receiving input speech from a user; a voiceprint feature information processing module 202, configured to determine a target text-to-image model based on the voiceprint feature information, and predict the user's image style preference; a fusion module 203, configured to fuse the text information and the image style preference to generate a text instruction; and an image generation module 204, configured to input the text instruction into the target text-to-image model to generate a target image that matches the user's intent.

[0054] In one embodiment, determining the target text-to-image model based on the voiceprint feature information and predicting the user's image style preference includes: constructing a user profile based on the voiceprint feature information; wherein the user profile includes, but is not limited to, age, gender, emotion, and occupational orientation; determining the target text-to-image model from a preset text-to-image model library based on the user profile, and predicting the user's image style preference based on the user profile.

[0055] In one embodiment, the device further includes an emotion determination module, configured to determine an emotion tag corresponding to the user's current emotional state based on the input speech; the fusion module is further configured to fuse the emotion tag, the image style preference, and the text information to generate a text instruction.

[0056] In one embodiment, determining the emotion label corresponding to the user's current emotional state based on the input speech includes: determining prosodic feature parameters based on the input speech; wherein the prosodic feature parameters include, but are not limited to, speech rate value, pitch variation value, and volume value; inputting the prosodic feature parameters into a pre-built emotion classification model to obtain the emotion category corresponding to the prosodic feature parameters, which is used as the emotion label.

[0057] In one embodiment, fusing the emotion tag, the image style preference, and the text information to generate a text instruction includes: determining an emotion intensity value corresponding to the emotion tag; dynamically assigning fusion weights to the emotion tag, the image style preference, and the text information based on the emotion intensity value; wherein, if the emotion intensity value exceeds a preset threshold, the fusion weight of the emotion tag is higher than the fusion weight of the image style preference; and weighting the emotion tag, the image style preference, and the text information based on the assigned fusion weights to generate the text instruction.

[0058] In one embodiment, determining the emotional intensity value corresponding to the emotional label includes: extracting acoustic features of the input speech and determining a first emotional intensity value using a first emotional intensity prediction model; wherein the first emotional intensity prediction model is configured to be trained based on a first dataset containing speech samples and their corresponding emotional intensity annotations; inputting the text information into a second emotional intensity prediction model to obtain a second emotional intensity value; wherein the second emotional intensity prediction model is trained based on a second dataset containing text samples and their corresponding emotional intensity annotations; and performing a weighted fusion of the first emotional intensity value and the second emotional intensity value to generate an emotional intensity value corresponding to the emotional label.

[0059] The voice-to-image device includes a processor and a memory. The extraction module, voiceprint feature information processing module, fusion module, and image generation module are all stored as program units in the memory. The processor executes the program modules stored in the memory to implement the corresponding functions.

[0060] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and the method of generating images from speech can be implemented by adjusting kernel parameters.

[0061] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0062] This application provides a processor configured to execute the speech-to-image method described above.

[0063] This application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the speech-to-image method described above.

[0064] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. Figure 3 This schematic diagram illustrates the internal structure of a computer device according to an embodiment of the present application. The computer device includes a processor A01, a network interface A02, a memory (not shown), and a database (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A04. The network interface A02 is used for communication with external terminals via a network connection. When the computer program B02 is executed by the processor A01, it implements a voice-to-image method.

[0065] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0066] This application also provides a computer program product, which, when executed on a data processing device, is adapted to execute an initialization program having the following method steps: in response to receiving a user's input voice, extracting text information and voiceprint feature information from the input voice; determining a target text-to-image model based on the voiceprint feature information, and predicting the user's image style preference; fusing the text information and the image style preference to generate a text instruction; and inputting the text instruction into the target text-to-image model to generate a target image that matches the user's intent.

[0067] In one embodiment, determining the target text-to-image model based on the voiceprint feature information and predicting the user's image style preference includes: constructing a user profile based on the voiceprint feature information; wherein the user profile includes, but is not limited to, age, gender, emotion, and occupational orientation; determining the target text-to-image model from a preset text-to-image model library based on the user profile, and predicting the user's image style preference based on the user profile.

[0068] In one embodiment, the method further includes: determining an emotion tag corresponding to the user's current emotional state based on the input speech; and fusing the emotion tag, the image style preference, and the text information to generate a text instruction.

[0069] In one embodiment, determining the emotion label corresponding to the user's current emotional state based on the input speech includes: determining prosodic feature parameters based on the input speech; wherein the prosodic feature parameters include, but are not limited to, speech rate value, pitch variation value, and volume value; inputting the prosodic feature parameters into a pre-built emotion classification model to obtain the emotion category corresponding to the prosodic feature parameters, which is used as the emotion label.

[0070] In one embodiment, fusing the emotion tag, the image style preference, and the text information to generate a text instruction includes: determining an emotion intensity value corresponding to the emotion tag; dynamically assigning fusion weights to the emotion tag, the image style preference, and the text information based on the emotion intensity value; wherein, if the emotion intensity value exceeds a preset threshold, the fusion weight of the emotion tag is higher than the fusion weight of the image style preference; and weighting the emotion tag, the image style preference, and the text information based on the assigned fusion weights to generate the text instruction.

[0071] In one embodiment, determining the emotional intensity value corresponding to the emotional label includes: extracting acoustic features of the input speech and determining a first emotional intensity value using a first emotional intensity prediction model; wherein the first emotional intensity prediction model is configured to be trained based on a first dataset containing speech samples and their corresponding emotional intensity annotations; inputting the text information into a second emotional intensity prediction model to obtain a second emotional intensity value; wherein the second emotional intensity prediction model is trained based on a second dataset containing text samples and their corresponding emotional intensity annotations; and performing a weighted fusion of the first emotional intensity value and the second emotional intensity value to generate an emotional intensity value corresponding to the emotional label.

[0072] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0076] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0077] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0078] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0079] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0080] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for generating images from speech, characterized in that, The method for generating images from speech includes: In response to receiving a user's voice input, text information and voiceprint feature information are extracted from the voice input. Based on the voiceprint feature information, the target text-to-image model is determined, and the user's image style preference is predicted; The text information is fused with the image style preference to generate text instructions; The text command is input into the target image model to generate a target image that matches the user's intent.

2. The method for generating images from speech according to claim 1, characterized in that, The step of determining the target text-to-image model based on the voiceprint feature information and predicting the user's image style preference includes: A user profile is constructed based on the voiceprint feature information; wherein, the user profile includes, but is not limited to, age, gender, emotion, and occupational orientation; Based on the user profile, the target text image model is determined from a preset text image model library, and the user's image style preference is predicted based on the user profile.

3. The method for generating images from speech according to claim 1, characterized in that, The method further includes: Based on the input voice, determine the emotion tag corresponding to the user's current emotional state; The emotional tags, image style preferences, and text information are fused together to generate text instructions.

4. The method for generating images from speech according to claim 3, characterized in that, The step of determining the emotion tag corresponding to the user's current emotional state based on the input speech includes: Based on the input speech, prosodic feature parameters are determined; wherein, the prosodic feature parameters include, but are not limited to: speech rate value, pitch variation value, and volume value; The prosodic feature parameters are input into a pre-built sentiment classification model to obtain the sentiment category corresponding to the prosodic feature parameters, which is used as the sentiment label.

5. The method for generating images from speech according to claim 3, characterized in that, The step of fusing the emotion tags, the image style preferences, and the text information to generate text instructions includes: Determine the emotional intensity value corresponding to the emotional tag; Based on the emotion intensity value, fusion weights are dynamically assigned to the emotion tag, the image style preference, and the text information respectively; wherein, when the emotion intensity value exceeds a preset threshold, the fusion weight of the emotion tag is higher than the fusion weight of the image style preference; Based on the assigned fusion weights, the sentiment tags, the image style preferences, and the text information are weighted and combined to generate the text instructions.

6. The method for generating images from speech according to claim 5, characterized in that, Determining the emotional intensity value corresponding to the emotional tag includes: The acoustic features of the input speech are extracted, and a first emotion intensity value is determined by a first emotion intensity prediction model; wherein the first emotion intensity prediction model is configured to be trained based on a first dataset containing speech samples and their corresponding emotion intensity annotations; The text information is input into the second sentiment intensity prediction model to obtain the second sentiment intensity value; wherein, the second sentiment intensity prediction model is trained based on a second dataset containing text samples and their corresponding sentiment intensity labels; The first emotional intensity value and the second emotional intensity value are weighted and fused to generate an emotional intensity value corresponding to the emotional tag.

7. A device for generating images from speech, characterized in that, The speech-to-image generation device includes: The extraction module is used to extract text information and voiceprint feature information from the user's input voice in response to receiving the user's input voice. The voiceprint feature information processing module is used to determine the target text-to-image model and predict the user's image style preference based on the voiceprint feature information. The fusion module is used to fuse the text information with the image style preference to generate text instructions; The image generation module is used to input the text instruction into the target text-to-image model and generate a target image that matches the user's intent.

8. The speech-to-image device according to claim 7, characterized in that, The step of determining the target text-to-image model based on the voiceprint feature information and predicting the user's image style preference includes: A user profile is constructed based on the voiceprint feature information; wherein, the user profile includes, but is not limited to, age, gender, emotion, and occupational orientation; Based on the user profile, the target text image model is determined from a preset text image model library, and the user's image style preference is predicted based on the user profile.

9. The speech-to-image device according to claim 7, characterized in that, The device also includes an emotion determination module, used to determine an emotion label corresponding to the user's current emotional state based on the input voice; The fusion module is also used to fuse the emotion tag, the image style preference, and the text information to generate text instructions.

10. The speech-to-image device according to claim 9, characterized in that, The step of determining the emotion tag corresponding to the user's current emotional state based on the input speech includes: Based on the input speech, prosodic feature parameters are determined; wherein, the prosodic feature parameters include, but are not limited to: speech rate value, pitch variation value, and volume value; The prosodic feature parameters are input into a pre-built sentiment classification model to obtain the sentiment category corresponding to the prosodic feature parameters, which is used as the sentiment label.

11. The speech-to-image device according to claim 9, characterized in that, The step of fusing the emotion tags, the image style preferences, and the text information to generate text instructions includes: Determine the emotional intensity value corresponding to the emotional tag; Based on the emotion intensity value, fusion weights are dynamically assigned to the emotion tag, the image style preference, and the text information respectively; wherein, when the emotion intensity value exceeds a preset threshold, the fusion weight of the emotion tag is higher than the fusion weight of the image style preference; Based on the assigned fusion weights, the sentiment tags, the image style preferences, and the text information are weighted and combined to generate the text instructions.

12. The speech-to-image device according to claim 11, characterized in that, Determining the emotional intensity value corresponding to the emotional tag includes: The acoustic features of the input speech are extracted, and a first emotion intensity value is determined by a first emotion intensity prediction model; wherein the first emotion intensity prediction model is configured to be trained based on a first dataset containing speech samples and their corresponding emotion intensity annotations; The text information is input into the second sentiment intensity prediction model to obtain the second sentiment intensity value; wherein, the second sentiment intensity prediction model is trained based on a second dataset containing text samples and their corresponding sentiment intensity labels; The first emotional intensity value and the second emotional intensity value are weighted and fused to generate an emotional intensity value corresponding to the emotional tag.

13. A processor, characterized in that, It is configured to perform the speech-to-image method according to any one of claims 1 to 6.

14. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the speech-to-image method as described in any one of claims 1 to 6.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the speech-to-image method according to any one of claims 1 to 6.

Images