Training methods, inference methods, systems, electronic devices, storage media, and program products for visual language models

Abstract

This disclosure provides a training method, inference method, system, electronic device, storage medium, and program product for a visual language model. The visual language model includes a visual encoder, a projection layer, a text segmenter, and a large language model. The training method includes: constructing a first training dataset for facial attribute perception; performing a first-stage training on the visual language model based on the first training dataset to update the parameters of the visual encoder and projection layer; the first-stage training is used to enable the visual language model to have facial attribute perception capabilities; constructing a second training dataset including question-and-answer data samples for wearability attributes, appearance attributes, and expression attributes; and performing a second-stage training on the visual language model based on the second training dataset to update the parameters of the projection layer and the large language model; the second-stage training is used to enable the visual language model to provide targeted feedback based on facial attributes.

Description

Technical Field

[0001] This disclosure generally relates to the field of multimodal models, and more specifically, to a training method, inference method, system, electronic device, storage medium, and program product for a visual language model. Background Technology

[0002] In recent years, with the rapid development of deep learning and large-scale pre-trained models, multimodal models have made significant progress in the fields of image understanding and natural language processing. These models are able to handle the relationships between images, text, and other data types.

[0003] In the process of human-computer interaction, if we can accurately perceive the user's emotions and state and provide adaptive feedback, we can significantly improve the user experience.

[0004] However, existing multimodal models cannot effectively perceive users' emotions and states, nor can they provide adaptive feedback based on the perception results. This limits the widespread application of multimodal models in real-world human-computer interaction scenarios. Summary of the Invention

[0005] Exemplary embodiments of this disclosure provide a training method, inference method, system, electronic device, storage medium, and program product for a visual language model, which can solve at least one of the aforementioned problems existing in the prior art.

[0006] According to a first aspect of the present disclosure, a method for training a visual language model is provided. The visual language model includes: a visual encoder, a projection layer, a text segmenter, and a large language model. The training method includes: constructing a first training dataset for facial attribute perception, wherein the facial attributes include: clothing attributes, appearance attributes, and expression attributes; performing a first-stage training on the visual language model based on the first training dataset to update the parameters of the visual encoder and the projection layer, wherein the first-stage training is used to enable the visual language model to have facial attribute perception capabilities; constructing a second training dataset, wherein the second training dataset includes: question-and-answer data samples for clothing attributes, question-and-answer data samples for appearance attributes, and question-and-answer data samples for expression attributes; after completing the first-stage training, performing a second-stage training on the visual language model based on the second training dataset to update the parameters of the projection layer and the large language model, wherein the second-stage training is used to enable the visual language model to provide targeted feedback for facial attributes.

[0007] Optionally, the training method further includes: constructing a third training dataset for character style transfer; after completing the second stage of training, performing a third stage of training on the visual language model based on the third training dataset to update the parameters of the projection layer and the large language model; wherein the third stage of training is used to enable the visual language model to provide personalized feedback that reflects the style of a specific character.

[0008] Optionally, the clothing attributes include at least one of the following: whether wearing a mask, whether wearing glasses, whether wearing a hat, and whether wearing earrings; the appearance attributes include at least one of the following: gender, age, ethnicity, mouth, eyes, beard, makeup, hair length, hair color, and hairstyle; the expression attributes include at least one of the following: happy, surprised, neutral, sad, afraid, angry, and disgusted.

[0009] Optionally, the function of providing targeted feedback based on facial attributes includes: providing lifestyle and health suggestions based on the wearable attributes; providing image management suggestions based on the appearance attributes; and providing emotional responses based on the facial expression attributes.

[0010] Optionally, constructing the third training dataset for character style transfer includes: generating character profiles of multiple specific characters appearing in multiple literary works based on structured dialogues extracted from multiple literary works, and generating inherent knowledge question-and-answer data samples for each of the multiple specific characters; adjusting the question-and-answer data samples in the second training dataset based on the character profile of each specific character to obtain question-and-answer data samples that reflect the style of that specific character; and generating general question-and-answer data samples that reflect the style of each specific character based on the character profile of that specific character.

[0011] According to a second aspect of the present disclosure, a reasoning method for a visual language model is provided. The visual language model includes: a visual encoder, a projection layer, a text segmenter, and a large language model. The reasoning method includes: acquiring text content corresponding to user input; acquiring a user's face image captured by an image acquisition device; inputting the face image into the visual encoder and inputting the vector output by the visual encoder into the projection layer to obtain visual features output by the projection layer; inputting the text content into the text segmenter to obtain text features output by the text segmenter; and inputting the visual features and the text features into the large language model to obtain feedback content output by the large language model for responding to the user input and the facial attributes of the face image. The visual language model is trained by performing the training method described above.

[0012] According to a third aspect of the present disclosure, a training system for a visual language model is provided. The visual language model includes: a visual encoder, a projection layer, a text segmenter, and a large language model. The training system includes: a first data construction unit configured to construct a first training dataset for facial attribute perception, wherein the facial attributes include: clothing attributes, appearance attributes, and expression attributes; a first training unit configured to perform a first-stage training on the visual language model based on the first training dataset to update the parameters of the visual encoder and the projection layer, wherein the first-stage training enables the visual language model to possess facial attribute perception capabilities; a second data construction unit configured to construct a second training dataset, wherein the second training dataset includes: question-and-answer data samples for the clothing attributes, question-and-answer data samples for the appearance attributes, and question-and-answer data samples for the expression attributes; and a second training unit configured to, after completing the first-stage training, perform a second-stage training on the visual language model based on the second training dataset to update the parameters of the projection layer and the large language model, wherein the second-stage training enables the visual language model to provide targeted feedback for facial attributes.

[0013] According to a fourth aspect of the present disclosure, a reasoning system for a visual language model is provided. The visual language model includes: a visual encoder, a projection layer, a text segmenter, and a large language model. The reasoning system includes: a text acquisition unit configured to acquire text content corresponding to user input; an image acquisition unit configured to acquire a user's face image captured by an image acquisition device; a visual feature acquisition unit configured to input the face image into the visual encoder and input the vector output by the visual encoder into the projection layer to obtain visual features output by the projection layer; a text feature acquisition unit configured to input the text content into the text segmenter to obtain text features output by the text segmenter; and a feedback content acquisition unit configured to input the visual features and the text features into the large language model to obtain feedback content output by the large language model for responding to the user input and the facial attributes of the face image. The visual language model is trained by performing the training method described above.

[0014] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided that stores instructions which, when executed by a processor of an electronic device, enable the electronic device to perform the training method and / or inference method as described above.

[0015] According to a sixth aspect of the present disclosure, an electronic device is provided, the electronic device comprising: at least one processor; at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the training method and / or inference method as described above.

[0016] According to a seventh aspect of the present disclosure, a computer program product is provided, including computer-executable instructions that, when executed by at least one processor, implement the training method and / or inference method as described above.

[0017] According to exemplary embodiments of the present disclosure, the training method, inference method, system, electronic device, storage medium, and program product of the visual language model can accurately perceive the user's emotions and state during human-computer interaction by recognizing and analyzing the user's facial image, and generate behavioral feedback that adapts to the user's emotions and state, thereby significantly improving the user experience.

[0018] In addition, the style of a specific role (e.g., a pre-specified system role) can be transferred to behavioral feedback to form personalized behavioral feedback through specific role-playing.

[0019] In the following description, some aspects and / or advantages of the general concept of this disclosure will be set forth, and other aspects and / or advantages will become apparent from the following description or from practice of the general concept of this disclosure. Attached Figure Description

[0020] These and / or other aspects and advantages of this application will become clearer and more readily understood from the following detailed description of embodiments of this application taken in conjunction with the accompanying drawings, wherein:

[0021] Figure 1 An example of a visual language model according to an exemplary embodiment of the present disclosure is shown;

[0022] Figure 2 A flowchart illustrating a method for training a visual language model according to an exemplary embodiment of the present disclosure;

[0023] Figure 3 Examples of training methods for visual language models according to exemplary embodiments of the present disclosure are shown;

[0024] Figure 4 A flowchart illustrating a method for constructing a third training dataset for character style transfer according to an exemplary embodiment of the present disclosure;

[0025] Figure 5 An example of a method for constructing a third training dataset for character style transfer according to an exemplary embodiment of the present disclosure is shown;

[0026] Figure 6 A flowchart illustrating a reasoning method for a visual language model according to an exemplary embodiment of the present disclosure;

[0027] Figure 7 Examples of reasoning methods for visual language models according to exemplary embodiments of the present disclosure are shown;

[0028] Figure 8 A structural block diagram of a training system for a visual language model according to an exemplary embodiment of the present disclosure is shown.

[0029] Figure 9 A structural block diagram of a reasoning system for a visual language model according to an exemplary embodiment of the present disclosure is shown.

[0030] Figure 10 A structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown. Detailed Implementation

[0031] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, examples of which are illustrated in the drawings, wherein the same reference numerals always refer to the same parts. The embodiments will now be described with reference to the accompanying drawings in order to explain this disclosure.

[0032] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0033] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0034] Currently, most multimodal models are still in the early stages of their ability to perceive facial attributes (such as emotion, gender, and age), and they have not yet been able to effectively provide real-time behavioral feedback based on facial attribute perception. This limits the widespread application of multimodal models in real-world human-computer interaction scenarios, especially those requiring personalized interaction. Furthermore, most existing facial attribute recognition systems focus on object recognition in static images, lacking sufficient understanding of changes in facial attributes and their meanings in dynamic environments, making efficient human-computer interaction difficult. Therefore, it is necessary to develop a system capable of perceiving facial attributes and providing real-time behavioral feedback to improve user experience and interaction effectiveness.

[0035] This disclosure aims to address the insufficient visual information perception capabilities of existing dialogue systems by combining facial attribute perception with text analysis to generate more adaptive and emotional intelligent feedback. The following will combine... Figures 1 to 10 Please provide a detailed explanation.

[0036] Figure 1 An example of a visual language model according to an exemplary embodiment of the present disclosure is shown.

[0037] A visual language model according to an exemplary embodiment of the present disclosure includes: a visual encoder, a projection layer, a text segmenter, and a large language model.

[0038] A visual encoder is used to encode a face image to obtain a corresponding vector.

[0039] The projection layer is used to extract features from the vectors output by the visual encoder to obtain visual features.

[0040] Text segmenters are used to process text content and obtain text features. As an example, a text segmenter can capture the semantic information of text content and generate a context-dependent embedding vector for each word in the text content.

[0041] Large language models are used to perform multimodal fusion of visual and textual features, outputting feedback content that responds to textual content and facial attributes of face images. Furthermore, large language models can transfer the style of specific roles (e.g., pre-defined system roles) to the feedback content, forming personalized feedback content that has been role-played.

[0042] Specifically, the large language model can perceive the user's emotions and state based on visual features, and generate appropriate behavioral feedback in real time to respond to changes in the user's state based on the perception results and text analysis results. In addition, it can inject the tone and expression of specific characters (e.g., language style, knowledge background and catchphrases) to reflect the character's personality in the behavioral feedback, thereby enhancing the interactive fun and user experience.

[0043] Figure 2A flowchart illustrating a method for training a visual language model according to an exemplary embodiment of the present disclosure is provided.

[0044] Reference Figure 2 and Figure 3 In step S201, a first training dataset for face attribute perception is constructed.

[0045] Facial attributes include: clothing attributes, appearance attributes, and expression attributes.

[0046] As an exemplary embodiment, wearable attributes may include, but are not limited to, at least one of the following: whether or not a mask is worn, whether or not glasses are worn, whether or not a hat is worn, and whether or not earrings are worn.

[0047] As an exemplary embodiment, appearance attributes may include, but are not limited to, at least one of the following: gender, age, ethnicity, mouth, eyes, beard, makeup, hair length, hair color, and hairstyle.

[0048] As an exemplary embodiment, the expression attributes may include, but are not limited to, at least one of the following: happy, surprised, neutral, sad, afraid, angry, disgusted.

[0049] As an exemplary embodiment, each piece of data (face image - attribute label) in a public and / or private face attribute dataset can be converted into a question-and-answer data format (face image - question - answer about the description of face attributes) through prompt word engineering and a specified large language model (e.g., a large language model other than the visual language model described above). Then, the converted data can be cleaned to remove ambiguity and vague answers, and the cleaned data can be used as a question-and-answer data sample for the first training dataset.

[0050] In step S202, the visual language model is trained in the first stage based on the first training dataset to update the parameters of the visual encoder and projection layer. The first stage of training is used to enable the visual language model to have the function of facial attribute perception.

[0051] As an example, in the first stage of training, the parameters of the text segmenter and the large language model can be frozen, and only the parameters of the visual encoder and the projection layer can be trained.

[0052] As an example, in the first stage of training, the face images from the question-and-answer data samples in the first training dataset can be input into the visual encoder, and the vectors output by the visual encoder can be input into the projection layer to obtain the visual features output by the projection layer; the questions from the question-and-answer data samples can be input into the text segmenter to obtain the text features output by the text segmenter; the visual features and text features can be input into the large language model to obtain the inference result output by the large language model; the loss can be calculated based on the difference between the answer and the inference result in the question-and-answer data sample; and the parameters of the visual encoder and the projection layer can be updated based on the calculated loss.

[0053] In step S203, a second training dataset is constructed.

[0054] The second training dataset includes: question-and-answer data samples oriented towards wearable attributes, question-and-answer data samples oriented towards appearance attributes, and question-and-answer data samples oriented towards facial expression attributes.

[0055] Question-and-answer data samples based on wearable attributes may include: facial images, questions, and answers regarding wearable attributes based on the facial images. As an example, answers regarding wearable attributes based on facial images could include lifestyle and health advice for users regarding whether they are wearing masks, glasses, hats, earrings, etc. For instance, if a user is not wearing a mask, advice could be provided on wearing a mask in environments with poor air quality to maintain health.

[0056] Sample question-and-answer data based on appearance attributes may include: facial images, questions, and answers based on the appearance attributes of the facial images. As an example, answers based on the appearance attributes of facial images could be: image management suggestions provided to users based on their gender, age, ethnicity, eyes, mouth, hair, and other appearance attributes. For example, recommending suitable hairstyle changes based on the user's hairstyle and hair color, or recommending appropriate makeup styles and products based on the user's skin tone and facial features.

[0057] Question-and-answer data samples oriented towards facial expression attributes may include: facial images, questions, and answers based on the facial expression attributes of the facial images. As an example, answers based on facial expression attributes could be: emotional responses provided to users based on the emotions and states reflected in their facial expressions. For example, providing targeted feedback (such as encouragement or comfort) or suggestions (such as guiding users through emotion regulation activities). For instance, providing comforting statements or recommending relaxation activities for users experiencing negative emotions.

[0058] As an exemplary embodiment, question-and-answer data samples oriented towards wear attributes, appearance attributes, and facial expression attributes can be generated through prompt word engineering and a specified large language model (e.g., a large language model other than the visual language model described above). Then, the generated question-and-answer data samples can be cleaned to remove ambiguous and vague answers, and the cleaned data can be used as question-and-answer data samples for a second training dataset.

[0059] It should be understood that step S203 can be executed before step S204. This disclosure does not limit the execution order of step S203 with steps S201 and S202. For example, step S203 can be executed after step S201.

[0060] In step S204, after completing the first stage of training, the visual language model is trained in the second stage based on the second training dataset to update the parameters of the projection layer and the large language model.

[0061] The second stage of training is used to enable the visual language model to provide targeted feedback based on facial attributes. As an exemplary embodiment, the function of providing targeted feedback based on facial attributes may include, but is not limited to: providing lifestyle and health suggestions based on clothing attributes; providing image management suggestions based on appearance attributes; and providing emotional responses based on expression attributes.

[0062] As an example, in the second stage of training, the parameters of the text segmenter and visual encoder can be frozen, and only the parameters of the projection layer and the large language model can be trained.

[0063] As an example, in the second stage of training, face images from question-answering data samples in the second training dataset can be input into a visual encoder, and the vectors output by the visual encoder can be input into a projection layer to obtain visual features output by the projection layer; questions from the question-answering data samples can be input into a text segmenter to obtain text features output by the text segmenter; visual features and text features can be input into a large language model to obtain the inference result output by the large language model; the loss can be calculated based on the difference between the answers and the inference results in the question-answering data samples; and the parameters of the projection layer and the large language model can be updated based on the calculated loss. For example, the loss can be optimized using image-text matching and language generation, and adaptive learning rate and mixed precision strategies can be used to improve training efficiency.

[0064] The visual language model trained according to the exemplary embodiments of this disclosure can perceive the user's emotions and state in real time through multimodal fusion technology, and generate feedback strategies for clothing matching, image management, emotional comfort, etc.

[0065] In addition, refer to Figure 3The training method for the visual language model according to an exemplary embodiment of the present disclosure may further include steps S205 and S206.

[0066] In step S205, a third training dataset for character style transfer is constructed.

[0067] The following will combine Figure 4 and Figure 5 An exemplary embodiment of step S205 will be described below, but will not be elaborated here.

[0068] In step S206, after completing the second stage of training, the visual language model is trained in the third stage based on the third training dataset to update the parameters of the projection layer and the large language model.

[0069] The third stage of training is used to enable the visual language model to provide personalized feedback that reflects the style of a specific character. That is, in the third stage of training, personalized behavioral feedback learning is carried out, so that the model has the ability to provide feedback according to the speaking style of a character (e.g., Sun Wukong), thereby reflecting the character's personality in the feedback and enhancing the interactive fun and user experience.

[0070] As an example, in the third stage of training, the parameters of the text segmenter and visual encoder can be frozen, and only the parameters of the projection layer and the large language model can be trained.

[0071] Figure 4 A flowchart illustrating a method for constructing a third training dataset for character style transfer according to an exemplary embodiment of the present disclosure is shown.

[0072] Reference Figure 4 and Figure 5 In step S401, based on the structured dialogue extracted from multiple literary works, character portraits of multiple specific characters appearing in the multiple literary works are generated, and inherent knowledge question-and-answer data samples for each of the multiple specific characters are generated.

[0073] As an example, the types of literary works may include, but are not limited to: novels, screenplays, plays, and fairy tales.

[0074] It should be understood that the multiple literary works mentioned may include a diverse range of high-quality and widely covered literary works to ensure the richness, representativeness, and completeness of the data sources.

[0075] As an example, GPT can be used to segment dialogues in literary works and extract structured dialogues.

[0076] As an example, character profiles can be generated using GPT based on structured dialogues extracted from literary works; that is, character definition can be performed. A character profile is a description of a character, which may include, but is not limited to: personality, language style (e.g., catchphrases, tone of voice, manner of expression), behavioral tendencies, and emotional characteristics.

[0077] As an example, structured dialogues extracted from literary works can be used to generate character-specific innate knowledge question-answering data through prompt word engineering (Prompt) and GPT to cover common questions related to character innate knowledge. For instance, the innate knowledge question-answering data can be a triple consisting of question-answer-confidence, from which high-quality, non-repeating data is selected as innate knowledge question-answering data samples in a third training dataset based on the confidence level.

[0078] In step S402, for each of the multiple specific roles, the question-and-answer data samples in the second training dataset are adjusted based on the role profile of that specific role to obtain question-and-answer data samples that reflect the style of that specific role.

[0079] That is, character style transfer is performed on the question-and-answer data samples in the second training dataset. As an example, character prompts, system instructions and retrieval enhancements based on dialogue engineering can be designed. GPT can be used to change the question-and-answer data samples in the second training dataset, such as injecting the character's language style, knowledge background and catchphrases into the answers.

[0080] In step S403, for each of the multiple specific roles, a general question-and-answer data sample reflecting the style of that specific role is generated based on the role profile of that specific role.

[0081] That is, character style transfer is performed on general question-and-answer data samples other than those oriented towards facial attributes. As an example, character prompts, system commands, and retrieval enhancements based on dialogue engineering can be designed, and GPT can be used to modify question-and-answer data samples in the general question-and-answer dataset, such as injecting the character's language style, knowledge background, and catchphrases into the answers.

[0082] It should be understood that the question-and-answer data samples generated in steps S401-S403 together constitute the third training dataset.

[0083] Figure 6 A flowchart illustrating an inference method for a visual language model according to exemplary embodiments of the present disclosure is provided. The visual language model includes a visual encoder, a projection layer, a text segmenter, and a large language model. The visual language model is trained by performing the training method described in the exemplary embodiments above.

[0084] Reference Figure 6In step S601, the text content corresponding to the user input is obtained.

[0085] As an example, user input can be either voice input or text input. If the user input is voice input, then the voice input is converted into text.

[0086] In step S602, the user's face image is acquired by the image acquisition device.

[0087] In step S603, the face image is input into the visual encoder, and the vector output by the visual encoder is input into the projection layer to obtain the visual features output by the projection layer.

[0088] In step S604, the text content is input into the text segmenter to obtain the text features output by the text segmenter.

[0089] In step S605, visual features and text features are input into the large language model to obtain feedback content output by the large language model for responding to user input and facial attributes of the face image, thereby responding to changes in the user's state in real time.

[0090] In addition, the large language model can inject the language style, knowledge background and catchphrases of specific characters into the feedback content, that is, generate feedback content that can reflect the personality of specific characters, and enhance the interactive fun and user experience.

[0091] Furthermore, as an exemplary embodiment, reference is made to Figure 7 This system can combine historical interaction data with users to achieve deep fusion of multimodal information through adaptive learning mechanisms. For example, based on historical interaction data with users, reinforcement learning can be used to optimize feedback strategies and improve interaction effectiveness. Specifically, historical interaction data with users can be recorded, and reinforcement learning can be used to generate more suitable feedback content based on users' historical emotional responses, preferences, and feedback. In other words, adaptive learning mechanisms can be used to dynamically adjust feedback content and optimize the interactive experience.

[0092] This invention innovatively applies multimodal data fusion to the field of behavioral feedback, which not only improves the accuracy of behavioral feedback but also enhances the personalization and interactivity of the user experience. It is applicable to multiple industries such as education, healthcare, and entertainment, and has significant application potential in fields such as intelligent assistants, virtual customer service, and affective computing, with broad application prospects and social value.

[0093] Figure 8 A block diagram illustrating a training system for a visual language model according to an exemplary embodiment of the present disclosure is shown. The visual language model includes: a visual encoder, a projection layer, a text segmenter, and a large language model.

[0094] Reference Figure 8The visual language model training system 800 according to an exemplary embodiment of the present disclosure includes: a first data construction unit 801, a first training unit 802, a second data construction unit 803, and a second training unit 804.

[0095] Specifically, the first data construction unit 801 is configured to construct a first training dataset for facial attribute perception, wherein the facial attributes include: clothing attributes, appearance attributes, and expression attributes.

[0096] The first training unit 802 is configured to perform a first-stage training on the visual language model based on the first training dataset to update the parameters of the visual encoder and the projection layer, wherein the first-stage training is used to enable the visual language model to have face attribute perception function.

[0097] The second data construction unit 803 is configured to construct a second training dataset, wherein the second training dataset includes: question-and-answer data samples oriented towards the wearable attributes, question-and-answer data samples oriented towards the appearance attributes, and question-and-answer data samples oriented towards the facial expression attributes.

[0098] The second training unit 804 is configured to perform a second phase of training on the visual language model based on the second training dataset after completing the first phase of training, so as to update the parameters of the projection layer and the large language model. The second phase of training is used to enable the visual language model to provide targeted feedback based on facial attributes.

[0099] Figure 9 A block diagram of a reasoning system for a visual language model according to exemplary embodiments of the present disclosure is shown. The visual language model includes: a visual encoder, a projection layer, a text segmenter, and a large language model. The visual language model is trained by performing the training method described in the exemplary embodiments above.

[0100] Reference Figure 9 The visual language model inference system 900 according to an exemplary embodiment of the present disclosure includes: a text acquisition unit 901, an image acquisition unit 902, a visual feature acquisition unit 903, a text feature acquisition unit 904, and a feedback content acquisition unit 905.

[0101] Specifically, the text acquisition unit 901 is configured to acquire the text content corresponding to user input.

[0102] The image acquisition unit 902 is configured to acquire a user's face image captured by an image acquisition device.

[0103] The visual feature acquisition unit 903 is configured to input the face image into the visual encoder and input the vector output by the visual encoder into the projection layer to obtain the visual features output by the projection layer.

[0104] The text feature acquisition unit 904 is configured to input the text content into the text segmenter to obtain the text features output by the text segmenter.

[0105] The feedback content acquisition unit 905 is configured to input the visual features and the text features into the large language model to obtain feedback content output by the large language model for responding to the user input and the facial attributes of the face image.

[0106] It should be understood that the specific processing performed by the above system has already referred to Figures 1 to 7 A detailed description has been provided, and the relevant details will not be repeated here.

[0107] It should be understood that each unit in the above system can be implemented as a hardware component and / or a software component.

[0108] Figure 10 A structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0109] Reference Figure 10 The electronic device 1000 includes at least one memory 1001 and at least one processor 1002. The at least one memory 1001 stores a set of computer-executable instructions. When the set of computer-executable instructions is executed by the at least one processor 1002, the training method and / or inference method as described in the exemplary embodiments above are executed.

[0110] As an example, an electronic device can be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, the electronic device is not necessarily a single device; it can be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. The electronic device can also be part of an integrated control system or system manager, or can be configured to interconnect locally or remotely (e.g., via wireless transmission) through an interface.

[0111] In an electronic device, processor 1002 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor 1002 may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0112] The processor 1002 can execute instructions or code stored in the memory 1001, which can also store data. Instructions and data can also be sent and received via a network through a network interface device, which can employ any known transmission protocol.

[0113] The memory 1001 may be integrated with the processor 1002, for example, by arranging RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 1001 may include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. The memory 1001 and the processor 1002 may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 1002 to read files stored in the memory.

[0114] In addition, electronic devices may include video displays (such as liquid crystal displays) and user interaction interfaces (such as keyboards, mice, touch input devices, etc.). All components of the electronic device may be interconnected via buses and / or networks.

[0115] According to exemplary embodiments of this disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, they cause at least one processor to perform the training method and / or inference method as described in the exemplary embodiments above. Examples of computer-readable storage media herein include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0116] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, wherein the instructions in the computer program product are executable by at least one processor as described in the exemplary embodiments above, such as the training method and / or inference method.

[0117] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0118] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for a visual language model, characterized in that, The visual language model includes: a visual encoder, a projection layer, a text segmenter, and a large language model, wherein the training method includes: Construct a first training dataset for face attribute perception, wherein the face attributes include: clothing attributes, appearance attributes, and expression attributes; The visual language model is trained in the first stage based on the first training dataset to update the parameters of the visual encoder and the projection layer. The first stage training is used to enable the visual language model to have the function of facial attribute perception. Construct a second training dataset, wherein the second training dataset includes: question-and-answer data samples oriented towards the wearable attributes, question-and-answer data samples oriented towards the appearance attributes, and question-and-answer data samples oriented towards the facial expression attributes; After completing the first stage of training, the visual language model is trained in the second stage based on the second training dataset to update the parameters of the projection layer and the large language model. The second stage of training is used to enable the visual language model to provide targeted feedback based on facial attributes. The function of providing targeted feedback based on facial attributes includes: The function provides lifestyle and health advice based on the aforementioned wearable attributes; The function provides image management suggestions based on the aforementioned appearance attributes; The function of providing emotional responses to the aforementioned facial expression attributes.

2. The training method according to claim 1, characterized in that, The training method also includes: Construct a third training dataset for character style transfer; After completing the second stage of training, the visual language model is trained in the third stage based on the third training dataset to update the parameters of the projection layer and the large language model. The third stage of training is used to enable the visual language model to provide personalized feedback that reflects the style of a specific character.

3. The training method according to claim 1, characterized in that, The wearable attributes include at least one of the following: whether or not a mask is worn, whether or not glasses are worn, whether or not a hat is worn, and whether or not earrings are worn; The appearance attributes include at least one of the following: gender, age, ethnicity, mouth, eyes, beard, makeup, hair length, hair color, and hairstyle; The facial expression attributes include at least one of the following: happy, surprised, neutral, sad, afraid, angry, disgusted.

4. The training method according to claim 2, characterized in that, The construction of the third training dataset for character style transfer includes: Based on structured dialogues extracted from multiple literary works, character profiles of multiple specific characters appearing in the multiple literary works are generated, and inherent knowledge question-and-answer data samples are generated for each of the multiple specific characters. For each of the multiple specific roles, the question-and-answer data samples in the second training dataset are adjusted based on the role profile of that specific role to obtain question-and-answer data samples that reflect the style of that specific role; For each of the multiple specific roles, a general question-and-answer data sample reflecting the style of that specific role is generated based on the role profile of that specific role.

5. A reasoning method for a visual language model, characterized in that, The visual language model includes: a visual encoder, a projection layer, a text segmenter, and a large language model, wherein the inference method includes: Get the text content corresponding to the user input; Acquire facial images of users captured by image acquisition devices; The face image is input into the visual encoder, and the vector output by the visual encoder is input into the projection layer to obtain the visual features output by the projection layer. The text content is input into the text segmenter to obtain the text features output by the text segmenter; The visual features and the text features are input into the large language model to obtain the feedback content output by the large language model for responding to the user input and the facial attributes of the face image; The visual language model is trained by performing the training method as described in any one of claims 1 to 4.

6. A training system for a visual language model, characterized in that, The visual language model includes: a visual encoder, a projection layer, a text segmenter, and a large language model, wherein the training system includes: The first data construction unit is configured to construct a first training dataset for facial attribute perception, wherein the facial attributes include: clothing attributes, appearance attributes, and expression attributes. The first training unit is configured to perform a first-stage training on the visual language model based on the first training dataset to update the parameters of the visual encoder and the projection layer, wherein the first-stage training is used to enable the visual language model to have the function of facial attribute perception. The second data construction unit is configured to construct a second training dataset, wherein the second training dataset includes: question-and-answer data samples oriented towards the wearable attributes, question-and-answer data samples oriented towards the appearance attributes, and question-and-answer data samples oriented towards the facial expression attributes; The second training unit is configured to perform a second phase of training on the visual language model based on the second training dataset after the first phase of training is completed, so as to update the parameters of the projection layer and the large language model. The second phase of training is used to enable the visual language model to provide targeted feedback for facial attributes. The function of providing targeted feedback based on facial attributes includes: The function provides lifestyle and health advice based on the aforementioned wearable attributes; The function provides image management suggestions based on the aforementioned appearance attributes; The function of providing emotional responses to the aforementioned facial expression attributes.

7. A reasoning system for a visual language model, characterized in that, The visual language model includes: a visual encoder, a projection layer, a text segmenter, and a large language model, wherein the inference system includes: The text acquisition unit is configured to acquire the text content corresponding to user input. The image acquisition unit is configured to acquire a user's face image captured by an image acquisition device; The visual feature acquisition unit is configured to input the face image into the visual encoder and input the vector output by the visual encoder into the projection layer to obtain the visual features output by the projection layer. The text feature acquisition unit is configured to input the text content into the text segmenter and obtain the text features output by the text segmenter; The feedback content acquisition unit is configured to input the visual features and the text features into the large language model to obtain the feedback content output by the large language model for responding to the user input and the facial attributes of the face image; The visual language model is trained by performing the training method as described in any one of claims 1 to 4.

8. A computer-readable storage medium for storing instructions, characterized in that, When the instructions are executed by the processor of the electronic device, the electronic device is enabled to perform the training method as described in any one of claims 1 to 4 and / or the inference method as described in claim 5.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; At least one memory that stores computer-executable instructions. Wherein, when the computer-executable instructions are executed by the at least one processor, they cause the at least one processor to perform the training method as described in any one of claims 1 to 4 and / or the inference method as described in claim 5.

10. A computer program product comprising computer-executable instructions, characterized in that, When the computer-executable instructions are executed by at least one processor, they implement the training method as described in any one of claims 1 to 4 and / or the inference method as described in claim 5.

Images