Multimodal text and image generation method and device, equipment and storage medium
By combining a lightweight network and a multilayer perceptron with an encoder to generate multimodal models, the problem of high computational resources and data requirements for multimodal models is solved, achieving efficient multimodal image and text generation and expanding its application scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2024-09-30
- Publication Date
- 2026-07-31
AI Technical Summary
Existing multimodal models have extremely high requirements for computing resources and training data, which limits their application scenarios.
Lightweight networks such as GhostNet are used for feature extraction, combined with multilayer perceptrons and encoders such as DistilBERT, to generate multimodal images and text through a multimodal model generator, and the model is trained using the cross-entropy loss function.
It reduces the computational resource and training data requirements of multimodal models, improves the efficiency of multimodal image and text generation, and expands its application scenarios.
Smart Images
Figure CN119399318B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for generating multimodal graphics and text. Background Technology
[0002] Image and text generation is a core branch of multimodal learning. With the emergence of pre-trained multimodal models, these models can flexibly handle various data formats such as images and text.
[0003] However, current multimodal models have extremely high demands on computing resources and training data, which limits their application scenarios. Therefore, how to generate current multimodal images and text using multimodal image generation models is an urgent problem to be solved. Summary of the Invention
[0004] This invention provides a method, apparatus, computer device, and storage medium for generating multimodal graphics and text, in order to solve the technical problem of how to generate current multimodal graphics and text through a multimodal graphics and text generation model.
[0005] Firstly, a multimodal image and text generation method is provided, including:
[0006] Obtain a preset product image and the preset multimodal graphics corresponding to the preset product image;
[0007] The preset product image is used to extract features through a lightweight network to generate a first feature vector. The first feature vector is then transformed using a preset multilayer perceptron to obtain the transformed first feature vector.
[0008] Feature extraction is performed on the preset multimodal image and text to generate a second feature vector. The converted first feature vector and the second feature vector are concatenated to obtain a third feature vector.
[0009] The third feature vector is encoded using a multimodal model encoder to obtain the fourth feature vector;
[0010] The fourth feature vector is input into the generator of the multimodal model to obtain the predicted multimodal image and text output by the generator based on the fourth feature vector. The multimodal model is trained according to the loss value between the predicted multimodal image and text and the preset multimodal image and text.
[0011] When the loss value meets the preset conditions, the current multimodal image and text generated by the trained multimodal model based on the current product image is obtained.
[0012] Further, the step of extracting features from the preset product image using a lightweight network to generate a first feature vector, and then transforming the first feature vector using a preset multilayer perceptron to obtain the transformed first feature vector, includes:
[0013] The GhostNet model, a multimodal model, is used to extract features from the preset product image through a lightweight network to generate a first feature vector.
[0014] The first feature vector is input into a preset multilayer perceptron, and the multilayer perceptron is used to transform the first feature vector to obtain the transformed first feature vector.
[0015] Further, the step of extracting features from the preset multimodal image and text to generate a second feature vector, and concatenating the converted first feature vector and the second feature vector to obtain a third feature vector, includes:
[0016] The preset multimodal image and text are input into the embedding layer of the multimodal model to obtain the second feature vector output by the embedding layer based on the preset multimodal image and text;
[0017] The transformed first feature vector and the second feature vector are concatenated to obtain the third feature vector.
[0018] Furthermore, the encoder using the multimodal model encodes the third feature vector to obtain the fourth feature vector, including:
[0019] The third feature vector is input into the encoder;
[0020] The third feature vector is encoded using the attention mechanism of the encoder to obtain the fourth feature vector.
[0021] Further, the step of inputting the fourth feature vector into the generator of the multimodal model to obtain the predicted multimodal graph and text output by the generator based on the fourth feature vector, and training the multimodal model according to the loss value between the predicted multimodal graph and text and the preset multimodal graph and text, includes:
[0022] The fourth feature vector is input into the generator of the multimodal model to obtain the predicted multimodal image and text output by the generator based on the fourth feature vector;
[0023] The loss value between the predicted multimodal image and the preset multimodal image is obtained by using a preset loss function. The multimodal model is trained based on the loss value. The loss function includes one or a combination of cross-entropy loss function and absolute loss function.
[0024] Further, the step of obtaining the current multimodal image and text generated by the trained multimodal model based on the current product image when the loss value meets a preset condition includes:
[0025] When the loss value converges, stop training the multimodal model and save the trained multimodal model;
[0026] The current product image is acquired, and the image encoder is used to extract features from the current product image to obtain a fifth feature vector. The fifth feature vector is then input into the trained multimodal model to obtain the current multimodal image and text generated by the trained multimodal model based on the fifth feature vector.
[0027] Further, after obtaining the current multimodal image and text generated by the trained multimodal model based on the current product image when the loss value meets the preset condition, the multimodal image and text generation method includes:
[0028] The current product image and the current multimodal graphics are displayed through a preset display window.
[0029] Secondly, a multimodal image and text generation device is provided, comprising:
[0030] The first acquisition module is used to acquire a preset product image and a preset multimodal graphic corresponding to the preset product image;
[0031] The first extraction module is used to extract features from the preset product image through a lightweight network, generate a first feature vector, and transform the first feature vector using a preset multilayer perceptron to obtain the transformed first feature vector.
[0032] The second extraction module is used to extract features from the preset multimodal image and text, generate a second feature vector, and concatenate the converted first feature vector and the second feature vector to obtain a third feature vector;
[0033] The encoding module is used to encode the third feature vector using the encoder of the multimodal model to obtain the fourth feature vector;
[0034] The input module is used to input the fourth feature vector into the generator of the multimodal model, obtain the predicted multimodal image and text output by the generator based on the fourth feature vector, and train the multimodal model according to the loss value between the predicted multimodal image and text and the preset multimodal image and text.
[0035] The generation module is used to obtain the current multimodal image and text generated by the trained multimodal model based on the current product image when the loss value meets the preset conditions.
[0036] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described multimodal image and text generation method.
[0037] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described multimodal image and text generation method.
[0038] This application provides a multimodal image and text generation method, apparatus, computer device, and storage medium. The method involves acquiring a preset product image and a corresponding preset multimodal image and text; extracting features from the preset product image using a lightweight network to generate a first feature vector; transforming the first feature vector using a preset multilayer perceptron to obtain a transformed first feature vector; extracting features from the preset multimodal image and text to generate a second feature vector; concatenating the transformed first feature vector and the second feature vector to obtain a third feature vector; encoding the third feature vector using an encoder of a multimodal model to obtain a fourth feature vector; and inputting the fourth feature vector into a generator of the multimodal model to obtain a predicted multimodal image and text output by the generator based on the fourth feature vector. The method involves training a multimodal model based on the loss value between the predicted multimodal image and the preset multimodal image. When the loss value meets a preset condition, the current multimodal image generated by the trained multimodal model based on the current product image is obtained. This method has two advantages: First, obtaining the current multimodal image generated by the trained multimodal model based on the current product image when the loss value meets the preset condition reduces the generation time of the current multimodal image for the current product image since manual writing is not required. This improves the generation efficiency of the current multimodal image for the current product image and can be applied to the fintech business field. Second, the lightweight network reduces the demand for computing resources and training data for the multimodal model, thus expanding the application scenarios of the multimodal model. Attached Figure Description
[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0040] Figure 1 This is a schematic diagram of an application environment for a multimodal image and text generation method according to an embodiment of the present invention;
[0041] Figure 2 This is a schematic flowchart of a multimodal image and text generation method provided in an embodiment of the present invention;
[0042] Figure 3 yes Figure 1 A flowchart illustrating a specific implementation of step S23;
[0043] Figure 4 yes Figure 1 A schematic diagram of a specific implementation method for step S25;
[0044] Figure 5 yes Figure 1 A schematic diagram of a specific implementation method for step S26;
[0045] Figure 6 This is a schematic diagram of a multimodal image and text generation device according to an embodiment of the present invention;
[0046] Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;
[0047] Figure 8 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation
[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0049] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application environment of the multimodal image and text generation method according to an embodiment of the present invention. The multimodal image and text generation method provided in this embodiment of the present invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network.
[0050] The server obtains a preset product image and the corresponding preset multimodal graphics and text from the client.
[0051] The preset product image is used to extract features through a lightweight network to generate a first feature vector. The first feature vector is then transformed using a preset multilayer perceptron to obtain the transformed first feature vector.
[0052] Feature extraction is performed on the preset multimodal image and text to generate a second feature vector. The converted first feature vector and the second feature vector are concatenated to obtain a third feature vector.
[0053] The third feature vector is encoded using a multimodal model encoder to obtain the fourth feature vector;
[0054] The fourth feature vector is input into the generator of the multimodal model to obtain the predicted multimodal image and text output by the generator based on the fourth feature vector. The multimodal model is trained according to the loss value between the predicted multimodal image and text and the preset multimodal image and text.
[0055] When the loss value meets the preset conditions, the current multimodal image and text generated by the trained multimodal model based on the current product image is obtained.
[0056] The beneficial effects of the above-mentioned multimodal image and text generation method, apparatus, device, and medium are twofold. Firstly, when the loss value meets the preset conditions, the current multimodal image and text generated by the trained multimodal model based on the current product image is obtained. Since no manual writing is required, the generation time of the current multimodal image and text of the current product image is reduced. Multimodal image and text can improve the generation efficiency of the current multimodal image and text of the current product image and can be applied to the financial technology business field. Secondly, due to the lightweight network, the demand of the multimodal model on computing resources and training data can be reduced, which is conducive to expanding the application scenarios of the multimodal model.
[0057] The client can include, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices.
[0058] The server can be implemented using a separate task database or a task database cluster consisting of multiple task databases. The invention will be described in detail below through specific embodiments.
[0059] Please see Figure 2 , Figure 2 A flowchart illustrating a multimodal image and text generation method according to an embodiment of the present invention includes the following steps:
[0060] S21, acquire a preset product image and a preset multimodal graphic corresponding to the preset product image;
[0061] The preset product image is a product image that has already been processed.
[0062] Among them, the preset multimodal graphics and text are preset multimodal graphics and text of product images.
[0063] Among them, the preset multi-modal graphics and texts can introduce the products in the preset product images.
[0064] S22, through a lightweight network, extract features from the preset product image to generate a first feature vector, and use a preset multi-layer perceptron to transform the first feature vector to obtain the transformed first feature vector;
[0065] Among them, the multi-layer perceptron (Multi-Layer Perceptron, MLP), as a feedforward neural network, one of its core functions is to process input data through multi-layer non-linear transformations.
[0066] Among them, the lightweight network includes but is not limited to the MobileNet model and the GhostNet model.
[0067] Among them, the full English name of the MobileNet model is MobileNets, and the MobileNet model achieves efficient computational performance and a relatively small number of parameters.
[0068] Among them, the full Chinese name of the GhostNet model is: Ghost Network, and the full English name is: GhostNet.
[0069] Among them, GhostNet is a lightweight convolutional neural network. GhostNet can generate more feature maps without adding too much computational effort, thereby enhancing the multi-modal model's ability to capture image details, making the multi-modal model highly valuable in environments with limited computational resources such as mobile devices and embedded systems.
[0070] Preferably, the lightweight network adopts the GhostNet model.
[0071] When the lightweight network adopts the GhostNet model, the step of extracting features from the preset product image through the lightweight network to generate a first feature vector and using a preset multi-layer perceptron to transform the first feature vector to obtain the transformed first feature vector includes:
[0072] Through the GhostNet model of the multi-modal model, use the lightweight network to extract features from the preset product image to generate a first feature vector; <00,00166>
[0073] Input the first feature vector into a preset multi-layer perceptron, and use the multi-layer perceptron to transform the first feature vector to obtain the transformed first feature vector.
[0074] Exemplarily, the multi-modal model includes the GhostNet model, the multi-layer perceptron, the embedding layer, the encoder, and the generator.
[0075] For example, the output of the GhostNet model is connected to the input of a multilayer perceptron, the output of the multilayer perceptron is connected to the input of an embedding layer, the output of the embedding layer is connected to the input of an encoder, and the output of the encoder is connected to the input of a generator.
[0076] In this process, a multilayer perceptron is used to perform dimensionality transformation on the first feature vector through its multiple hidden layers to obtain the transformed first feature vector.
[0077] For ease of explanation, the following example is provided:
[0078] For example, if the first feature vector is 1024-dimensional, a multilayer perceptron can be used to transform the dimension of the first feature vector through its multiple hidden layers, resulting in a transformed first feature vector of 768 dimensions.
[0079] In this model, the preset product image data and the preset multimodal image and text data have different dimensions. The multilayer perceptron, through appropriate dimensionality transformation, can convert these preset product image data and preset multimodal image and text data into a common representation space, allowing the data from the preset product images and preset multimodal image and text to complement and enhance each other. This cross-modal feature fusion is beneficial for improving the generalization ability and performance of the multimodal model.
[0080] S23, extract features from the preset multimodal image and text to generate a second feature vector, and concatenate the converted first feature vector and the second feature vector to obtain a third feature vector;
[0081] S24, using the encoder of the multimodal model, the third feature vector is encoded to obtain the fourth feature vector;
[0082] The process of encoding the third feature vector using a multimodal model to obtain the fourth feature vector includes:
[0083] The third feature vector is input into the encoder;
[0084] The third feature vector is encoded using the attention mechanism of the encoder to obtain the fourth feature vector.
[0085] For example, the encoder is a DistilBERT model or a BERT model.
[0086] Among them, the full Chinese name of the DistilBERT model is: Simplified BERT, and the full English name of the DistilBERT model is: Distilled Bidirectional Encoder Representations from Transformers.
[0087] Among them, the full Chinese name of the BERT model is: Bidirectional Encoder Representations from Transformers, and the full English name of the BERT model is: Bidirectional Encoder Representations from Transformers.
[0088] Among them, the full Chinese name of the DistilBERT model is: Simplified BERT, and the full English name of the DistilBERT model is: Distilled Bidirectional Encoder Representations from Transformers.
[0089] Preferably, the encoder is a DistilBERT model.
[0090] Among them, the DistilBERT model is a lightweight and efficient version of the BERT model. It extracts key knowledge from the original BERT model through knowledge distillation technology, thus significantly reducing the model size and computational complexity while maintaining high performance.
[0091] S25, input the fourth feature vector into the generator of the multimodal model, obtain the predicted multimodal text and image based on the fourth feature vector output by the generator, and train the multimodal model according to the loss value between the predicted multimodal text and image and the preset multimodal text and image;
[0092] S26, when the loss value meets the preset condition, obtain the current multimodal text and image generated by the trained multimodal model based on the current product image.
[0093] Among them, the current product image is an unprocessed product image.
[0094] Among them, the current multimodal text and image is the multimodal text and image of the current product image.
[0095] Among them, the products in the current product image include financial products and insurance products.
[0096] Among them, the multimodal text and image is used to describe the text and image content that combines multiple modalities, and multiple modalities include but are not limited to text, image, sound, and video.
[0097] Multimodal graphics and text are becoming increasingly important in modern information expression and dissemination because they can provide a richer and more comprehensive information experience. By combining different modalities, multimodal graphics and text can break through the limitations of traditional single-modal communication, enabling cross-modal transmission and understanding of information.
[0098] For example, in news reports, in addition to text descriptions, relevant pictures, videos and audio content can be added to enable readers to understand the whole picture of the event more intuitively.
[0099] In advertising design, combining multiple elements such as images, text, and sound can create more attractive and impactful advertising effects.
[0100] Wherein, after obtaining the current multimodal image and text generated by the trained multimodal model based on the current product image when the loss value meets the preset condition, the multimodal image and text generation method includes:
[0101] The current product image and the current multimodal graphics are displayed through a preset display window.
[0102] Using multimodal graphics and text to introduce the product in the current product image is an efficient and accurate way to convey information. This approach can explain and supplement the product's features, functions, advantages, and other information in the current product image in text form.
[0103] In this embodiment of the invention, the beneficial effects are twofold. Firstly, when the loss value meets the preset conditions, the current multimodal image and text generated by the trained multimodal model based on the current product image are obtained. Since no manual writing is required, the generation time of the current multimodal image and text of the current product image is reduced. The multimodal image and text is beneficial to improving the generation efficiency of the current multimodal image and text of the current product image, and can be applied to the financial technology business field. Secondly, due to the lightweight network, the demand of the multimodal model on computing resources and training data can be reduced, which is beneficial to expanding the application scenarios of the multimodal model.
[0104] Please see Figure 3 , Figure 3 yes Figure 1 A detailed flowchart of a specific implementation method for step S23 is described below:
[0105] S31, input the preset multimodal image and text into the embedding layer of the multimodal model, and obtain the second feature vector output by the embedding layer based on the preset multimodal image and text;
[0106] S32, the transformed first feature vector and the second feature vector are concatenated to obtain the third feature vector.
[0107] In this embodiment of the invention, the transformed first feature vector and the second feature vector are concatenated, allowing the third feature vector to contain more comprehensive information. This increase in information helps the multimodal model capture more complex correlations and patterns between data, thereby improving the multimodal model's ability to understand and analyze data.
[0108] Please see Figure 4 , Figure 4 yes Figure 1 A detailed flowchart of a specific implementation method for step S25 is described below:
[0109] S41, input the fourth feature vector into the generator of the multimodal model, and obtain the predicted multimodal image and text output by the generator based on the fourth feature vector;
[0110] S42, using a preset loss function, obtain the loss value between the predicted multimodal image and the preset multimodal image, and train the multimodal model based on the loss value. The loss function includes one or a combination of cross-entropy loss function and absolute loss function.
[0111] In this embodiment of the invention, during the training process of the multimodal model, the loss value serves as a key indicator for measuring the difference between the predicted multimodal graph and the preset multimodal graph, and its optimization level directly determines the technical performance of the multimodal model. By continuously iteratively adjusting the model parameters of the multimodal model, the loss value is reduced, thereby improving the fitting ability and generalization performance of the multimodal model.
[0112] Please see Figure 5 , Figure 5 yes Figure 1 A detailed flowchart of a specific implementation method for step S26 is described below:
[0113] S51, When the loss value converges, stop training the multimodal model and save the trained multimodal model;
[0114] S52, acquire the current product image, use the image encoder to extract features from the current product image to obtain a fifth feature vector, input the fifth feature vector into the trained multimodal model, and acquire the current multimodal image and text generated by the trained multimodal model based on the fifth feature vector.
[0115] For example, acquiring a current product image, using the image encoder to extract features from the current product image to obtain a fifth feature vector, inputting the fifth feature vector into the trained multimodal model, and acquiring the current multimodal image and text generated by the trained multimodal model based on the fifth feature vector, includes:
[0116] The system connects to a target platform, acquires a current product image sent by the target platform, extracts features from the current product image using the image encoder to obtain a fifth feature vector, inputs the fifth feature vector into the trained multimodal model, and acquires the current multimodal image and text generated by the trained multimodal model based on the fifth feature vector. The target platform includes an insurance platform and a financial platform.
[0117] In this embodiment of the invention, the current multimodal image and text generated by the trained multimodal model based on the fifth feature vector is obtained, which reduces the generation time of the current multimodal image and text of the current product image. The multimodal image and text is beneficial to improving the generation efficiency of the current multimodal image and text of the current product image, and can be applied to the field of financial technology business.
[0118] Please see Figure 6 , Figure 6 This is a schematic diagram of a multimodal image and text generation device according to an embodiment of the present invention, as shown below. Figure 6 As shown, the multimodal image and text generation device includes a first acquisition module 101, a first extraction module 102, a second extraction module 103, an encoding module 104, an input module 105, and a generation module 106. Detailed descriptions of each functional module are as follows:
[0119] The first acquisition module 101 is used to acquire a preset product image and a preset multimodal graphic image corresponding to the preset product image;
[0120] The first extraction module 102 is used to extract features from the preset product image through a lightweight network, generate a first feature vector, and transform the first feature vector using a preset multilayer perceptron to obtain the transformed first feature vector.
[0121] The second extraction module 103 is used to extract features from the preset multimodal image and text, generate a second feature vector, and concatenate the converted first feature vector and the second feature vector to obtain a third feature vector.
[0122] Encoding module 104 is used to encode the third feature vector using an encoder of a multimodal model to obtain a fourth feature vector;
[0123] The input module 105 is used to input the fourth feature vector into the generator of the multimodal model, obtain the predicted multimodal image and text output by the generator based on the fourth feature vector, and train the multimodal model according to the loss value between the predicted multimodal image and text and the preset multimodal image and text.
[0124] The generation module 106 is used to obtain the current multimodal image and text generated by the trained multimodal model based on the current product image when the loss value meets the preset conditions.
[0125] In one embodiment, the first extraction module 102 includes:
[0126] The extraction subunit is used to extract features from the preset product image through a lightweight network using the GhostNet model of the multimodal model, and generate a first feature vector.
[0127] The transformation subunit is used to input the first feature vector into a preset multilayer perceptron and use the multilayer perceptron to transform the first feature vector to obtain the transformed first feature vector.
[0128] In one embodiment, the second extraction module 103 includes:
[0129] A sub-unit is used to input the preset multimodal image and text into the embedding layer of the multimodal model and obtain the second feature vector output by the embedding layer based on the preset multimodal image and text;
[0130] The splicing subunit is used to splice the transformed first feature vector and the second feature vector to obtain the third feature vector.
[0131] In one embodiment, the encoding module 104 includes:
[0132] The first input subunit is used to input the third feature vector into the encoder;
[0133] The encoding subunit is used to encode the third feature vector using the attention mechanism of the encoder to obtain the fourth feature vector.
[0134] In one embodiment, the input module 105 includes:
[0135] The second input subunit is used to input the fourth feature vector into the generator of the multimodal model and obtain the predicted multimodal image and text output by the generator based on the fourth feature vector;
[0136] The training subunit is used to obtain the loss value between the predicted multimodal image and the preset multimodal image through a preset loss function, and to train the multimodal model based on the loss value. The loss function includes one or a combination of cross-entropy loss function and absolute loss function.
[0137] In one embodiment, the generation module 106 includes:
[0138] A save subunit is used to stop training the multimodal model and save the trained multimodal model when the loss value converges.
[0139] A sub-unit is generated to acquire the current product image, and the image encoder is used to extract features from the current product image to obtain a fifth feature vector. The fifth feature vector is then input into the trained multimodal model to obtain the current multimodal image and text generated by the trained multimodal model based on the fifth feature vector.
[0140] In one embodiment, the multimodal image and text generation apparatus further includes:
[0141] The display module is used to display the current product image and the current multimodal graphics through a preset display window.
[0142] In this embodiment of the invention, the beneficial effects are twofold. Firstly, when the loss value meets the preset conditions, the current multimodal image and text generated by the trained multimodal model based on the current product image are obtained. Since no manual writing is required, the generation time of the current multimodal image and text of the current product image is reduced. The multimodal image and text is beneficial to improving the generation efficiency of the current multimodal image and text of the current product image, and can be applied to the financial technology business field. Secondly, due to the lightweight network, the demand of the multimodal model on computing resources and training data can be reduced, which is beneficial to expanding the application scenarios of the multimodal model.
[0143] For specific limitations on the multimodal image generation device, please refer to the limitations on the multimodal image generation method above, which will not be repeated here.
[0144] Each module in the aforementioned multimodal image and text generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0145] Please see Figure 7 , Figure 7 This is a schematic diagram of a computer device according to one embodiment of the present invention. In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown below. Figure 7 As shown. This computer device includes a processor, memory, network interface, and database connected via a system bus.
[0146] The processor of this computer device provides computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface of the computer device is used for communication with external clients via a network connection. When the computer program is executed by the processor, it can implement the functions or steps of a multimodal image and text generation method on the server side.
[0147] Please see Figure 8 , Figure 8 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. In one embodiment, a computer device is provided, which can be a client, and its internal structure diagram can be as follows. Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external task database via a network connection. When the computer program is executed by the processor, it can implement the functions or steps of a multimodal image and text generation method on a client side.
[0148] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor.
[0149] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions of the server and client in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.
[0150] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), graphics processing units (GPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0151] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and subsamples of some embodiments may be included in or replace parts and subsamples of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used herein means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated subsamples, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other subsamples, wholes, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes the element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.
[0152] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0153] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some sub-samples may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to implement this embodiment according to actual needs. Furthermore, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. 。
Claims
1. A method for generating a multi-modal graph-text, the method comprising: include: Obtain a preset product image and a preset multimodal graphic text corresponding to the preset product image, wherein the modal types of the multimodal graphic text include at least the following four types: text, image, sound, and video; The GhostNet model of the multimodal model is used to extract features from the preset product image to generate a first feature vector. The first feature vector is then input into a preset multilayer perceptron, and the multilayer perceptron is used to transform the first feature vector to obtain the transformed first feature vector. The preset multimodal image and text are input into the embedding layer of the multimodal model to obtain the second feature vector output by the embedding layer based on the preset multimodal image and text. The converted first feature vector and the second feature vector are concatenated to obtain the third feature vector. The third feature vector is encoded using the encoder of the multimodal model to obtain the fourth feature vector; The fourth feature vector is input into the generator of the multimodal model to obtain the predicted multimodal image and text output by the generator based on the fourth feature vector. The multimodal model is trained according to the loss value between the predicted multimodal image and text and the preset multimodal image and text. When the loss value meets the preset conditions, the current multimodal image and text generated by the trained multimodal model based on the current product image is obtained.
2. The multi-modal map generation method of claim 1, wherein, The encoder using the multimodal model encodes the third feature vector to obtain the fourth feature vector, including: The third feature vector is input into the encoder; The third feature vector is encoded using the attention mechanism of the encoder to obtain the fourth feature vector.
3. The multimodal image and text generation method according to claim 1, characterized in that, The step of inputting the fourth feature vector into the generator of the multimodal model, obtaining the predicted multimodal image and text output by the generator based on the fourth feature vector, and training the multimodal model according to the loss value between the predicted multimodal image and text and the preset multimodal image and text includes: The fourth feature vector is input into the generator of the multimodal model to obtain the predicted multimodal image and text output by the generator based on the fourth feature vector; The loss value between the predicted multimodal image and the preset multimodal image is obtained by using a preset loss function. The multimodal model is trained based on the loss value. The loss function includes one or a combination of cross-entropy loss function and absolute loss function.
4. The multimodal image and text generation method according to claim 1, characterized in that, When the loss value meets a preset condition, obtaining the current multimodal image and text generated by the trained multimodal model based on the current product image includes: When the loss value converges, stop training the multimodal model and save the trained multimodal model; The current product image is acquired, and features are extracted from the current product image using an image encoder to obtain a fifth feature vector. The fifth feature vector is then input into the trained multimodal model to obtain the current multimodal image and text generated by the trained multimodal model based on the fifth feature vector.
5. The multimodal image and text generation method according to any one of claims 1 to 4, characterized in that, After obtaining the current multimodal image and text generated by the trained multimodal model based on the current product image when the loss value meets a preset condition, the multimodal image and text generation method includes: The current product image and the current multimodal graphics are displayed through a preset display window.
6. A multimodal image and text generation device, characterized in that, include: The acquisition module is used to acquire a preset product image and a preset multimodal graphic text corresponding to the preset product image, wherein the modal types of the multimodal graphic text include at least the following four types: text, image, sound, and video; The extraction module is used to extract features from the preset product image through the GhostNet model of the multimodal model and a lightweight network to generate a first feature vector. The first feature vector is then input into a preset multilayer perceptron, and the multilayer perceptron is used to transform the first feature vector to obtain the transformed first feature vector. The splicing module is used to input the preset multimodal image and text into the embedding layer of the multimodal model, obtain the second feature vector output by the embedding layer based on the preset multimodal image and text, and splice the converted first feature vector and the second feature vector to obtain a third feature vector; The encoding module is used to encode the third feature vector using the encoder of the multimodal model to obtain the fourth feature vector; The input module is used to input the fourth feature vector into the generator of the multimodal model, obtain the predicted multimodal image and text output by the generator based on the fourth feature vector, and train the multimodal model according to the loss value between the predicted multimodal image and text and the preset multimodal image and text. The generation module is used to obtain the current multimodal image and text generated by the trained multimodal model based on the current product image when the loss value meets the preset conditions.
7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the multimodal image and text generation method as described in any one of claims 1 to 5.
8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal image generation method as described in any one of claims 1 to 5.