Content generation method and device, equipment, storage medium and product

Abstract

The invention discloses a content generation method and device, equipment, a storage medium and a product. The method comprises the following steps: inputting a target image and a target prompt text into a trained multi-modal language model; performing visual coding on the target image through at least two visual encoders in the multi-modal language model to obtain at least two visual coding features, and performing feature fusion on the at least two visual coding features through a feature fusion module in the multi-modal language model to obtain visual fusion features; performing text coding on the target prompt text through a text feature network in the multi-modal language model to obtain text coding features; according to the method, the visual fusion features and the text coding features are input into the content output network in the multi-modal language model, the output response text content corresponding to the multi-modal data is obtained, the visual information expression ability of the multi-modal language model is improved, and therefore the accuracy of the multi-modal language model is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a content generation method, device, equipment, storage medium and product. Background Art

[0002] With the rapid development of artificial intelligence technology, single-modal language models can no longer meet the increasingly complex application requirements, so multimodal language models have emerged. Multimodal language models can integrate visual information and text information to achieve cross-modal interactive question and answer.

[0003] In the process of implementing the present invention, it is found that there are at least the following technical problems in the prior art:

[0004] Multimodal language models have deficiencies in visual processing and find it difficult to fully capture and express complex and diverse visual information, resulting in low accuracy of multimodal language models. Summary of the invention

[0005] Embodiments of the present invention provide a content generation method, apparatus, device, storage medium and product to solve the problem that multimodal language models have poor expression capabilities for visual information and improve the accuracy of multimodal language models.

[0006] According to an embodiment of the present invention, a content generation method is provided, the method comprising:

[0007] Inputting multimodal data into a trained multimodal language model; wherein the multimodal data includes a target image and a target prompt text, the multimodal language model includes a visual feature network, a text feature network and a content output network, and the visual feature network includes a feature fusion module and a visual encoder corresponding to at least two visual encoding characteristics;

[0008] By means of at least two visual encoders, the target image is visually encoded respectively to obtain at least two visual encoding features, and by means of the feature fusion module, the at least two visual encoding features are feature-fused to obtain a visual fusion feature;

[0009] By means of the text feature network, the target prompt text is encoded to obtain text encoding features;

[0010] The visual fusion feature and the text encoding feature are input into the content output network to obtain output response text content corresponding to the multimodal data.

[0011] According to another embodiment of the present invention, a content generation device is provided, the device comprising:

[0012] A multimodal data input module, used to input multimodal data into a trained multimodal language model; wherein the multimodal data includes a target image and a target prompt text, the multimodal language model includes a visual feature network, a text feature network and a content output network, and the visual feature network includes a feature fusion module and a visual encoder corresponding to at least two visual encoding characteristics;

[0013] A visual fusion feature determination module is used to perform visual encoding on the target image through at least two visual encoders to obtain at least two visual encoding features, and to perform feature fusion on the at least two visual encoding features through the feature fusion module to obtain a visual fusion feature;

[0014] A text encoding feature determination module, used to perform text encoding on the target prompt text through the text feature network to obtain text encoding features;

[0015] The response text content output module is used to input the visual fusion feature and the text encoding feature into the content output network to obtain the output response text content corresponding to the multimodal data.

[0016] According to another embodiment of the present invention, there is provided an electronic device, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the content generating method described in any embodiment of the present invention.

[0020] According to another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores computer instructions. The computer instructions are used to enable a processor to implement the content generation method described in any embodiment of the present invention when executed.

[0021] According to another embodiment of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the content generating method according to any embodiment of the present invention is implemented.

[0022] An embodiment of the above invention has at least the following advantages or beneficial effects:

[0023] By configuring visual encoders corresponding to at least two visual encoding characteristics in a multimodal language model, visually encoding the target image through the at least two visual encoders to obtain at least two visual encoding features, and fusing the at least two visual encoding features through a feature fusion module to obtain a visual fusion feature, the problem of poor expression ability of the multimodal language model for visual information is solved, the response performance of the multimodal language model to complex image tasks is improved, and the accuracy and richness of the response text content of the multimodal language model are improved.

[0024] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A flow chart of a content generation method provided by an embodiment of the present invention;

[0027] Figure 2 A model architecture diagram of a multimodal language model provided by an embodiment of the present invention;

[0028] Figure 3 A flowchart of another content generation method provided by an embodiment of the present invention;

[0029] Figure 4 A network architecture diagram of a visual feature network provided by an embodiment of the present invention;

[0030] Figure 5 A schematic diagram of the structure of a content generation device provided by an embodiment of the present invention;

[0031] Figure 6 The present invention is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", "initial", "target", "reference", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Figure 1 This is a flowchart of a content generation method provided by an embodiment of the present invention. This embodiment is applicable to the case of generating response text content matching multi-modal input data. The method can be executed by a content generation device. The content generation device can be implemented in the form of hardware and / or software. The content generation device can be configured in a terminal device. Figure 1 As shown, the method includes:

[0035] S110: input the multimodal data into the trained multimodal language model.

[0036] In this embodiment, the multimodal data includes a target image and a target prompt text. Specifically, the target image provides a visual object for the multimodal language model, and the target prompt text can be used to guide the multimodal language model to focus on the specified image content in the target image, guide the multimodal language model to understand the response task for the target image, provide additional content supplementary information for the multimodal language model, and define the standard format of the response text content of the multimodal language model, etc.

[0037] In this embodiment, the multimodal language model includes a visual feature network, a text feature network and a content output network. The visual feature network includes a feature fusion module and a visual encoder corresponding to at least two visual encoding characteristics.

[0038] Specifically, the network architectures of multiple visual encoders in the visual feature network may be the same or different. When the network architectures of multiple visual encoders are the same, the differences in training parameters such as training samples and training loss functions of the visual encoders will also make the visual encoders with the same network architecture have different visual encoding characteristics.

[0039] Specifically, the visual coding characteristics represent the intrinsic performance of the visual encoder in converting the target image into a feature vector. Exemplarily, the visual coding characteristics include feature extraction, hierarchy, and object selection characteristics. For example, with respect to feature extraction, the visual encoder A focuses on extracting the color features and shape features of the target image, and the visual encoder B focuses on extracting the texture features of the target image. With respect to hierarchy, the visual encoder A uses a serial hierarchy to perform serial feature encoding on the target image, and the visual encoder B uses a parallel hierarchy to perform parallel feature encoding on the target image. With respect to object selectivity, the visual encoder A focuses on the classification and recognition of the target objects in the target image, such as if the target image contains items A and B, and the visual encoder B focuses on optical character recognition in the target image.

[0040] The differences in visual encoding characteristics of different visual encoders are only schematically illustrated here, and are not limited thereto. The visual encoder can be selected and configured according to actual needs.

[0041] S120. Visually encode the target image using at least two visual encoders to obtain at least two visual encoding features, and fuse the at least two visual encoding features using a feature fusion module to obtain a visual fusion feature.

[0042] In a specific embodiment, exemplarily, the at least two visual encoders include but are not limited to convolutional neural networks, autoencoders, recurrent neural networks, and attention networks, etc.

[0043] In another specific embodiment, the visual encoder includes a hierarchical fusion module and at least two hierarchical encoding modules, wherein the hierarchical encoding module is used to encode the corresponding input data to obtain visual hierarchical features, and the hierarchical fusion module is used to fuse at least two visual hierarchical features to output visual coding features.

[0044] Specifically, the visual encoder is a serial hierarchical network. When the hierarchical encoding module is the first hierarchical encoding module in the visual encoder, the input data is the target image. When the hierarchical encoding module is a hierarchical encoding module other than the first hierarchical encoding module, the input data is the visual hierarchical features output by the previous hierarchical encoding module connected in series with the hierarchical encoding module in the visual encoder.

[0045] In a specific embodiment, the fusion processing adopted by the hierarchical fusion module is splicing processing, addition processing, multiplication processing, average processing or attention processing.

[0046] The advantage of setting up multi-level fusion is that it integrates the semantic information and spatial information expressed by different levels of encoding modules in the visual encoder, further improving the ability of the multimodal language model to express visual information.

[0047] In another specific embodiment, the hierarchical fusion module includes a hierarchical splicing unit and a dimensionality reduction unit. The hierarchical splicing unit is used to splice at least two visual hierarchical features to obtain a visual splicing feature, and the dimensionality reduction unit is used to perform dimensionality reduction processing on the visual splicing feature and output a visual coding feature.

[0048] Exemplarily, the dimension reduction unit is a 1×1 convolutional layer or a fully connected layer, but is not limited to the example case.

[0049] For example, the visual encoding feature h output by the i-th visual encoder in the visual feature network is i Satisfies the following formula:

[0050]

[0051] in, represents the visual splicing feature, Represents the visual level features output by the L-th level encoding module in the i-th visual encoder.

[0052] The benefit of setting up a dimensionality reduction unit is that it prevents the feature dimension of the visual encoding features from being too large, occupying too many visual encoding resources, and increasing the risk of overfitting of the multimodal language model, while ensuring the efficiency of content generation.

[0053] In a specific embodiment, a feature fusion module is used to perform feature fusion on at least two visual coding features to obtain a visual fusion feature, including: adding the at least two visual coding features to obtain the visual fusion feature through the feature fusion module; or, multiplying the at least two visual coding features to obtain the visual fusion feature through the feature fusion module; or, concatenating the at least two visual coding features to obtain the visual fusion feature through the feature fusion module; or, performing mean processing on the at least two visual coding features to obtain the visual fusion feature through the feature fusion module, or, performing attention processing on the at least two visual coding features to obtain the visual fusion feature through the feature fusion module.

[0054] Specifically, the visual fusion feature represents the summation result, product result, concatenation result, mean result or attention result of at least two visual encoding features.

[0055] Exemplarily, when the visual fusion feature represents the sum of at least two visual encoding features, the visual fusion feature h final Satisfies the following formula:

[0056]

[0057] Wherein, E represents a visual encoder set consisting of at least two visual encoders in the visual feature network.

[0058] S130. Using a text feature network, text encoding is performed on the target prompt text to obtain text encoding features.

[0059] Exemplarily, the text feature network can be one-hot encoding, bag-of-words model, word embedding layer, character-level encoding, N-gram encoding, and attention model, but is not limited to the above example scenarios and can be customized according to actual needs.

[0060] S140: Input the visual fusion features and the text encoding features into the content output network to obtain the output response text content corresponding to the multimodal data.

[0061] In a specific embodiment, the content output network includes a mapping layer, a splicing layer and a content output layer, and the visual fusion features and the text encoding features are input into the content output network to obtain the output response text content corresponding to the multimodal data, including: inputting the visual fusion features into the mapping layer to obtain the output visual mapping features, and splicing the visual mapping features and the text encoding features through the splicing layer to obtain the multimodal splicing features; inputting the multimodal splicing features into the content output layer to obtain the output response text content corresponding to the multimodal data.

[0062] In this embodiment, the visual mapping feature represents a feature vector in the text embedding space corresponding to the content output layer.

[0063] In a specific embodiment, the content output layer is a general domain language model. The general domain language model represents a machine learning model with a large parameter scale and complexity, or a neural network model with millions to billions of parameters. The general domain language model requires a large amount of computing resources and storage space, often using distributed computing and special hardware technology, and has stronger generalization and expression capabilities. The general domain language model can adapt to a variety of natural language processing tasks without the need to rebuild complex language rules for each specific task.

[0064] Exemplarily, the response text content includes, but is not limited to, description text content for the target image, reply text content related to the target image for the target prompt text, translation text content for the target image, image classification content for the target image, and the like.

[0065] Figure 2 A model architecture diagram of a multimodal language model provided for one embodiment of the present invention. Specifically, the multimodal language model includes a visual feature network, a text feature network and a content output network. The visual feature network includes a feature fusion module and n visual encoders, and the content output network includes a mapping layer, a splicing layer and a content output layer. The images are respectively input into the n visual encoders to obtain n output visual coding features, and the n visual coding features are input into the feature fusion module to obtain output visual fusion features. The prompt text is input into the text feature network to obtain output text coding features. The visual fusion features are input into the mapping layer to obtain output visual mapping features, the visual mapping features and the text coding features are input into the splicing layer to obtain output multimodal splicing features, and the multimodal splicing features are input into the content output layer to obtain output response text content corresponding to the multimodal data.

[0066] The technical solution of this embodiment solves the problem of poor expression ability of the multimodal language model for visual information, improves the response performance of the multimodal language model to complex image tasks, and improves the accuracy and richness of the response text content of the multimodal language model.

[0067] Figure 3 A flowchart of another content generation method provided by an embodiment of the present invention, this embodiment further refines the "visual feature network" in the above embodiment. In this embodiment, the visual feature network also includes a weight allocation module, and the feature fusion module is used to fuse the at least two visual coding features to obtain a visual fusion feature, including: inputting the at least two visual coding features and at least two visual coding weights into the feature fusion module to obtain an output visual fusion feature. Figure 3 As shown, the method includes:

[0068] S210: Input the multimodal data into the trained multimodal language model.

[0069] S210 in this embodiment is similar to that in the above embodiment. Figure 1 The S110 shown corresponds to the same or similar steps, and will not be described in detail in this embodiment.

[0070] S220: Input the target image and / or target prompt text into a weight distribution module to obtain a visual encoding weight corresponding to each visual encoder outputted.

[0071] Specifically, the target image is input into the weight distribution module to obtain the visual encoding weight corresponding to each output visual encoder; or, the target prompt text is input into the weight distribution module to obtain the visual encoding weight corresponding to each output visual encoder; or, the target image and the target prompt text are input into the weight distribution module to obtain the visual encoding weight corresponding to each output visual encoder.

[0072] Exemplarily, the weight allocation module may be a gated network, a factorization machine, an attention network, etc., but is not limited to the example scenario.

[0073] Exemplarily, the visual encoding weight corresponding to each visual encoder satisfies the following conditions:

[0074]

[0075] Where n represents the number of visual encoders in the visual feature network, g i Represents the visual encoding weight corresponding to the i-th visual encoder.

[0076] In a specific embodiment, the weight allocation module uses a Softmax function to normalize the visual encoding weights. For example, the visual encoding weight g corresponding to the i-th visual encoder is i Satisfies the following formula:

[0077]

[0078] Among them, z i and z j Represents the visual encoding weights before normalization corresponding to the i-th visual encoder and the j-th visual encoder respectively.

[0079] S230. Visually encode the target image using at least two visual encoders to obtain at least two visual coding features.

[0080] In a specific embodiment, visually encoding the target image by at least two visual encoders to obtain at least two visual coding features includes: visually encoding the target image by all visual encoders in the visual feature network to obtain at least two visual coding features, wherein the visual encoders correspond to the visual coding features one by one.

[0081] In another specific embodiment, the visual feature network also includes a visual screening module, and the method also includes: when the number of visual encoders is at least three, the visual coding weights corresponding to the at least three visual encoders are input into the visual screening module to obtain output of at least two visual encoders that match the multimodal data; accordingly, the target image is visually encoded by at least two visual encoders to obtain at least two visual coding features, including: the target image is visually encoded by at least two visual encoders that match the multimodal data to obtain at least two visual coding features.

[0082] In a specific embodiment, the visual screening module is used to use visual encoders whose visual encoding weights are greater than or equal to a weight threshold as at least two visual encoders that match the multimodal data.

[0083] Exemplarily, a visual encoder set consisting of at least two visual encoders matching the multimodal data is represented as S, and the visual screening module satisfies the following formula:

[0084]

[0085] Among them, threshold represents the weight threshold.

[0086] In another specific embodiment, the visual screening module is used to sort at least three visual encoders according to at least three visual encoding weights to obtain encoder sorting results, and determine at least two visual encoders that match the multimodal data according to preset screening conditions and the encoder sorting results.

[0087] Specifically, the preset screening condition is a preset screening ratio or a preset screening number. For example, the preset screening ratio may be 30%, and the preset screening number may be 3, but this is not limited to the example case.

[0088] The benefit of setting up a visual screening module is that, on the one hand, it enables the multimodal language model to adapt to different content generation tasks and avoids the noise interference of redundant visual encoding features on the response text content; on the other hand, it reduces the computational complexity of the multimodal language model and avoids the waste of visual encoding resources, thereby ensuring the operating efficiency and stability of the multimodal language model.

[0089] S240, inputting at least two visual coding features and at least two visual coding weights into a feature fusion module to obtain output visual fusion features.

[0090] In a specific embodiment, when the at least two visual coding features include visual coding features corresponding to all visual encoders in the visual feature network, the visual fusion feature represents the sum of the at least two visual coding features as an example, the visual fusion feature hfinal Satisfies the following formula:

[0091]

[0092] In another specific embodiment, when the at least two visual coding features include visual coding features respectively corresponding to at least two visual encoders matching the multimodal data in the visual feature network, taking the visual fusion feature as an example, the visual fusion feature h represents the sum of the at least two visual coding features. final Satisfies the following formula:

[0093]

[0094] Figure 4 A network architecture diagram of a visual feature network provided for an embodiment of the present invention. Specifically, the visual feature network includes a gating network, a visual screening module, n visual encoders and a feature fusion module, wherein the gating network is used to output the visual coding weights corresponding to each visual encoder according to the image in the multimodal data, and the visual screening module is used to screen out m visual encoders matching the multimodal data from the n visual encoders according to the n visual coding weights. The image is input into the m visual encoders matching the multimodal data to obtain m visual coding features as output, wherein n≥m≥2. The m visual coding features and the m visual coding weights are input into the feature fusion module to obtain the output visual fusion features.

[0095] S250, encoding the target prompt text through a text feature network to obtain text encoding features.

[0096] S260: Input the visual fusion features and the text encoding features into the content output network to obtain the output response text content corresponding to the multimodal data.

[0097] S250-S260 in this embodiment are similar to those in the above embodiment. Figure 1 The S130 - S140 shown correspond to the same or similar ones, and are not described in detail in this embodiment.

[0098] The technical solution of this embodiment, by inputting the target image and / or target prompt text into the weight allocation module, obtains the visual coding weight corresponding to each output visual encoder, and inputs at least two visual coding features and at least two visual coding weights into the feature fusion module to obtain the output visual fusion feature, which not only ensures the expression of important visual coding features in the visual fusion features, but also reduces the noise interference of unimportant visual coding features to the visual fusion features, so that the multimodal language model has the ability to adapt to different content generation tasks, and improves the generalization ability of the multimodal language model.

[0099] It should be noted that in the technical solution of the present invention, the collection, use, storage, sharing and transfer of user personal information involved are in compliance with the provisions of relevant laws and regulations, and it is necessary to inform the user and obtain the user's consent or authorization. When applicable, the user's personal information is de-identified and / or anonymized and / or encrypted.

[0100] The following is an embodiment of a content generating device provided in an embodiment of the present invention. The device and the content generating method of the above embodiment belong to the same inventive concept. For details not described in detail in the embodiment of the content generating device, please refer to the contents about the content generating method in the above embodiment.

[0101] Figure 5 FIG. 1 is a schematic diagram of a content generation device provided by an embodiment of the present invention. Figure 5 As shown, the device includes: a multimodal data input module 310, a visual fusion feature determination module 320, a text encoding feature determination module 330 and a response text content output module 340.

[0102] The multimodal data input module 310 is used to input the multimodal data into the trained multimodal language model; wherein the multimodal data includes a target image and a target prompt text, the multimodal language model includes a visual feature network, a text feature network and a content output network, and the visual feature network includes a feature fusion module and a visual encoder corresponding to at least two visual encoding characteristics;

[0103] A visual fusion feature determination module 320 is used to perform visual encoding on the target image through at least two visual encoders to obtain at least two visual encoding features, and to perform feature fusion on the at least two visual encoding features through a feature fusion module to obtain a visual fusion feature;

[0104] The text encoding feature determination module 330 is used to perform text encoding on the target prompt text through a text feature network to obtain text encoding features;

[0105] The response text content output module 340 is used to input the visual fusion features and the text encoding features into the content output network to obtain the output response text content corresponding to the multimodal data.

[0106] The technical solution of this embodiment solves the problem of poor expression ability of the multimodal language model for visual information, improves the response performance of the multimodal language model to complex image tasks, and improves the accuracy and richness of the response text content of the multimodal language model.

[0107] In a specific embodiment, the visual feature network further includes a weight allocation module, and the device further includes:

[0108] A visual encoding weight output module is used to input the target image and / or target prompt text into the weight distribution module to obtain the visual encoding weight corresponding to each visual encoder output;

[0109] The visual fusion feature determination module 320 includes:

[0110] The visual fusion feature output unit is used to input at least two visual coding features and at least two visual coding weights into the feature fusion module to obtain an output visual fusion feature.

[0111] In a specific embodiment, the visual feature network further includes a visual screening module, and the device further includes:

[0112] A visual encoder screening module, for inputting visual encoding weights corresponding to the at least three visual encoders into the visual screening module when the number of the visual encoders is at least three, to obtain at least two visual encoders matched with the multimodal data as output;

[0113] In a specific embodiment, the visual fusion feature determination module 320 includes:

[0114] The visual coding feature determination unit is used to obtain at least two visual coding features by visually encoding the target image through at least two visual encoders matched with the multimodal data.

[0115] In a specific embodiment, the visual encoder includes a hierarchical fusion module and at least two hierarchical encoding modules, wherein the hierarchical encoding module is used to encode the corresponding input data to obtain visual hierarchical features, and the hierarchical fusion module is used to fuse the at least two visual hierarchical features to output visual coding features;

[0116] When the hierarchical encoding module is the first hierarchical encoding module in the visual encoder, the input data is the target image. When the hierarchical encoding module is a hierarchical encoding module other than the first hierarchical encoding module, the input data is the visual hierarchical features output by the previous hierarchical encoding module connected in series with the hierarchical encoding module in the visual encoder.

[0117] In a specific embodiment, the hierarchical fusion module includes a hierarchical splicing unit and a dimensionality reduction unit. The hierarchical splicing unit is used to splice at least two visual hierarchical features to obtain a visual splicing feature, and the dimensionality reduction unit is used to perform dimensionality reduction processing on the visual splicing feature and output a visual coding feature.

[0118] In a specific embodiment, the content output network includes a mapping layer, a splicing layer and a content output layer, and responds to the text content output module 340, specifically for:

[0119] The visual fusion features are input into the mapping layer to obtain the output visual mapping features, and the visual mapping features and the text encoding features are spliced ​​through the splicing layer to obtain the multimodal splicing features;

[0120] Input the multimodal concatenation features into the content output layer to obtain the output response text content corresponding to the multimodal data;

[0121] Among them, the visual mapping feature represents the feature vector in the text embedding space corresponding to the content output layer.

[0122] The content generating device provided in the embodiment of the present invention can execute the content generating method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0123] Figure 6 A schematic diagram of the structure of an electronic device provided for one embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0124] like Figure 6As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor 11, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the ROM 12 or the computer program loaded from the storage unit 18 to the RAM 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0125] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information or data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0126] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the content generation method provided in the above embodiments.

[0127] In some embodiments, the content generation method provided in the above embodiments may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps in the content generation method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the content generation method in any other appropriate manner (e.g., by means of firmware).

[0128] Various embodiments of the systems and techniques described above herein may be implemented in the following systems or combinations thereof: digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] The computer programs for implementing the content generation method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0130] In the context of the present application, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device or equipment. A computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable storage medium. Examples of machine readable storage media may include electrical connections based on at least one line, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM), flash memories, optical fibers, portable compact disc read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein may be implemented on a terminal device having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the terminal device. Other types of devices may also provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a Local Area Network (LAN), a Wide Area Network (WAN), a blockchain network, and the Internet.

[0133] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0134] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0135] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A content generation method, characterized in that: include: Inputting multimodal data into a trained multimodal language model; wherein the multimodal data includes a target image and a target prompt text, the multimodal language model includes a visual feature network, a text feature network and a content output network, and the visual feature network includes a feature fusion module and a visual encoder corresponding to at least two visual encoding characteristics; By means of at least two visual encoders, the target image is visually encoded respectively to obtain at least two visual encoding features, and by means of the feature fusion module, the at least two visual encoding features are feature-fused to obtain a visual fusion feature; By means of the text feature network, the target prompt text is encoded to obtain text encoding features; The visual fusion feature and the text encoding feature are input into the content output network to obtain output response text content corresponding to the multimodal data.

2. The method according to claim 1, characterized in that The visual feature network also includes a weight allocation module, and the method further includes: Inputting the target image and / or the target prompt text into the weight allocation module to obtain the visual encoding weight corresponding to each output visual encoder; Correspondingly, the feature fusion module is used to perform feature fusion on the at least two visual coding features to obtain a visual fusion feature, including: The at least two visual coding features and the at least two visual coding weights are input into the feature fusion module to obtain output visual fusion features.

3. The method according to claim 2, characterized in that The visual feature network also includes a visual screening module, and the method further includes: When the number of the visual encoders is at least three, the visual encoding weights corresponding to the at least three visual encoders are input into the visual screening module to obtain at least two visual encoders that match the multimodal data as output; Correspondingly, the target image is visually encoded by at least two visual encoders to obtain at least two visual encoding features, including: The target image is visually encoded by at least two visual encoders matched with the multimodal data to obtain at least two visual encoding features.

4. The method according to claim 1, characterized in that: The visual encoder includes a hierarchical fusion module and at least two hierarchical encoding modules, wherein the hierarchical encoding module is used to encode the corresponding input data to obtain visual hierarchical features, and the hierarchical fusion module is used to fuse the at least two visual hierarchical features to output visual coding features; When the hierarchical encoding module is the first hierarchical encoding module in the visual encoder, the input data is the target image; when the hierarchical encoding module is a hierarchical encoding module other than the first hierarchical encoding module, the input data is the visual hierarchical features output by the previous hierarchical encoding module connected in series with the hierarchical encoding module in the visual encoder.

5. The method according to claim 4, characterized in that The hierarchical fusion module includes a hierarchical splicing unit and a dimensionality reduction unit. The hierarchical splicing unit is used to splice at least two visual hierarchical features to obtain a visual splicing feature. The dimensionality reduction unit is used to perform dimensionality reduction processing on the visual splicing feature and output a visual coding feature.

6. The method according to any one of claims 1 to 5, characterized in that: The content output network includes a mapping layer, a splicing layer and a content output layer, and the visual fusion feature and the text encoding feature are input into the content output network to obtain the output response text content corresponding to the multimodal data, including: Inputting the visual fusion feature into the mapping layer to obtain an output visual mapping feature, and performing splicing processing on the visual mapping feature and the text encoding feature through the splicing layer to obtain a multimodal splicing feature; Inputting the multimodal splicing features into the content output layer to obtain outputted response text content corresponding to the multimodal data; The visual mapping feature represents a feature vector in the text embedding space corresponding to the content output layer.

7. A content generating device, characterized in that: include: A multimodal data input module, used to input multimodal data into a trained multimodal language model; wherein the multimodal data includes a target image and a target prompt text, the multimodal language model includes a visual feature network, a text feature network and a content output network, and the visual feature network includes a feature fusion module and a visual encoder corresponding to at least two visual encoding characteristics; A visual fusion feature determination module is used to perform visual encoding on the target image through at least two visual encoders to obtain at least two visual encoding features, and to perform feature fusion on the at least two visual encoding features through the feature fusion module to obtain a visual fusion feature; A text encoding feature determination module, used to perform text encoding on the target prompt text through the text feature network to obtain text encoding features; The response text content output module is used to input the visual fusion feature and the text encoding feature into the content output network to obtain the output response text content corresponding to the multimodal data.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the content generation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the content generation method according to any one of claims 1 to 6 when executed.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the content generation method according to any one of claims 1 to 6.

Images