Text data generation method and apparatus, electronic device, and medium

Abstract

This disclosure relates to the field of text data generation technology, providing a text data generation method, apparatus, electronic device, and medium. The method involves semantic processing of image data using a multi-granularity image encoder to obtain semantic features; processing these semantic features using a first multilayer perceptron to obtain first semantic features; processing prompt text data using a first and second text encoder to obtain second and third semantic features; concatenating the first, second, and third semantic features to obtain target semantic data for the image data; and processing the target semantic data using a large language model to obtain feedback text data for the image data. This disclosure significantly improves the semantic understanding capability between images and text, solves several key problems in the prior art, and provides strong support for the development of multimodal artificial general intelligence systems.

Description

Technical Field

[0001] This disclosure relates to the field of text data generation technology, and in particular to a text data generation method, apparatus, electronic device and medium. Background Technology

[0002] With the rapid development of internet technology, multimodal large language models have demonstrated enormous potential in image and text recognition and understanding. These models can provide richer semantic understanding through the fusion analysis of images and text, thereby driving progress in the fields of computer vision and natural language processing.

[0003] However, existing multimodal large language models face several significant drawbacks in their evolution towards artificial general intelligence systems. First, while large language models have made significant progress in parameter scale and performance, existing multimodal models have failed to keep pace with the development of visual and visual-language foundational models. This results in a significant gap between the parameter scale and feature representation capabilities of visual encoders and large language models, limiting the overall performance of multimodal models. Second, current multimodal large language models typically employ lightweight "glue" layers to align features between visual and language models. However, this approach suffers from numerous problems regarding differences in parameter scales, consistency of representations, and connection efficiency, affecting the overall performance and application effectiveness of the model. Furthermore, the diversity and complexity of images in the real world pose a severe challenge to the robustness of multimodal large language models. Existing models exhibit limitations when processing low-resolution and multi-scale images, struggling to fully capture the details and rich information within images, thus affecting the accuracy and depth of semantic understanding. Finally, existing multimodal large language models lack the ability to effectively integrate target-level features and process multi-dimensional features. This deficiency limits the model's performance in complex visual scenes, reducing its depth of understanding and scope of application. Therefore, there is an urgent need to improve multimodal large language models to enhance their performance in visual and language interactions, thereby promoting the further development of artificial general intelligence systems. Summary of the Invention

[0004] In view of this, the present disclosure provides a text data generation method, apparatus, electronic device, and computer-readable storage medium to solve the technical problem of low accuracy in multimodal large model recognition of images and text in the prior art.

[0005] A first aspect of this disclosure provides a text data generation method, comprising: acquiring image data and prompt text data; performing semantic processing on the image data using a multi-granularity image encoder to obtain semantic features of the image data, wherein the multi-granularity image encoder is trained based on historical image data and real semantic text data of historical image data; processing the semantic features of the image data using a first multilayer perceptron to obtain a first semantic feature; encoding the prompt text data using a first text encoder and processing the result of the first text encoder using a second multilayer perceptron to obtain a second semantic feature, wherein the first text encoder is trained based on real semantic text data of historical image data and the output result of the multi-granularity image encoder; encoding the prompt text data using a second text encoder and processing the result of the second text encoder using a third multilayer perceptron to obtain a third semantic feature, wherein the second text encoder is trained based on historical image data and historical prompt text data; concatenating the first semantic feature, the second semantic feature, and the third semantic feature to obtain target semantic data of the image data; and processing the target semantic data of the image data using a large language model to obtain feedback text data of the image data.

[0006] A second aspect of this disclosure provides a text data generation apparatus, comprising: an acquisition module for acquiring image data and prompt text data; a first processing module for performing semantic processing on the image data using a multi-granularity image encoder to obtain semantic features of the image data, wherein the multi-granularity image encoder is trained based on historical image data and real semantic text data of the historical image data; a second processing module for processing the semantic features of the image data using a first multilayer perceptron to obtain first semantic features; and a third processing module for encoding the prompt text data using a first text encoder and processing the result of the first text encoder using a second multilayer perceptron to obtain second semantic features. The first text encoder is trained based on real semantic text data from historical image data and the output of a multi-granularity image encoder. The fourth processing module is used to encode the prompt text data through the second text encoder and process it through the third multilayer perceptron multi-encoder to obtain the third semantic feature. The second text encoder is trained based on historical image data and historical prompt text data. The concatenation module is used to concatenate the first semantic feature, the second semantic feature, and the third semantic feature to obtain the target semantic data of the image data. The text generation module is used to process the target semantic data of the image data through a large language model to obtain the feedback text data of the image data.

[0007] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0008] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0009] The beneficial effects of this disclosure compared to existing technologies are as follows: This disclosure can perform semantic processing on image data through a multi-granularity image encoder, enabling the extraction of richer and more detailed semantic features. This training method based on historical image data and real semantic text data allows the model to better understand image content, thereby improving the depth of understanding of complex visual scenes. By combining the outputs of the first text encoder and the multi-granularity image encoder, the differences between different feature representations can be effectively reduced, improving the alignment effect of image and text features. This consistency helps the model provide more accurate semantic feedback when processing multimodal data. By concatenating the first, second, and third semantic features, effective integration of features of different dimensions is achieved. This multi-level feature fusion can more comprehensively reflect the target semantics of image data, improving the model's performance in complex tasks. When processing low-resolution and multi-scale images, this invention enhances the model's robustness to image diversity and complexity through a multi-granularity image encoder. This allows the model to better adapt to image data under different scenarios and conditions in practical applications. By processing the target semantic data of image data through a large language model, more accurate and relevant feedback text data can be generated. This high-quality text feedback not only enhances the user experience but also provides a more reliable information foundation for subsequent applications. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of embodiments of the present invention can be applied is shown;

[0012] Figure 2 This is a flowchart illustrating a text data generation method provided in an embodiment of this disclosure;

[0013] Figure 3 This is a flowchart illustrating another text data generation method provided in this embodiment of the disclosure;

[0014] Figure 4 This is a flowchart illustrating another text data generation method provided in this disclosure embodiment;

[0015] Figure 5 This is a flowchart illustrating another text data generation method provided in this disclosure embodiment;

[0016] Figure 6 This is a flowchart illustrating another text data generation method provided in this disclosure embodiment;

[0017] Figure 7 This is a block diagram of a text data generation apparatus provided in an embodiment of this disclosure;

[0018] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0019] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0020] It should be noted that the user information (including but not limited to terminal device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0021] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of embodiments of the present invention can be applied is shown.

[0022] like Figure 1 As shown, the system architecture 100 may include one or more of a first terminal device 101, a second terminal device 102, and a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0023] It should be understood that Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, server 105 could be a server cluster composed of multiple servers.

[0024] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send data, etc. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays, including but not limited to smartphones, tablets, laptops, and desktop computers, etc.

[0025] Server 105 can be a server that provides various services. For example, server 105 can acquire image data and prompt text data from first terminal device 103 (or second terminal device 102 or third terminal device 103); perform semantic processing on the image data using a multi-granularity image encoder to obtain semantic features of the image data. The multi-granularity image encoder is trained based on historical image data and real semantic text data of historical image data; process the semantic features of the image data using a first multilayer perceptron to obtain a first semantic feature; encode the prompt text data using a first text encoder, and process the result of the first text encoder using a second multilayer perceptron to obtain a second semantic feature. The first text encoder is trained based on real semantic text data of historical image data and the output result of the multi-granularity image encoder; encode the prompt text data using a second text encoder, and process the result of the second text encoder using a third multilayer perceptron to obtain a third semantic feature. The second text encoder is trained based on historical image data and historical prompt text data; concatenate the first, second, and third semantic features to obtain the target semantic data of the image data; process the target semantic data of the image data using a large language model to obtain the feedback text data of the image data.

[0026] In some embodiments, the text data generation method provided in this invention is generally executed by server 105, and correspondingly, the text data generation device is generally located in server 105. In other embodiments, certain terminal devices may have functions similar to those of a server to execute this method. Therefore, the text data generation method provided in this invention is not limited to execution on the server side.

[0027] The text data generation method and apparatus according to embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0028] Figure 2This is a flowchart illustrating a text data generation method provided in an embodiment of this disclosure. The method provided in this embodiment can be executed by any electronic device with computer processing capabilities; for example, the electronic device may be... Figure 1 The server shown.

[0029] like Figure 2 As shown, the text data generation method includes steps S210 to S270.

[0030] In step S210, image data and prompt text data are acquired.

[0031] Step S220: Semantic processing of image data is performed by a multi-granularity image encoder to obtain semantic features of the image data. The multi-granularity image encoder is trained based on historical image data and real semantic text data of historical image data.

[0032] In step S230, the semantic features of the image data are processed by the first multilayer perceptron to obtain the first semantic features.

[0033] In step S240, the prompt text data is encoded by the first text encoder, and the result of the first text encoder is processed by the second multilayer perceptron to obtain the second semantic feature. The first text encoder is trained based on the real semantic text data of historical image data and the output of the multi-granularity image encoder.

[0034] In step S250, the prompt text data is encoded by the second text encoder, and the result of the second text encoder is processed by the third multilayer perceptron to obtain the third semantic feature. The second text encoder is trained based on historical image data and historical prompt text data.

[0035] In step S260, the first semantic feature, the second semantic feature, and the third semantic feature are concatenated to obtain the target semantic data of the image data.

[0036] In step S270, the target semantic data of the image data is processed by a large language model to obtain the feedback text data of the image data.

[0037] This method utilizes a multi-granularity image encoder to perform semantic processing on image data, enabling the extraction of richer and more detailed semantic features. This training approach, based on historical image data and real semantic text data, allows the model to better understand image content, thereby enhancing its understanding of complex visual scenes. By combining the outputs of the first text encoder with the multi-granularity image encoder, the differences between different feature representations are effectively reduced, improving the alignment between image and text features. This consistency helps the model provide more accurate semantic feedback when processing multimodal data. Through the concatenation of the first, second, and third semantic features, effective integration of features from different dimensions is achieved. This multi-level feature fusion more comprehensively reflects the target semantics of image data, improving the model's performance in complex tasks. When processing low-resolution and multi-scale images, this invention enhances the model's robustness to image diversity and complexity through a multi-granularity image encoder. This allows the model to better adapt to image data under different scenarios and conditions in practical applications. Processing the target semantic data of image data using a large language model generates more accurate and relevant feedback text data. This high-quality text feedback not only improves the user experience but also provides a more reliable information foundation for subsequent applications.

[0038] In some embodiments of this disclosure, the image data can be obtained from multiple image databases, including public datasets, commercial image libraries, social media platforms, etc. These data sources typically contain a large number of labeled and categorized images, suitable for various application scenarios. For example, image data can be automatically extracted from selected databases by writing web crawlers or API interfaces. When actively acquiring images, filtering can be performed based on specific keywords, tags, or conditions to ensure that the acquired images are highly relevant to user needs or task objectives. For example, images can be categorized based on their theme, style, or content type to improve the efficiency and accuracy of subsequent processing. In this embodiment, users can also upload image data through the user interface of a terminal device, allowing users to upload images through various terminal devices (such as mobile phones, tablets, computers, etc.). Users can upload images by simply dragging and dropping, clicking, or selecting files, supporting multiple image formats (such as JPEG, PNG, GIF, etc.) to meet the needs of different users. After a user uploads an image, real-time feedback can be provided, such as image preview and processing progress, improving the user experience. In addition, uploaded images can immediately enter the subsequent semantic processing flow to ensure efficiency. When users upload images, it is necessary to ensure user privacy and data security. This can be achieved through encrypted transmission and data anonymization, ensuring that user-uploaded images are not accessed by unauthorized third parties. These two methods enable flexible acquisition of image data, allowing for both proactive extraction of relevant images from rich databases and the acquisition of personalized data through user uploads. This diversified image acquisition strategy not only enhances the richness and diversity of the data but also provides a solid data foundation for subsequent multimodal processing.

[0039] In some embodiments of this disclosure, a prompt template library containing various themes and scenarios can be established. These templates can be designed based on historical data, user needs, or specific application scenarios, covering different types of images and content. After image data is acquired, the most suitable prompt template (i.e., the aforementioned prompt text data) can be automatically selected based on the image's content, features, or classification information. This automated selection process can be achieved through pre-set rules or machine learning algorithms to ensure that the selected template is highly semantically relevant to the image. Users can also customize existing prompt templates in the system to meet specific needs. This flexibility allows users to generate personalized prompt text according to different application scenarios or project requirements. In this embodiment, users can also input prompt descriptions through the user interface of the terminal device, for example, allowing users to input custom prompt descriptions while viewing image data. Users can quickly express their understanding or needs about the image through text boxes or voice input. Real-time feedback and suggestions can be provided when users input prompt descriptions, such as automatically completing or recommending relevant descriptions based on the keywords entered by the user. This intelligent assistance function can improve the user's input efficiency and help the user express their intentions more accurately. The system supports multiple input methods, including text, voice, and image annotation, allowing users to choose the most convenient method for their prompts. This diverse input approach not only enhances the user experience but also provides the system with richer contextual information. Through these two methods, prompt text data can be flexibly acquired. On one hand, designed and optimized prompt templates can be retrieved from the database to ensure efficiency and consistency; on the other hand, users can also input personalized prompt descriptions based on specific images, enhancing the system's adaptability and flexibility. This diversified prompt text acquisition strategy provides rich contextual information for subsequent multimodal processing, improving the overall performance of the system.

[0040] In some embodiments of this disclosure, the above-described multi-granularity image encoder can be used to extract semantic features from image data. This multi-granularity image encoder includes a Contrastive Language-Image Pre-trained Image Encoder.

[0041] Image Pre-training (CLIP), first convolutional layer, Vision Transformer (VIT), second convolutional layer, open object detector, average pooling layer, cross attention layer, fourth multilayer perceptron, and fifth multilayer perceptron. The following will demonstrate... Figure 3 and Figure 4 The training process of this multi-granularity image encoder is described as follows:

[0042] Figure 3This is a flowchart illustrating another text data generation method provided in this disclosure. Figure 3 As shown, before performing semantic processing on the image data through a multi-granularity image encoder, the above method further includes steps S310 to S350.

[0043] In step S310, historical image data is acquired and input into a multi-granularity image encoder. The multi-granularity image encoder includes a contrastive language-image pre-trained image encoder, a first convolutional layer, a visual image encoder, a second convolutional layer, an open object detector, and an average pooling layer.

[0044] In step S320, the historical image data is encoded by a contrastive language-image pre-trained image encoder, and the result of the contrastive language-image pre-trained image encoder is convolved by a first convolutional layer to obtain the first convolutional semantic features of the historical image.

[0045] In step S330, the historical image data is encoded by a visual image encoder, and the output of the visual image encoder is convolved by a second convolutional layer to obtain the second convolutional semantic features of the historical image.

[0046] In step S340, the historical image data is detected and processed by an open object detector, and the output of the open object detector is pooled by an average pooling layer to obtain at least one pooled semantic feature of the historical image.

[0047] In step S350, the target semantic features of the historical image are determined based on the first convolutional semantic features of the historical image, the second convolutional semantic features of the historical image, and at least one pooling semantic feature of the historical image.

[0048] This method encodes historical image data using a contrastive language-image pre-trained image encoder, and then convolves the result of the pre-trained image encoder through a first convolutional layer to obtain the first convolutional semantic features of the historical image. Next, it encodes the historical image data using a visual image encoder, and then convolves the output of the visual image encoder through a second convolutional layer to obtain the second convolutional semantic features of the historical image. Finally, it detects the historical image data using an open object detector, and then pools the output of the open object detector through an average pooling layer to obtain at least one pooled semantic feature of the historical image. Based on the first convolutional semantic features, the second convolutional semantic features, and at least one pooled semantic feature of the historical image, the target semantic features of the historical image are determined. This approach ensures the comprehensiveness and accuracy of feature extraction, laying a solid foundation for further analysis and application.

[0049] In some embodiments of this disclosure, historical image data is acquired and input into a multi-granularity image encoder. For example, historical image data can come from existing image databases, such as public datasets (e.g., ImageNet, COCO) or specific historical image archives. Relevant historical images are automatically collected from the internet using web crawling techniques. Users are allowed to upload their historical images, forming a dynamic dataset. The data can be image files in different formats, such as JPEG, PNG, TIFF, etc., containing various historical scenes, people, objects, etc. The required number of images is determined to ensure the effectiveness of model training and testing. The size of the data may affect the model's performance and generalization ability. Before inputting the historical image data into the multi-granularity image encoder, several preprocessing steps are typically required to ensure data quality and consistency: image processing techniques are used to remove noise from the images to improve image quality; image integrity is checked, and damaged or blurry images are removed; all images are resized to a uniform size to ensure shape consistency when input into the encoder; and color space conversion (e.g., RGB to grayscale) or color normalization is performed to reduce color differences between different images. Image enhancement techniques such as rotation, flipping, scaling, and cropping increase the diversity of training data and improve model robustness. The preprocessed image data is then converted to a format suitable for encoder input. Typically, image data needs to be converted to tensor format for processing in deep learning frameworks (such as TensorFlow or PyTorch). Image data is also divided into multiple batches for sequential input into the encoder. This approach improves computational efficiency and reduces memory usage.

[0050] Each batch of image data is passed to a multi-granularity image encoder. The encoder's input layer receives this image data and begins the feature extraction process.

[0051] In some embodiments of this disclosure, historical image data is encoded using a contrastive language-image pre-trained image encoder, and the result of the contrastive language-image pre-trained image encoder is convolved by a first convolutional layer to obtain the first convolutional semantic features of the historical image. In this embodiment, the first convolutional layer can be a one-dimensional convolutional layer. For example, a contrastive language-image pre-trained image encoder (such as CLIP) typically consists of two parts: an image encoder and a text encoder. The image encoder is responsible for extracting image features, and the text encoder is responsible for extracting text features associated with the image. The image is feature extracted using a convolutional neural network (CNN) or a transformer architecture. This model associates the image with the corresponding text through contrastive learning, thereby generating a semantic feature representation of the image. The output is a high-dimensional feature vector representing the semantic information of the image, called the "contrastive encoded feature". The output of the contrastive language-image pre-trained image encoder (contrastive encoded feature) is input into the first convolutional layer. In this embodiment, the first convolutional layer is configured as a one-dimensional convolutional layer. While one-dimensional convolutional layers are typically used to process sequential data (such as time series or text), in this context, they can also be used to process feature vectors, especially when the features are sequential. One-dimensional convolutional layers use multiple one-dimensional convolutional kernels (filters), the size and number of which can be adjusted according to the specific task. The convolutional kernel slides across the feature vector, extracting local features. It generates new feature maps by dot product and summation. Each kernel produces a feature map representing the activation of a specific pattern or feature. Typically, a non-linear activation function (such as ReLU) is applied after the convolution operation to introduce non-linearity, enabling the model to learn more complex features. The output is a new feature representation called the "first convolutional semantic feature." This feature represents a combination of local and global features of the input image, aiding in subsequent analysis and processing. The first convolutional semantic feature contains low-level and mid-level feature information of the image, such as edges, textures, and shapes. These features form the basis for further analysis and classification by subsequent deep networks. The first convolutional semantic features can be passed as input to subsequent convolutional layers or other network structures to further extract higher-level semantic features, ultimately used for tasks such as image classification and object detection. By encoding historical image data using a contrastive language-image pre-trained image encoder and then convolving the encoded results using a one-dimensional convolutional layer, the first convolutional semantic features of historical images can be effectively extracted. This process combines image feature extraction and convolution operations, providing rich semantic information for subsequent analysis and applications.

[0052] In some embodiments of this disclosure, historical image data is encoded using a visual image encoder, and the output of the visual image encoder is convolved by a second convolutional layer to obtain the second convolutional semantic features of the historical image. In this embodiment, the second convolutional layer is a one-dimensional convolutional layer. A visual image encoder is typically a deep convolutional neural network (CNN) specifically designed for extracting image features. Common architectures include ResNet, VGG, and Inception. The image is processed through multiple convolutional layers, pooling layers, and activation functions, extracting low-level features (such as edges and textures) and mid-level features (such as shape and pattern) layer by layer. The output is a high-dimensional feature vector representing the semantic information of the image, called "visually encoded features." These features provide rich information for subsequent processing. The features output by the visual image encoder (visually encoded features) are input into the second convolutional layer. The second convolutional layer is designed as a one-dimensional convolutional layer (1D ConvolutionLayer). While one-dimensional convolutional layers are typically used to process sequential data, they can be used here to process feature vectors, especially when the features are sequential. One-dimensional convolutional layers use multiple one-dimensional convolutional kernels (filters), the size and number of which can be adjusted according to the specific task. The convolutional kernels slide across the feature vectors, extracting local features. They generate new feature maps by dot product and summation. Each kernel produces a feature map representing the activation of a specific pattern or feature. Typically, a non-linear activation function (such as ReLU) is applied after the convolution operation to introduce non-linearity, enabling the model to learn more complex features. The output is a new feature representation called a "second convolutional semantic feature." This feature represents a higher-level semantic information of the input image, combining features extracted from the visual image encoder. The second convolutional semantic feature contains richer semantic information, capturing complex patterns and relationships in the image. These features form the basis for further analysis and classification in subsequent deep networks. The second convolutional semantic features can be passed as input to subsequent convolutional layers or other network structures to further extract higher-level semantic features, ultimately used for tasks such as image classification and object detection. By encoding historical image data using a visual image encoder and then convolving the encoded results using a one-dimensional convolutional layer, the second convolutional semantic features of historical images can be effectively extracted. This process combines image feature extraction and convolution operations, providing deeper semantic information for subsequent analysis and applications.

[0053] In some embodiments of this disclosure, historical image data is processed by an open object detector, and the output of the open object detector is pooled by an average pooling layer to obtain at least one pooled semantic feature of the historical image. In this embodiment, the open object detector identifies at least one object bounding box in the image and extracts features. For example, open object detectors are typically based on deep learning models such as YOLO (You Only LookOnce), Faster R-CNN, or SSD (Single Shot MultiBox Detector). These models are specifically designed to detect objects in images. The open object detector processes the input image, identifies multiple objects in the image, and generates a bounding box for each object. The bounding box is typically represented by four coordinates: the coordinates of the top-left corner and the bottom-right corner. Within each identified bounding box, the open object detector extracts corresponding features. These features can be appearance features, shape features, etc., of the object, and are typically extracted through convolutional layers. The output includes multiple bounding boxes and their corresponding feature vectors, representing the detected objects in the image and their related information. The features output by the open object detector (i.e., the feature vectors within each bounding box) are input into an average pooling layer. Average pooling layers downsample input features, reducing their dimensionality while preserving important semantic information. The average pooling layer slides across the input features, generating new feature maps by calculating the average value within each window. The average value of each window represents the feature information of that region. After pooling, the dimensionality of the output feature map is typically reduced, thus lowering computational complexity while retaining important semantic information. The resulting pooled semantic features represent the comprehensive information of objects detected in historical images. Through average pooling, noise and redundant information in the feature map are suppressed, highlighting important semantic features. These pooled semantic features can be passed as input to subsequent multilayer perceptrons. By using an open object detector to process historical image data and then pooling the detection results using an average pooling layer, at least one pooled semantic feature can be effectively extracted from the historical image. This process combines object detection and feature pooling, providing important semantic information for subsequent analysis and applications.

[0054] In some embodiments of this disclosure, the multi-granularity image encoder further includes a cross-attention layer, a fourth multilayer perceptron, and a fifth multilayer perceptron. Determining the target semantic features of a historical image based on a first convolutional semantic feature, a second convolutional semantic feature, and at least one pooling semantic feature of the historical image includes: processing the first convolutional semantic feature through the cross-attention layer and processing the output of the cross-attention layer through the fourth multilayer perceptron to obtain a first historical target semantic feature; processing the second convolutional semantic feature through the cross-attention layer and processing the output of the cross-attention layer through the fourth multilayer perceptron to obtain a second historical target semantic feature; concatenating at least one pooling semantic feature and processing the concatenation result through the fifth multilayer perceptron to obtain a third historical target semantic feature; and concatenating the first historical target semantic feature, the first historical target semantic feature, and the first historical target semantic feature to obtain the target semantic features of the historical image. For example, the first convolutional semantic feature (from the first convolutional layer of the historical image). The first convolutional semantic feature is processed through the cross-attention layer. The cross-attention mechanism calculates the correlation between this feature and other features, enabling the model to focus on more important information and suppress irrelevant features. The output of the cross-attention layer is input into the fourth multilayer perceptron. This MLP transforms the features through multiple fully connected layers and activation functions to extract the first historical target semantic feature. The second convolutional semantic feature (from the second convolutional layer of the historical image) is then extracted. Similar to the generation of the first historical target semantic feature, the second convolutional semantic feature is processed through the cross-attention layer to enhance the relationship between features. The output of the cross-attention layer is input into the fourth multilayer perceptron to extract the second historical target semantic feature. At least one pooling semantic feature (from the pooling layer of the historical image) is then extracted. At least one pooling semantic feature is concatenated to form a new feature representation. This operation integrates features from multiple object bounding boxes, increasing the richness of information. The concatenated result is input into the fifth multilayer perceptron to further extract the third historical target semantic feature. The first, second, and third historical target semantic features are concatenated to form a comprehensive feature representation. Through the concatenation operation, the model integrates features from different sources to form the target semantic feature of the historical image. This feature integrates multi-granular information and can more comprehensively reflect the semantic content of the image. By combining the cross-attention layer and multilayer perceptron in the multi-granularity image encoder, target semantic features of historical images can be effectively extracted. This process enhances the model's ability to understand historical images through the processing and integration of different convolutional and pooling features, providing rich semantic information for subsequent tasks (such as classification and retrieval).

[0055] In some embodiments of this invention, semantic processing of image data using a multi-granularity image encoder to obtain semantic features of the image data includes: encoding the image data using a contrastive language-image pre-trained image encoder, and convolving the result of the contrastive language-image pre-trained image encoder through a first convolutional layer to obtain a first convolutional semantic feature of the image; encoding the image data using a visual image encoder, and convolving the output of the visual image encoder through a second convolutional layer to obtain a second convolutional semantic feature of the image; detecting the image data using an open object detector, and pooling the output of the open object detector through an average pooling layer to obtain at least one pooled semantic feature of the image; and determining the target semantic feature of the image based on the first convolutional semantic feature, the second convolutional semantic feature, and at least one pooled semantic feature. The processing flow of this example is similar to the process of training a multi-granularity image encoder described above and will not be repeated here.

[0056] Figure 4 This is a flowchart illustrating another text data generation method provided in this disclosure. Figure 4 As shown, before encoding the prompt text data by the first text encoder, the above method further includes steps S410 to S440.

[0057] In step S410, the real semantic text data of the historical image data is obtained.

[0058] In step S420, the real semantic text data of the historical image data is encoded by the first text encoder to obtain the semantic features of the real semantic text data.

[0059] In step S430, the contrastive loss is calculated based on the target semantic features of historical images and the semantic features of real semantic text data using a contrastive loss function.

[0060] In step S440, the parameters in the multi-granularity image encoder and the parameters in the first text encoder are updated based on the contrast loss.

[0061] This method obtains semantic features of the text by acquiring real semantic text data from historical image data and encoding it using a first text encoder. The similarity between the target semantic features of the historical image and the semantic features of the real semantic text data is calculated using a comparative loss function, thereby updating the parameters of the multi-granularity image encoder and the first text encoder. This process helps optimize the model's correlation between images and text, enhancing its performance in multimodal learning tasks.

[0062] In some embodiments of this disclosure, real semantic text data from historical image data is obtained. For example, descriptions or tags manually provided for each image typically include information such as objects, scenes, and actions within the image. Speech-to-text data extracted from videos, especially image sequences within videos, can also provide contextual information. Specifically, methods for obtaining real semantic text data include: using web crawlers to scrape images and their accompanying text information from websites (such as Flickr and Pinterest). This requires compliance with the website's terms of use and conditions. Utilizing APIs provided by social media platforms or datasets (such as the Twitter API and Flickr API) to obtain relevant image and text data. The collected text data typically requires preprocessing to improve its quality and usability. The obtained image and text data is stored in a database for easy access and processing. Commonly used databases include SQL and NoSQL databases. Appropriate data formats (such as JSON and CSV) are chosen to store the correspondence between images and text, ensuring data readability and usability.

[0063] In some embodiments of this disclosure, a first text encoder encodes the real semantic text data of historical image data to obtain semantic features of the real semantic text data. For example, the first text encoder is the core component for converting text data into semantic feature representations. Common text encoders include: RNNs (Recurrent Neural Networks), suitable for processing sequential data and capable of capturing contextual relationships in text; LSTMs (Long Short-Term Memory Networks), which better capture long-distance dependencies and are suitable for complex text data; GRUs (Gated Recurrent Units), a simplified version of LSTM with similar performance but faster training speeds; Transformers, which process text data through a self-attention mechanism, can process input sequences in parallel, are suitable for encoding long texts, and are commonly used in modern natural language processing tasks; and BERT (Bidirectional Encoder-Representation Transformer), based on the Transformer architecture, can capture contextual information and is suitable for various text understanding tasks. Before inputting the real semantic text data into the encoder, some preprocessing steps are required: removing irrelevant characters, punctuation marks, and noise to ensure the purity of the input; and decomposing the text into words or sub-words, typically using word segmentation tools (such as BERT's WordPiece segmenter). The segmented text is converted into a numerical representation, typically using word embeddings (such as Word2Vec and GloVe) or encoder-specific embedding layers (such as BERT's input embeddings). For variable-length input sequences, padding is applied to ensure consistent input shape, and masking techniques are used when necessary to ensure the model focuses only on the valid input portion. The processed text data is then fed into a first text encoder for encoding. After passing through the embedding layer, the text data propagates forward through each layer of the encoder. For RNNs and LSTMs, data is passed progressively; for Transformers, data is processed in parallel using a self-attention mechanism. Relationships between words in the input are calculated to generate context-dependent representations and capture long-distance dependencies. During encoding, RNNs and LSTMs generate hidden states, and Transformers generate contextual representations for each word, ultimately outputting a fixed-size vector representing the semantic features of the entire text. After encoder processing, the output features are the semantic representation of the text. The output feature vector is typically a high-dimensional floating-point vector that effectively represents the semantic information of the input text. In some cases, pooling layers (such as max pooling or average pooling) can be used to convert the sequence features into a fixed-length vector for subsequent processing. These semantic features can be used for downstream tasks such as classification, retrieval, or comparison with image features. Encoding the real semantic text data of historical image data through a first text encoder can convert the text into a high-dimensional semantic feature representation. This process involves selecting a suitable encoder, preprocessing the input text, performing forward propagation through the encoder, and generating the final semantic feature vector.These features are crucial in multimodal learning, helping to improve the model's ability to understand the relationship between images and text.

[0064] In some embodiments of this disclosure, a contrastive loss is calculated based on the target semantic features of historical images and the semantic features of real semantic text data using a contrastive loss function. The parameters in the multi-granularity image encoder and the first text encoder are updated based on the contrastive loss. For example, text features obtained from the first text encoder and image features obtained from the multi-dimensional image encoder are compared and learned, using a feature contrastive loss (such as Contrastive Loss) to ensure that similar images and texts have similar feature representations. During training, the parameters of the CLIP, VIT encoder, open object detector, and first text encoder are frozen, and only two one-dimensional convolutional layers and two MLP layers (i.e., the fourth and fifth multilayer perceptrons) are trained to learn image-text aligned feature representations while optimizing image feature fusion.

[0065] Figure 5 This is a flowchart illustrating another text data generation method provided in this disclosure. Figure 5 As shown, before encoding the prompt text data by the second text encoder, the above method further includes steps S510 to S540.

[0066] In step S510, the output of the cross-attention layer is enhanced based on the learnable query vector to obtain the semantic enhancement features of the historical image.

[0067] In step S520, historical prompt text data is obtained, and the historical prompt text data is encoded by a second text encoder to obtain the semantic features of the historical prompt text data.

[0068] In step S530, the semantic enhancement features of the historical image and the semantic features of the historical prompt text data are fused to obtain the fused features of the historical image.

[0069] In step S540, supervised learning is performed based on the fusion features of historical images and the loss function of the text generation task to update the parameters of the second text encoder.

[0070] This method combines the output of a multi-granularity image encoder with the text features of a second text encoder, employing a cross-attention mechanism and learnable query vectors to ultimately achieve text generation. This process not only enhances the model's generation capability but also strengthens its understanding of multimodal data. Through supervised learning, the parameters of the second text encoder are further optimized to ensure that the generated text is closely related to the input image and prompt text.

[0071] In some embodiments of this disclosure, after the multi-granularity image encoder is trained, text generation is performed by combining the image features learned by the multi-granularity image encoder with the text features output by the second encoder to enhance the model's generation capability. This text generation refers to the question text before inputting into the large language model, i.e., the combination of image and prompt text. In this embodiment, the second text encoder is trained using the output (key-value pairs) of the multi-granularity image encoder, the output of the second text encoder, and the introduced learnable query vector (Q). The second text encoder uses the basic structure and weights of the first text encoder. Specifically, the output (key-value pairs) of the multi-granularity image encoder and the introduced learnable query vector (Q) need to be processed through a cross-attention layer. For example, historical image data is processed by the trained multi-granularity image encoder to obtain corresponding semantic features. These semantic features are processed through a cross-attention layer, and then the output of the cross-attention layer is enhanced based on the learnable query vector to obtain semantically enhanced features of the historical images. Historical prompt text data is obtained, and the historical prompt text data is encoded by the second text encoder to obtain the semantic features of the historical prompt text data. The semantic enhancement features of historical images and the semantic features of historical prompt text data are fused to obtain the fused features of historical images. Supervised learning is then performed based on the fused features of historical images and the loss function of the text generation task to update the parameters of the second text encoder. In this way, by combining the output of the multi-granularity image encoder with the text features of the second text encoder, a cross-attention mechanism and learnable query vectors are used for processing, ultimately achieving the text generation task. This process not only improves the model's generation capability but also enhances its understanding of multimodal data. Through supervised learning, the parameters of the second text encoder are further optimized to ensure that the generated text is closely related to the input image and prompt text.

[0072] In some embodiments of this disclosure, the output of the cross-attention layer is enhanced based on a learnable query vector to obtain semantically enhanced features of the historical image. For example, the cross-attention layer combines features from different modalities (images and text) to generate a richer representation. Specifically, image features are weighted according to the input query vector (here, the learnable query vector Q) to highlight text-related parts of the image. The learnable query vector is an important component, typically learned through backpropagation during training. Its role is to provide contextual information to the cross-attention layer, enabling the model to better understand the intent of the current task. The specific learnable query vector is usually randomly initialized at the start of training and then progressively optimized through training. The dimension of the query vector should match the dimension of the output features of the cross-attention layer for efficient computation. The output of the cross-attention layer is a weighted result based on the input image features (from a multi-granularity image encoder) and the learnable query vector. The image features are weighted and summed using the calculated attention weights to obtain the output of the cross-attention layer. The output of the cross-attention layer is enhanced by introducing learnable linear transformations or non-linear activation functions (such as ReLU and GELU) to further process the output and improve the expressive power of the features. The enhanced features can be concatenated with the original image features or weighted summed to further enrich the feature representation. After these enhancements, the semantically enhanced features of the historical image better represent the relationship between the image content and the prompt text. These enhanced features will serve as input to subsequent text generation models, helping to generate more relevant and accurate text descriptions. By enhancing the output of the cross-attention layer based on learnable query vectors, the semantic feature representation capability of historical images can be effectively improved. This process not only improves the model's understanding of image content but also enhances its performance in multimodal generation tasks.

[0073] In some embodiments of this disclosure, historical prompt text data is acquired and encoded by a second text encoder to obtain semantic features of the historical prompt text data. For example, historical prompt text data refers to text information related to historical images, typically used to guide the model in generating descriptions that match the image content. The second text encoder is based on the structure and weights of the first text encoder, typically employing a Transformer architecture or other deep learning models. Its main function is to convert the input text data into a high-dimensional semantic feature representation. The preprocessed historical prompt text data is input into the second text encoder, usually represented in the form of word vectors. The text is first converted into a fixed-dimensional vector for each word through an embedding layer. Pre-trained word vectors (such as Word2Vec, GloVe) or randomly initialized embeddings can be used. After inputting the historical prompt text, the second text encoder encodes it. The self-attention mechanism in the encoder calculates the relationships between words in the text, generating a contextual representation for each word. For each word, its attention weights with other words are calculated to capture global information of the text. The output of the self-attention layer is passed to a feedforward neural network, typically consisting of two linear transformations and an activation function. After each major step, layer normalization is typically performed to ensure numerical stability and accelerate training. Following the above encoding process, the second text encoder generates a semantic feature representation of the historical prompt text data: the final output is a high-dimensional vector or a set of vectors representing the semantic information of the entire text. These features capture information such as the text's context, sentiment, and topic. If the input historical prompt text is a sequence, a fixed-length feature vector is typically obtained using the last hidden state of the sequence or through pooling operations (such as max pooling or average pooling). The resulting semantic features of the historical prompt text will be used for subsequent tasks, such as text generation and feature fusion. These features help the model better understand the text content, thereby generating more accurate descriptions related to the image. By acquiring historical prompt text data and encoding it using the second text encoder, the semantic features of the text can be obtained. This process not only converts the text into a machine-understandable form but also provides rich contextual information for subsequent generation tasks.

[0074] In some embodiments of this disclosure, semantic enhancement features of historical images and semantic features of historical prompt text data are fused to obtain fused features of the historical images. Supervised learning is then performed based on the fused features of the historical images and the loss function of the text generation task to update the parameters of the second text encoder. For example, the main purpose of fusing the semantic enhancement features of historical images and the semantic features of historical prompt text data is to generate a comprehensive feature representation that can simultaneously capture information from both images and text, thereby providing richer context for subsequent text generation tasks. The fusion process can employ various methods. For instance, the semantic enhancement features of historical images and the semantic features of historical prompt text can be directly concatenated to form a higher-dimensional feature vector. Alternatively, the two features can be weighted and summed, with the weights being learnable parameters or set based on prior knowledge. Furthermore, features can be fused through interactive operations (e.g., through attention mechanisms), which allows the model to adaptively learn how to combine image and text features. The above fusion methods generate fused features of the historical images, which will serve as input in subsequent text generation tasks. The fused features not only contain the semantic information of the images but also incorporate the contextual information of the text, making the generated text more relevant and accurate. In text generation tasks, supervised learning methods are typically used to update model parameters. The following is a detailed description of this process: The goal of text generation tasks is to generate relevant text descriptions based on the fused features of historical images.

[0075] Loss Function Definition: The loss function measures the difference between the generated text and the real text. Commonly used loss functions include: Cross-entropy loss: used for classification tasks, it calculates the difference between the probability distribution of each generated word and the target word. Parameter Update: By calculating the gradient of the loss function, optimization algorithms (such as Adam, SGD, etc.) are used to update the parameters of the second text encoder. Through the above process, the parameters of the second text encoder will be adjusted according to the quality of the generated text. This supervised learning approach allows the model to progressively optimize its understanding of the relationship between images and text, thereby improving the relevance and quality of the generated text. By fusing the semantic enhancement features of historical images and the semantic features of historical prompt text data, a comprehensive historical image fusion feature can be obtained. Based on this fusion feature and the loss function of the text generation task, supervised learning can effectively update the parameters of the second text encoder, thereby improving the model's performance in multimodal text generation tasks.

[0076] Figure 6 This is a flowchart illustrating another text data generation method provided in this disclosure. Figure 6 As shown, before processing the semantic features of the image data through the first multilayer perceptron, the above method further includes steps S610 to S650.

[0077] In step S610, the target semantic features of the historical image output by the multi-granularity image encoder are processed by the first multilayer perceptron to obtain the target semantic enhancement features of the historical image.

[0078] In step S620, the semantic features of the historical prompt text data output by the first text encoder are processed by the second multilayer perceptron to obtain the first semantic enhancement feature of the historical prompt text data.

[0079] In step S630, the semantic features of the historical prompt text data output by the second text encoder are processed by the third multilayer perceptron to obtain the second semantic enhancement features of the historical prompt text data.

[0080] In step S640, the target semantic enhancement features of the historical image, the first semantic enhancement features of the historical prompt text data, and the second semantic enhancement features of the historical prompt text data are fused to obtain the target semantic fusion features.

[0081] In step S650, the target semantic fusion features are input into the large language model, and the parameters of the first multilayer perceptron, the second multilayer perceptron, and the third multilayer perceptron are updated based on the target semantic fusion features using the loss function in the large language model.

[0082] This method fuses the target semantic enhancement features of historical images with the first and second semantic enhancement features of historical prompt text data. The resulting target semantic fusion feature can fully utilize multimodal information and enhance the model's performance. After inputting this feature into a large language model, gradient calculation and parameter updates are performed through a loss function, thereby improving the performance of the first, second, and third multilayer perceptrons and ultimately achieving a more accurate multimodal learning task.

[0083] In some embodiments of this disclosure, after the multi-granularity image encoder and the second text encoder are trained, a first multilayer perceptron, a second multilayer perceptron, and a third multilayer perceptron can be introduced to fine-tune the entire model to further improve its accuracy. For example, the first multilayer perceptron processes the target semantic features of the historical images output by the multi-granularity image encoder to obtain target semantic enhancement features of the historical images. The second multilayer perceptron processes the semantic features of the historical prompt text data output by the first text encoder to obtain first semantic enhancement features of the historical prompt text data. The third multilayer perceptron processes the semantic features of the historical prompt text data output by the second text encoder to obtain second semantic enhancement features of the historical prompt text data. The target semantic enhancement features of the historical images, the first semantic enhancement features of the historical prompt text data, and the second semantic enhancement features of the historical prompt text data are fused to obtain target semantic fusion features. The target semantic fusion features are input into a large language model, and the parameters of the first multilayer perceptron, the second multilayer perceptron, and the third multilayer perceptron are updated based on the target semantic fusion features using the loss function in the large language model.

[0084] In some embodiments of this disclosure, the target semantic features of a historical image output by a multi-granularity image encoder are processed by a first multilayer perceptron to obtain target semantic enhancement features of the historical image. For example, the target semantic features of the historical image are feature vectors generated by the multi-granularity image encoder, which typically contain a high-dimensional representation of the image and can capture important information and semantic content in the image. These features may be obtained after processing by convolutional layers, pooling layers, and activation functions, and have a certain degree of abstraction. The first multilayer perceptron (MLP) typically consists of multiple fully connected layers, each of which may contain the following components: an input layer, the number of nodes in which is equal to the dimension of the target semantic features of the historical image; hidden layers, which may contain one or more hidden layers, with a common structure of two to three layers. The number of nodes in each hidden layer can be adjusted according to the complexity of the task, and is usually reduced layer by layer; activation functions: non-linear activation functions (such as ReLU, Leaky ReLU, or ELU) are used to introduce non-linear feature transformations, with ReLU being the most commonly used choice; and output nodes, the number of nodes in the output layer can be set according to the required feature dimension, and is usually matched with the dimension of the target semantic enhancement features. After inputting the target semantic features of the historical image, the processing flow of the first multilayer perceptron is as follows: the input features of each layer undergo a linear transformation through the weight matrix, and an activation function is applied to the result of the linear transformation to obtain the activation output of the current layer. Through multi-layer nonlinear transformation, the first multilayer perceptron can extract more discriminative features, thereby enhancing the target semantic features of the historical image. This enhanced feature can better represent the semantic information of the image. After processing by the first multilayer perceptron, the resulting feature vector is an enhanced version of the input features, containing richer semantic information. These features will be used for subsequent feature fusion or input into other models (such as large language models). By processing the target semantic features of the historical image output by the multi-granularity image encoder through the first multilayer perceptron, the semantic information of the image can be effectively extracted and enhanced. This process is achieved through a multi-layer fully connected structure and a nonlinear activation function, and the final target semantic enhanced features of the historical image will provide more powerful feature support for subsequent multimodal learning tasks.

[0085] Based on the aforementioned embodiments, the second and third multilayer perceptrons are similar to the processing flow of the first multilayer perceptron described above. Through this process, the first semantic enhancement features and the second semantic enhancement features of the historical prompt text data can be obtained.

[0086] In some embodiments of this disclosure, the target semantic enhancement features of historical images, the first semantic enhancement features of historical prompt text data, and the second semantic enhancement features of historical prompt text data are fused to obtain target semantic fusion features. These target semantic fusion features are input into a large language model, and the parameters of the first, second, and third multilayer perceptrons are updated based on the target semantic fusion features using the loss function in the large language model. For example, the target semantic enhancement features of historical images, processed by the first multilayer perceptron, contain rich semantic information about the image. The first semantic enhancement features of historical prompt text data, processed by the second multilayer perceptron, better represent the semantic information of the text. The second semantic enhancement features of historical prompt text data, processed by the third multilayer perceptron, also better represent the semantic information of the text. Feature fusion combines features from different sources to enable the model to obtain more comprehensive information from multimodal data. The fusion processing steps are as follows: Concatenation operation: The three feature vectors are concatenated to form a large feature vector. Linear transformation: A linear layer can be used to transform the concatenated features to better adapt to the input of subsequent models. The fused target semantic features are input into a large language model (such as GPT, BERT, etc.) for further text generation or understanding tasks. The large language model calculates a loss function based on its task (such as text generation, classification, etc.). This loss function is typically based on the difference between the model's predicted output and the true label. Model parameters are updated using the backpropagation algorithm, with the following steps: Gradient calculation: The gradient of the loss function with respect to each model parameter is calculated using the chain rule, including the parameters of the first, second, and third multilayer perceptrons. Parameter update: The parameters of each multilayer perceptron are updated using optimization algorithms (such as SGD, Adam, etc.). By fusing the target semantic enhancement features of historical images with the first and second semantic enhancement features of historical prompt text data, the resulting target semantic fusion feature can fully utilize multimodal information and enhance model performance. After inputting this feature into the large language model, gradient calculation and parameter updates are performed using the loss function, thereby improving the performance of the first, second, and third multilayer perceptrons, ultimately achieving a more accurate multimodal learning task.

[0087] Figure 7 This is a block diagram of a text data generation apparatus provided in an embodiment of the present disclosure.

[0088] like Figure 7 As shown, the text data generation device 700 includes an acquisition module 710, a first processing module 720, a second processing module 730, a third processing module 740, a fourth processing module 750, a splicing module 760, and a text generation module 770.

[0089] Specifically, the acquisition module 710 is used to acquire image data and prompt text data.

[0090] The first processing module 720 is used to perform semantic processing on image data through a multi-granularity image encoder to obtain semantic features of the image data. The multi-granularity image encoder is trained based on historical image data and real semantic text data of historical image data.

[0091] The second processing module 730 is used to process the semantic features of the image data through the first multilayer perceptron to obtain the first semantic features.

[0092] The third processing module 740 is used to encode the prompt text data through the first text encoder and process the result of the first text encoder through the second multilayer perceptron to obtain the second semantic feature. The first text encoder is trained based on the real semantic text data of historical image data and the output result of the multi-granularity image encoder.

[0093] The fourth processing module 750 is used to encode the prompt text data through the second text encoder and process the result of the second text encoder through the third multilayer perceptron to obtain the third semantic feature. The second text encoder is trained based on historical image data and historical prompt text data.

[0094] The splicing module 760 is used to splice the first semantic feature, the second semantic feature, and the third semantic feature to obtain the target semantic data of the image data.

[0095] The text generation module 770 is used to process the target semantic data of the image data through a large language model to obtain the feedback text data of the image data.

[0096] The text data generation device 700 can perform semantic processing on image data through a multi-granularity image encoder, enabling the extraction of richer and more detailed semantic features. This training method, based on historical image data and real semantic text data, allows the model to better understand image content, thereby improving the depth of understanding of complex visual scenes. By combining the outputs of the first text encoder and the multi-granularity image encoder, the differences between different feature representations can be effectively reduced, improving the alignment between image and text features. This consistency helps the model provide more accurate semantic feedback when processing multimodal data. Through the concatenation of the first, second, and third semantic features, effective integration of features from different dimensions is achieved. This multi-level feature fusion can more comprehensively reflect the target semantics of image data, improving the model's performance in complex tasks. When processing low-resolution and multi-scale images, this invention enhances the model's robustness to image diversity and complexity through a multi-granularity image encoder. This allows the model to better adapt to image data under different scenarios and conditions in practical applications. By processing the target semantic data of image data through a large language model, more accurate and relevant feedback text data can be generated. This high-quality text feedback not only enhances the user experience but also provides a more reliable information foundation for subsequent applications.

[0097] In some embodiments of this disclosure, before semantic processing of image data by a multi-granularity image encoder, the text data generation apparatus 700 is further configured to: acquire historical image data and input the historical image data into a multi-granularity image encoder, the multi-granularity image encoder comprising a contrastive language-image pre-trained image encoder, a first convolutional layer, a visual image encoder, a second convolutional layer, an open object detector, and an average pooling layer; encode the historical image data by the contrastive language-image pre-trained image encoder, and convolve the result of the contrastive language-image pre-trained image encoder by the first convolutional layer to obtain a first convolutional semantic feature of the historical image; encode the historical image data by the visual image encoder, and convolve the result output by the visual image encoder by the second convolutional layer to obtain a second convolutional semantic feature of the historical image; detect the historical image data by the open object detector, and pool the result output by the open object detector by the average pooling layer to obtain at least one pooled semantic feature of the historical image; and determine the target semantic feature of the historical image based on the first convolutional semantic feature, the second convolutional semantic feature, and the at least one pooled semantic feature of the historical image.

[0098] In some embodiments of this disclosure, before encoding the prompt text data by the first text encoder, the text data generation device 700 is further configured to: acquire real semantic text data of historical image data; encode the real semantic text data of historical image data by the first text encoder to obtain semantic features of the real semantic text data; calculate a contrast loss based on the target semantic features of the historical image and the semantic features of the real semantic text data using a contrast loss function; and update the parameters in the multi-granularity image encoder and the parameters in the first text encoder based on the contrast loss.

[0099] In some embodiments of this disclosure, the multi-granularity image encoder further includes a cross-attention layer, a fourth multilayer perceptron, and a fifth multilayer perceptron. Determining the target semantic features of a historical image based on a first convolutional semantic feature, a second convolutional semantic feature, and at least one pooling semantic feature of the historical image includes: processing the first convolutional semantic feature through the cross-attention layer and processing the output of the cross-attention layer through the fourth multilayer perceptron to obtain a first historical target semantic feature; processing the second convolutional semantic feature through the cross-attention layer and processing the output of the cross-attention layer through the fourth multilayer perceptron to obtain a second historical target semantic feature; concatenating at least one pooling semantic feature and processing the concatenation result through the fifth multilayer perceptron to obtain a third historical target semantic feature; and concatenating the first historical target semantic feature, the first historical target semantic feature, and the first historical target semantic feature to obtain the target semantic features of the historical image.

[0100] In some embodiments of this disclosure, before encoding the prompt text data by the second text encoder, the text data generation device 700 is further configured to: enhance the output of the cross-attention layer based on the learnable query vector to obtain semantic enhancement features of the historical image; acquire historical prompt text data, encode the historical prompt text data by the second text encoder to obtain semantic features of the historical prompt text data; fuse the semantic enhancement features of the historical image and the semantic features of the historical prompt text data to obtain fused features of the historical image; and perform supervised learning based on the fused features of the historical image and the loss function of the text generation task to update the parameters of the second text encoder.

[0101] In some embodiments of this disclosure, before processing the semantic features of the image data using the first multilayer perceptron, the text data generation apparatus 700 is further configured to: process the target semantic features of the historical image output by the multi-granularity image encoder using the first multilayer perceptron to obtain target semantic enhancement features of the historical image; process the semantic features of the historical prompt text data output by the first text encoder using the second multilayer perceptron to obtain first semantic enhancement features of the historical prompt text data; process the semantic features of the historical prompt text data output by the second text encoder using the third multilayer perceptron to obtain second semantic enhancement features of the historical prompt text data; fuse the target semantic enhancement features of the historical image, the first semantic enhancement features of the historical prompt text data, and the second semantic enhancement features of the historical prompt text data to obtain target semantic fusion features; input the target semantic fusion features into a large language model, and update the parameters of the first multilayer perceptron, the second multilayer perceptron, and the third multilayer perceptron based on the target semantic fusion features using the loss function in the large language model.

[0102] In some embodiments of this disclosure, the first processing module 720 is configured to: encode image data using a contrastive language-image pre-trained image encoder, and convolve the result of the contrastive language-image pre-trained image encoder using a first convolutional layer to obtain a first convolutional semantic feature of the image; encode image data using a visual image encoder, and convolve the result of the visual image encoder using a second convolutional layer to obtain a second convolutional semantic feature of the image; detect image data using an open object detector, and pool the result of the open object detector using an average pooling layer to obtain at least one pooling semantic feature of the image; and determine a target semantic feature of the image based on the first convolutional semantic feature, the second convolutional semantic feature, and the at least one pooling semantic feature.

[0103] Figure 8 This is a schematic diagram of the electronic device 8 provided in an embodiment of this disclosure. Figure 8 As shown, the electronic device 8 of this embodiment includes a processor 801, a memory 802, and a computer program 803 stored in the memory 802 and executable on the processor 801. When the processor 801 executes the computer program 803, it implements the steps in the various method embodiments described above. Alternatively, when the processor 801 executes the computer program 803, it implements the functions of each module in the various device embodiments described above.

[0104] Electronic device 8 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 8 may include, but is not limited to, processor 801 and memory 802. Those skilled in the art will understand that... Figure 8This is merely an example of electronic device 8 and does not constitute a limitation on electronic device 8. It may include more or fewer components than shown, or different components.

[0105] The processor 801 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0106] The memory 802 can be an internal storage unit of the electronic device 8, such as a hard disk or RAM of the electronic device 8. The memory 802 can also be an external storage device of the electronic device 8, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the electronic device 8. The memory 802 can also include both internal and external storage units of the electronic device 8. The memory 802 is used to store computer programs and other programs and data required by the electronic device.

[0107] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0108] If the integrated module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0109] The above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit it. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be included within the protection scope of this disclosure.

Claims

1. A method for generating text data, characterized in that, include: Acquire image data and prompt text data; The image data is semantically processed by a multi-granularity image encoder to obtain the semantic features of the image data. The multi-granularity image encoder is trained based on historical image data and the real semantic text data of the historical image data. The semantic features of the image data are processed by a first multilayer perceptron to obtain the first semantic features; The prompt text data is encoded by a first text encoder, and the result of the first text encoder is processed by a second multilayer perceptron to obtain a second semantic feature. The first text encoder is trained based on the real semantic text data of the historical image data and the output of the multi-granularity image encoder. The prompt text data is encoded by a second text encoder, and the result of the second text encoder is processed by a third multilayer perceptron to obtain a third semantic feature. The second text encoder is trained based on the historical image data and historical prompt text data. The first semantic feature, the second semantic feature, and the third semantic feature are concatenated to obtain the target semantic data of the image data; The target semantic data of the image data is processed by a large language model to obtain the feedback text data of the image data.

2. The method according to claim 1, characterized in that, Before performing semantic processing on the image data using a multi-granularity image encoder, the method further includes: The historical image data is acquired and input into the multi-granularity image encoder, which includes a contrastive language-image pre-trained image encoder, a first convolutional layer, a visual image encoder, a second convolutional layer, an open object detector, and an average pooling layer. The historical image data is encoded by the contrastive language-image pre-trained image encoder, and the result of the contrastive language-image pre-trained image encoder is convolved by the first convolutional layer to obtain the first convolutional semantic features of the historical image. The historical image data is encoded by the visual image encoder, and the output of the visual image encoder is convolved by the second convolutional layer to obtain the second convolutional semantic features of the historical image. The historical image data is detected and processed by the open object detector, and the output of the open object detector is pooled by the average pooling layer to obtain at least one pooled semantic feature of the historical image. The target semantic features of the historical image are determined based on the first convolutional semantic features of the historical image, the second convolutional semantic features of the historical image, and at least one pooling semantic feature of the historical image.

3. The method according to claim 2, characterized in that, Before encoding the prompt text data using the first text encoder, the method further includes: Obtain the true semantic text data of the historical image data; The first text encoder encodes the real semantic text data of the historical image data to obtain the semantic features of the real semantic text data. The contrast loss is calculated by comparing the target semantic features of the historical image with the semantic features of the real semantic text data using a contrast loss function. The parameters in the multi-granularity image encoder and the parameters in the first text encoder are updated based on the contrast loss.

4. The method according to claim 2, characterized in that, The multi-granularity image encoder also includes a cross-attention layer, a fourth multilayer perceptron, and a fifth multilayer perceptron. Determining the target semantic features of the historical image based on the first convolutional semantic features, the second convolutional semantic features, and at least one pooling semantic feature of the historical image includes: The first convolutional semantic features are processed by the cross-attention layer, and the output of the cross-attention layer is processed by the fourth multilayer perceptron to obtain the first historical target semantic features. The second convolutional semantic features are processed by the cross-attention layer, and the output of the cross-attention layer is processed by the fourth multilayer perceptron to obtain the second historical target semantic features. At least one of the pooled semantic features is concatenated, and the concatenation result is processed by the fifth multilayer perceptron to obtain the third historical target semantic feature; The first historical target semantic features, the first historical target semantic features, and the first historical target semantic features are concatenated to obtain the target semantic features of the historical image.

5. The method according to claim 4, characterized in that, Before encoding the prompt text data using the second text encoder, the method further includes: The output of the cross-attention layer is enhanced based on the learnable query vector to obtain the semantic enhancement features of the historical image; Historical prompt text data is acquired, and the historical prompt text data is encoded using the second text encoder to obtain the semantic features of the historical prompt text data; The semantic enhancement features of the historical image and the semantic features of the historical prompt text data are fused to obtain the fused features of the historical image; Supervised learning is performed based on the fusion features of the historical images and the loss function of the text generation task to update the parameters of the second text encoder.

6. The method according to claim 5, characterized in that, Before processing the semantic features of the image data using a first multilayer perceptron, the method further includes: The target semantic features of the historical image output by the multi-granularity image encoder are processed by the first multi-layer perceptron to obtain the target semantic enhancement features of the historical image; The semantic features of the historical prompt text data output by the first text encoder are processed by the second multilayer perceptron to obtain the first semantic enhancement feature of the historical prompt text data; The semantic features of the historical prompt text data output by the second text encoder are processed by the third multilayer perceptron to obtain the second semantic enhancement features of the historical prompt text data; The target semantic enhancement features of the historical image, the first semantic enhancement features of the historical prompt text data, and the second semantic enhancement features of the historical prompt text data are fused to obtain the target semantic fusion features; The target semantic fusion features are input into the large language model, and the parameters of the first multilayer perceptron, the second multilayer perceptron, and the third multilayer perceptron are updated based on the target semantic fusion features using the loss function in the large language model.

7. The method according to claim 2, characterized in that, The semantic features of the image data obtained by performing semantic processing on the image data through a multi-granularity image encoder include: The image data is encoded by the contrastive language-image pre-trained image encoder, and the result of the contrastive language-image pre-trained image encoder is convolved by the first convolutional layer to obtain the first convolutional semantic feature of the image. The image data is encoded by the visual image encoder, and the output of the visual image encoder is convolved by the second convolutional layer to obtain the second convolutional semantic features of the image. The image data is detected and processed by the open object detector, and the result output by the open object detector is pooled by the average pooling layer to obtain at least one pooled semantic feature of the image. The target semantic features of the image are determined based on the first convolutional semantic features of the image, the second convolutional semantic features of the image, and at least one pooling semantic feature of the image.

8. A text data generation device, characterized in that, include: The acquisition module is used to acquire image data and prompt text data; The first processing module is used to perform semantic processing on the image data through a multi-granularity image encoder to obtain the semantic features of the image data. The multi-granularity image encoder is trained based on historical image data and the real semantic text data of the historical image data. The second processing module is used to process the semantic features of the image data through the first multilayer perceptron to obtain the first semantic features; The third processing module is used to encode the prompt text data through the first text encoder and process the result of the first text encoder through the second multilayer perceptron to obtain the second semantic feature. The first text encoder is trained based on the real semantic text data of the historical image data and the output result of the multi-granularity image encoder. The fourth processing module is used to encode the prompt text data through the second text encoder and process the result of the second text encoder through the third multilayer perceptron to obtain the third semantic feature. The second text encoder is trained based on the historical image data and historical prompt text data. The splicing module is used to splice the first semantic feature, the second semantic feature, and the third semantic feature to obtain the target semantic data of the image data; The text generation module is used to process the target semantic data of the image data through a large language model to obtain the feedback text data of the image data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images