Text data generation method and device, electronic equipment and medium

Abstract

The invention relates to the technical field of text data generation, and provides a text data generation method and device, electronic equipment and a medium. According to the method, semantic processing can be performed on image data through a multi-granularity image encoder to obtain semantic features of the image data; processing the semantic features of the image data through a first multi-layer perceptron to obtain first semantic features; processing the prompt text data through a first text encoder and a second text encoder to obtain a second semantic feature and a third semantic feature; splicing the first semantic feature, the second semantic feature and the third semantic feature to obtain target semantic data of the image data; and processing the target semantic data of the image data through the large language model to obtain feedback text data of the image data. According to the method, the semantic understanding capability between the image and the text is remarkably improved, a plurality of key problems in the prior art are solved, and powerful support is provided for development of a multi-mode manual universal intelligent system.

Description

Technical Field

[0001] The present disclosure relates to the technical field of text data generation, and in particular to a text data generation method, device, electronic device and medium. Background Art

[0002] With the rapid development of Internet technology, multimodal large language models have shown great potential in the recognition and understanding of images and texts. These models can provide richer semantic understanding through the fusion analysis of images and texts, thus promoting the progress in the fields of computer vision and natural language processing.

[0003] However, existing multimodal large language models face several significant key shortcomings in their evolution towards artificial general intelligence systems. First, despite the significant progress in parameter scale and performance of large language models, existing multimodal models have failed to keep pace with the development of vision and visual language base models. This leads to a significant gap between the parameter scale and feature representation capability of the visual encoder and the large language model, limiting the overall performance of the multimodal model. Second, current multimodal large language models usually use a lightweight "glue" layer to align the features of the visual and language models. However, this approach has many problems in terms of parameter scale differences, representation consistency, and connection efficiency, which affects the overall performance and application effect of the model. In addition, the diversity and complexity of images in the real world pose severe challenges to the robustness of multimodal large language models. Existing models show certain limitations in processing low-resolution and multi-scale images, making it difficult to fully capture the details and rich information in the image, thus affecting the accuracy and depth of semantic understanding. Finally, existing multimodal large language models lack the ability to effectively integrate object-level features and process multi-dimensional features. This deficiency limits the model's performance in complex visual scenes, reducing its depth of understanding and scope of application. Therefore, there is an urgent need to improve the multimodal large language model to enhance its performance in visual and language interaction, thereby promoting the further development of artificial general intelligence systems. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure provide a text data generation method, device, electronic device and computer-readable storage medium to solve the technical problem of low accuracy in recognizing images and texts by large multimodal models in the prior art.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for generating text data is provided, including: acquiring image data and prompt text data; semantically processing the image data through a multi-granularity image encoder to obtain semantic features of the image data, the multi-granularity image encoder being trained based on historical image data and real semantic text data of the historical image data; processing the semantic features of the image data through a first multi-layer perceptron to obtain a first semantic feature; encoding the prompt text data through a first text encoder, and processing the result of the first text encoder through a second multi-layer perceptron to obtain a second semantic feature, the first text encoder being trained based on the real semantic text data of the historical image data and the result output by the multi-granularity image encoder; encoding the prompt text data through a second text encoder, and processing the result of the second text encoder through a third multi-layer perceptron to obtain a third semantic feature, the second text encoder being trained based on the historical image data and the historical prompt text data; concatenating the first semantic feature, the second semantic feature, and the third semantic feature to obtain target semantic data of the image data; processing the target semantic data of the image data through a large language model to obtain feedback text data of the image data.

[0006] According to a second aspect of an embodiment of the present disclosure, a text data generating device is provided, including: an acquisition module for acquiring image data and prompt text data; a first processing module for performing semantic processing on the image data through a multi-granularity image encoder to obtain semantic features of the image data, wherein the multi-granularity image encoder is trained based on historical image data and real semantic text data of the historical image data; a second processing module for processing the semantic features of the image data through a first multi-layer perceptron to obtain a first semantic feature; and a third processing module for encoding the prompt text data through the first text encoder, and processing the result of the first text encoder through the second multi-layer perceptron to obtain a second semantic feature. The first text encoder is trained based on the real semantic text data of the historical image data and the output results of the multi-granularity image encoder; the fourth processing module is used to encode the prompt text data through the second text encoder, and process the result of the second text encoder through the third multi-layer perceptron to obtain the third semantic feature, and the second text encoder is trained based on the historical image data and the historical prompt text data; the splicing module is used to splice the first semantic feature, the second semantic feature, and the third semantic feature to obtain the target semantic data of the image data; the text generation module is used to process the target semantic data of the image data through the large language model to obtain the feedback text data of the image data.

[0007] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0008] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0009] Compared with the prior art, the embodiments of the present disclosure have the following beneficial effects: the embodiments of the present disclosure can perform semantic processing on image data through a multi-granularity image encoder, and can extract richer and more detailed semantic features. This training method based on historical image data and real semantic text data enables the model to better understand the image content, thereby improving the depth of understanding of complex visual scenes. By combining the output results of the first text encoder with the multi-granularity image encoder, the difference between different feature representations can be effectively reduced, and the alignment effect of image and text features can be improved. This consistency helps the model to provide more accurate semantic feedback when processing multimodal data. By splicing the first, second, and third semantic features, effective integration of features of different dimensions is achieved. This multi-level feature fusion can more comprehensively reflect the target semantics of image data and improve the performance of the model in complex tasks. When processing low-resolution and multi-scale images, the present invention enhances the robustness of the model to image diversity and complexity through a multi-granularity image encoder. This enables the model to better adapt to image data under different scenarios and conditions in practical applications. By processing the target semantic data of image data through a large language model, more accurate and relevant feedback text data can be generated. This high-quality text feedback not only improves the user experience, but also provides a more reliable information basis for subsequent applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present invention can be applied;

[0012] Figure 2 It is a flowchart of a method for generating text data provided by an embodiment of the present disclosure;

[0013] Figure 3 is a flowchart of another text data generation method provided by an embodiment of the present disclosure;

[0014] Figure 4 It is a flowchart of another text data generation method provided by an embodiment of the present disclosure;

[0015] Figure 5 It is a flowchart of another method for generating text data provided by an embodiment of the present disclosure;

[0016] Figure 6 It is a flowchart of another method for generating text data provided by an embodiment of the present disclosure;

[0017] Figure 7 is a block diagram of a text data generating device provided by an embodiment of the present disclosure;

[0018] Figure 8 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0020] It should be noted that the user information (including but not limited to terminal device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0021] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present invention can be applied.

[0022] like Figure 1 As shown, the system architecture 100 may include one or more of a first terminal device 101, a second terminal device 102, and a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0023] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0024] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send data, etc. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, desktop computers, and the like.

[0025] The server 105 may be a server that provides various services. For example, the server 105 can obtain image data and prompt text data from the first terminal device 103 (which can also be the second terminal device 102 or the third terminal device 103); perform semantic processing on the image data through a multi-granularity image encoder to obtain semantic features of the image data, and the multi-granularity image encoder is trained based on historical image data and real semantic text data of the historical image data; process the semantic features of the image data through a first multi-layer perceptron to obtain a first semantic feature; encode the prompt text data through a first text encoder, and process the result of the first text encoder through a second multi-layer perceptron to obtain a second semantic feature, and the first text encoder is trained based on the real semantic text data of the historical image data and the result output by the multi-granularity image encoder; encode the prompt text data through a second text encoder, and process the result of the second text encoder through a third multi-layer perceptron to obtain a third semantic feature, and the second text encoder is trained based on historical image data and historical prompt text data; splice the first semantic feature, the second semantic feature, and the third semantic feature to obtain target semantic data of the image data; process the target semantic data of the image data through a large language model to obtain feedback text data of the image data.

[0026] In some embodiments, the text data generation method provided in the embodiments of the present invention is generally executed by the server 105, and accordingly, the text data generation device is generally arranged in the server 105. In other embodiments, some terminal devices may have functions similar to those of the server to execute the method. Therefore, the text data generation method provided in the embodiments of the present invention is not limited to being executed on the server side.

[0027] The text data generating method and device according to the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0028] Figure 2is a flowchart of a text data generation method provided by an embodiment of the present disclosure. The method provided by an embodiment of the present disclosure can be executed by any electronic device with computer processing capabilities, for example, the electronic device can be Figure 1 The server is shown.

[0029] like Figure 2 As shown, the text data generating method includes steps S210 to S270.

[0030] In step S210, image data and prompt text data are acquired.

[0031] Step S220, semantic processing is performed on the image data by a multi-granularity image encoder to obtain semantic features of the image data, wherein the multi-granularity image encoder is trained based on historical image data and real semantic text data of the historical image data.

[0032] In step S230, the semantic features of the image data are processed by a first multi-layer perceptron to obtain a first semantic feature.

[0033] In step S240, the prompt text data is encoded by a first text encoder, and the result of the first text encoder is processed by a second multi-layer perceptron to obtain a second semantic feature. The first text encoder is trained based on the real semantic text data of the historical image data and the output result of the multi-granularity image encoder.

[0034] In step S250, the prompt text data is encoded by a second text encoder, and the result of the second text encoder is processed by a third multi-layer perceptron to obtain a third semantic feature, wherein the second text encoder is trained based on historical image data and historical prompt text data.

[0035] In step S260, the first semantic feature, the second semantic feature, and the third semantic feature are spliced ​​to obtain target semantic data of the image data.

[0036] In step S270, the target semantic data of the image data is processed by the large language model to obtain feedback text data of the image data.

[0037] The method can perform semantic processing on image data through a multi-granularity image encoder, and can extract richer and more detailed semantic features. This training method based on historical image data and real semantic text data enables the model to better understand the image content, thereby improving the depth of understanding of complex visual scenes. By combining the output results of the first text encoder with the multi-granularity image encoder, the difference between different feature representations can be effectively reduced, and the alignment effect of image and text features is improved. This consistency helps the model to provide more accurate semantic feedback when processing multimodal data. By splicing the first, second, and third semantic features, effective integration of features of different dimensions is achieved. This multi-level feature fusion can more comprehensively reflect the target semantics of image data and improve the performance of the model in complex tasks. When processing low-resolution and multi-scale images, the present invention enhances the robustness of the model to image diversity and complexity through a multi-granularity image encoder. This enables the model to better adapt to image data under different scenes and conditions in practical applications. By processing the target semantic data of image data through a large language model, more accurate and relevant feedback text data can be generated. This high-quality text feedback not only improves the user experience, but also provides a more reliable information basis for subsequent applications.

[0038] In some embodiments of the present disclosure, the above-mentioned image data can be obtained from multiple image databases, including public data sets, commercial galleries, social media platforms, etc. These data sources usually contain a large number of annotated and classified images, which are suitable for a variety of application scenarios. For example, by writing a crawler program or an API interface, image data can be automatically extracted from a selected database. When actively acquiring images, they can be screened according to specific keywords, tags or conditions to ensure that the acquired images are highly relevant to user needs or task objectives. For example, images can be classified according to their themes, styles or content types to improve the efficiency and accuracy of subsequent processing. In this embodiment, users can also upload image data through the user interface of the terminal device, for example, allowing users to upload images through various terminal devices (such as mobile phones, tablets, computers, etc.). Users can upload by simply dragging, clicking or selecting files, and support multiple image formats (such as JPEG, PNG, GIF, etc.) to meet the needs of different users. After the user uploads the image, real-time feedback, such as image preview, processing progress, etc., can be provided to improve the user experience. In addition, the uploaded image can immediately enter the subsequent semantic processing process to ensure efficiency. When users upload images, user privacy and data security need to be ensured. This can be achieved through encrypted transmission, data anonymization and other means to ensure that images uploaded by users are not accessed by unauthorized third parties. Through the above two methods, image data can be obtained flexibly, and relevant images can be actively extracted from a rich database, and personalized data can be obtained through user uploads. This diversified image acquisition strategy not only improves the richness and diversity of data, but also provides a solid data foundation for subsequent multimodal processing.

[0039] In some embodiments of the present disclosure, a prompt template library containing multiple themes and scenarios can be established. These templates can be designed based on historical data, user needs or specific application scenarios, covering different types of images and content. After the image data is acquired, the most suitable prompt template (i.e., the above-mentioned prompt text data) can be automatically selected according to the content, features or classification information of the image. This automated selection process can be implemented by pre-set rules or machine learning algorithms to ensure that the selected template is highly relevant to the semantics of the image. Users can also customize existing prompt templates in the system to meet specific needs. This flexibility enables users to generate personalized prompt texts according to different application scenarios or project requirements. In this embodiment, the user can also enter a prompt description through the user interface of the terminal device, such as allowing the user to enter a customized prompt description while viewing the image data. The user can quickly express his understanding or needs of the image through a text box or voice input. When the user enters the prompt description, real-time feedback and suggestions can be provided, such as automatically completing or recommending related descriptions based on the keywords entered by the user. This intelligent assistance function can improve the user's input efficiency and help the user express his intention more accurately. There are many input methods, including text, voice, and image annotation. Users can choose the most convenient way to describe the prompt according to their own habits. This diverse input method not only improves the user experience, but also provides the system with richer context information. Through the above two methods, prompt text data can be flexibly obtained. On the one hand, the designed and optimized prompt templates can be obtained from the database to ensure efficiency and consistency; on the other hand, users can also input personalized prompt descriptions based on specific images, which enhances the adaptability and flexibility of the system. This diverse prompt text acquisition strategy provides rich context information for subsequent multimodal processing and improves the overall performance of the system.

[0040] In some embodiments of the present disclosure, the multi-granularity image encoder can be used to extract semantic features of image data. The multi-granularity image encoder includes a contrastive language-image pre-trained image encoder (ContrastiveLanguage-

[0041] Image Pre-training, CLIP), first convolutional layer, Vision Transformer (VIT), second convolutional layer, open object detector, average pooling layer, cross attention layer, fourth multi-layer perceptron and fifth multi-layer perceptron. Figure 3 and Figure 4 The training process of the multi-granularity image encoder is described as follows:

[0042] Figure 3FIG. 1 is a flow chart of another text data generation method provided by an embodiment of the present disclosure. Figure 3 As shown, before the image data is semantically processed by the multi-granularity image encoder, the method further includes steps S310 to S350.

[0043] In step S310, historical image data is obtained and input into a multi-granularity image encoder, which includes a contrastive language-image pre-trained image encoder, a first convolutional layer, a visual image encoder, a second convolutional layer, an open object detector, and an average pooling layer.

[0044] In step S320, the historical image data is encoded by a comparative language-image pre-trained image encoder, and the result of the comparative language-image pre-trained image encoder is convolved by a first convolutional layer to obtain a first convolutional semantic feature of the historical image.

[0045] In step S330, the historical image data is encoded by a visual image encoder, and the result output by the visual image encoder is convolved by a second convolutional layer to obtain a second convolutional semantic feature of the historical image.

[0046] In step S340, the historical image data is detected and processed by an open object detector, and the result output by the open object detector is pooled by an average pooling layer to obtain at least one pooled semantic feature of the historical image.

[0047] In step S350, a target semantic feature of the historical image is determined according to the first convolution semantic feature of the historical image, the second convolution semantic feature of the historical image, and at least one pooled semantic feature of the historical image.

[0048] The method can encode the historical image data by using a contrasting language-image pre-trained image encoder, and convolve the result of the contrasting language-image pre-trained image encoder through a first convolution layer to obtain a first convolution semantic feature of the historical image. The historical image data is encoded by a visual image encoder, and the result output by the visual image encoder is convoluted by a second convolution layer to obtain a second convolution semantic feature of the historical image. The historical image data is detected by an open target detector, and the result output by the open target detector is pooled by an average pooling layer to obtain at least one pooled semantic feature of the historical image. The target semantic feature of the historical image is determined based on the first convolution semantic feature of the historical image, the second convolution semantic feature of the historical image, and at least one pooled semantic feature of the historical image, thereby ensuring the comprehensiveness and accuracy of feature extraction and laying a solid foundation for further analysis and application.

[0049] In some embodiments of the present disclosure, historical image data is obtained and input into a multi-granular image encoder. For example, the historical image data may come from an existing image database, such as a public dataset (e.g., ImageNet, COCO) or a specific historical image archive. Relevant historical images are automatically collected from the Internet through web crawler technology. Users are allowed to upload their historical images to form a dynamic data set. The data may be image files in different formats, such as JPEG, PNG, TIFF, etc., containing various historical scenes, characters, objects, etc. Determine the number of images required to ensure the effectiveness of model training and testing. The size of the data may affect the performance and generalization ability of the model. Before the historical image data is input into the multi-granular image encoder, some preprocessing steps are usually required to ensure the quality and consistency of the data: Use image processing technology to remove noise from the image and improve the image quality. Check the integrity of the image and remove damaged or blurred images. Adjust all images to a uniform size to ensure shape consistency when input into the encoder. Perform color space conversion (such as RGB to grayscale) or color normalization on the image to reduce color differences between different images. Enhance the image by rotating, flipping, scaling, cropping, etc. to increase the diversity of training data and improve the robustness of the model. Convert the preprocessed image data into a format suitable for encoder input. Typically, image data needs to be converted into tensor format for processing in deep learning frameworks such as TensorFlow or PyTorch. Divide the image data into batches so that they can be fed into the encoder for processing batch by batch. This approach can improve computational efficiency and reduce memory usage.

[0050] Each batch of image data is passed to the multi-granular image encoder. The encoder's input layer receives the image data and begins the feature extraction process.

[0051] In some embodiments of the present disclosure, the historical image data is encoded by a contrastive language-image pre-trained image encoder, and the result of the contrastive language-image pre-trained image encoder is convoluted by a first convolutional layer to obtain the first convolutional semantic feature of the historical image. In this embodiment, the first convolutional layer can be a one-dimensional convolutional layer. For example, a contrastive language-image pre-trained image encoder (such as CLIP) is usually composed of two parts: an image encoder and a text encoder. The image encoder is responsible for extracting the features of the image, and the text encoder is responsible for extracting the text features related to the image. The image is feature extracted through a convolutional neural network (CNN) or a transformer (Transformer) architecture. The model associates the image with the corresponding text by contrastive learning, thereby generating a semantic feature representation of the image, and the output obtained is a high-dimensional feature vector representing the semantic information of the image, which is called a "contrastive encoding feature". The output result (contrastive encoding feature) of the contrastive language-image pre-trained image encoder is input to the first convolutional layer. In this embodiment, the first convolutional layer is configured as a one-dimensional convolutional layer (1D Convolution Layer). Although the one-dimensional convolutional layer is usually used to process sequential data (such as time series or text), in this context, it can also be used to process feature vectors, especially when the features are of sequential nature. The one-dimensional convolutional layer uses multiple one-dimensional convolutional kernels (filters), and the size and number of each convolutional kernel can be adjusted according to the specific task. The convolution kernel slides on the feature vector to extract local features. The convolution kernel slides on the input feature vector and generates a new feature map by dot product and summation. Each convolution kernel produces a feature map that represents the activation of a specific pattern or feature. Usually after the convolution operation, a nonlinear activation function (such as ReLU) is applied to introduce nonlinear characteristics, so that the model can learn more complex features. The output is a new feature representation called the "first convolution semantic feature". This feature represents a combination of local and global features of the input image, which is helpful for subsequent analysis and processing. The first convolution semantic feature contains low-level and mid-level feature information of the image, such as edges, textures, shapes, etc. These features are the basis for further analysis and classification by subsequent deep networks. The first convolution semantic feature can be passed as input to subsequent convolution layers or other network structures to further extract higher-level semantic features, which are ultimately used for tasks such as image classification, object detection, etc. By encoding the historical image data using the contrastive language-image pre-trained image encoder and convolving the encoding results using a one-dimensional convolution layer, the first convolution semantic feature of the historical image can be effectively extracted. This process combines image feature extraction and convolution operations, providing rich semantic information for subsequent analysis and application.

[0052] In some embodiments of the present disclosure, the historical image data is encoded by a visual image encoder, and the result output by the visual image encoder is convolved by a second convolutional layer to obtain a second convolutional semantic feature of the historical image. In this embodiment, the second convolutional layer is a one-dimensional convolutional layer. The visual image encoder is usually a deep convolutional neural network (CNN) specially designed to extract image features. Common architectures include ResNet, VGG, Inception, etc. The image is processed through multiple convolutional layers, pooling layers, and activation functions to extract low-level features (such as edges, textures) and mid-level features (such as shapes, patterns) of the image layer by layer. The output is a high-dimensional feature vector representing the semantic information of the image, called "visual coding features". These features provide rich information for subsequent processing. The features (visual coding features) output by the visual image encoder are input into the second convolutional layer. The second convolutional layer is designed as a one-dimensional convolutional layer (1D ConvolutionLayer). Although 1D convolutional layers are usually used to process sequence data, here it can be used to process feature vectors, especially when the features are of sequential nature. 1D convolutional layers use multiple 1D convolutional kernels (filters), and the size and number of each convolutional kernel can be adjusted according to the specific task. The convolution kernel slides on the feature vector to extract local features. The convolution kernel slides on the input feature vector and generates a new feature map by dot product and summation. Each convolution kernel produces a feature map that represents the activation of a specific pattern or feature. Usually after the convolution operation, a nonlinear activation function (such as ReLU) is applied to introduce nonlinear characteristics so that the model can learn more complex features. The output is a new feature representation called the "second convolutional semantic feature". This feature represents the higher-level semantic information of the input image, combined with the features extracted from the visual image encoder. The second convolutional semantic feature contains richer semantic information and can capture complex patterns and relationships in the image. These features are the basis for further analysis and classification by subsequent deep networks. The second convolution semantic features can be passed as input to subsequent convolution layers or other network structures to further extract higher-level semantic features, which are ultimately used for tasks such as image classification, object detection, etc. The second convolution semantic features of historical images can be effectively extracted by encoding the historical image data through the visual image encoder and convolving the encoding results with a one-dimensional convolution layer. This process combines image feature extraction and convolution operations, providing deeper semantic information for subsequent analysis and application.

[0053] In some embodiments of the present disclosure, the historical image data is detected and processed by an open target detector, and the result output by the open target detector is pooled by an average pooling layer to obtain at least one pooled semantic feature of the historical image. In this embodiment, the open target detector identifies at least one object bounding box in the image and extracts features. For example, the open target detector is usually based on a deep learning model, such as YOLO (You Only Look Once), Faster R-CNN or SSD (Single Shot MultiBox Detector). These models are specifically designed to detect objects in images. The open target detector processes the input image, identifies multiple objects in the image, and generates a bounding box for each object. The bounding box is usually represented by four coordinates: the coordinates of the upper left corner and the lower right corner. In each identified bounding box, the open target detector extracts corresponding features. These features can be appearance features, shape features, etc. of the object, and are usually extracted by a convolutional layer. The obtained output includes multiple bounding boxes and their corresponding feature vectors, which represent the objects detected in the image and their related information. The features output by the open target detector (i.e., the feature vectors in each bounding box) are input to the average pooling layer. The average pooling layer downsamples the input features to reduce the dimensionality of the features while retaining important semantic information. The average pooling layer slides over the input features and generates a new feature map by calculating the average value within each window. The average value of each window represents the feature information of the region. After pooling, the dimension of the output feature map is usually reduced, thereby reducing the computational complexity while retaining important semantic information. The resulting pooled semantic features represent the comprehensive information of the objects detected in the historical image. Through average pooling, the noise and redundant information in the feature map are suppressed, thereby highlighting important semantic features. The pooled semantic features can be passed as input to the subsequent multi-layer perceptron. By detecting and processing the historical image data through an open object detector and pooling the detection results using an average pooling layer, at least one pooled semantic feature of the historical image can be effectively extracted. This process combines object detection and feature pooling, providing important semantic information for subsequent analysis and application.

[0054] In some embodiments of the present disclosure, the multi-granularity image encoder further includes a cross-attention layer, a fourth multi-layer perceptron, and a fifth multi-layer perceptron; determining the target semantic feature of the historical image according to the first convolution semantic feature of the historical image, the second convolution semantic feature of the historical image, and at least one pooled semantic feature of the historical image includes: processing the first convolution semantic feature through the cross-attention layer, and processing the output result of the cross-attention layer through the fourth multi-layer perceptron to obtain the first historical target semantic feature; processing the second convolution semantic feature through the cross-attention layer, and processing the output result of the cross-attention layer through the fourth multi-layer perceptron to obtain the second historical target semantic feature; splicing at least one pooled semantic feature, and processing the splicing result through the fifth multi-layer perceptron to obtain the third historical target semantic feature; splicing the first historical target semantic feature, the first historical target semantic feature, and the first historical target semantic feature to obtain the target semantic feature of the historical image. For example, the first convolution semantic feature (from the first convolution layer of the historical image). The first convolution semantic feature is processed through the cross-attention layer. The cross-attention mechanism calculates the correlation between the feature and other features, so that the model can focus on more important information and suppress irrelevant features. The output of the cross attention layer is input into the fourth multi-layer perceptron. The MLP transforms the features through multiple fully connected layers and activation functions to extract the first historical target semantic feature. The second convolutional semantic feature (from the second convolutional layer of the historical image). Similar to the generation of the first historical target semantic feature, the second convolutional semantic feature is processed by the cross attention layer to enhance the relationship between the features. The output of the cross attention layer is input into the fourth multi-layer perceptron to extract the second historical target semantic feature. At least one pooled semantic feature (from the pooling layer of the historical image). At least one pooled semantic feature is spliced ​​to form a new feature representation. This operation can integrate the features in multiple object borders to increase the richness of information. The splicing result is input into the fifth multi-layer perceptron to further extract the third historical target semantic feature. The first historical target semantic feature, the second historical target semantic feature and the third historical target semantic feature are spliced ​​to form a comprehensive feature representation. Through the splicing operation, the model integrates features from different sources to form the target semantic feature of the historical image. This feature integrates multi-granular information and can more comprehensively reflect the semantic content of the image. Through the combined processing of the cross-attention layer and the multi-layer perceptron in the multi-granularity image encoder, the target semantic features of the historical image can be effectively extracted. This process enhances the model's ability to understand historical images by processing and integrating different convolutional features and pooling features, providing rich semantic information for subsequent tasks (such as classification, retrieval, etc.).

[0055] In some embodiments of the present disclosure, semantic processing of image data by a multi-granularity image encoder to obtain semantic features of the image data includes: encoding the image data by a contrastive language-image pre-trained image encoder, and convolving the result of the contrastive language-image pre-trained image encoder by a first convolutional layer to obtain a first convolutional semantic feature of the image; encoding the image data by a visual image encoder, and convolving the result output by the visual image encoder by a second convolutional layer to obtain a second convolutional semantic feature of the image; detecting the image data by an open target detector, and pooling the result output by the open target detector by an average pooling layer to obtain at least one pooled semantic feature of the image; determining the target semantic feature of the image based on the first convolutional semantic feature of the image, the second convolutional semantic feature of the image, and at least one pooled semantic feature of the image. The processing flow of this example is similar to the process of training the multi-granularity image encoder described above and will not be repeated here.

[0056] Figure 4 FIG. 1 is a flow chart of another method for generating text data provided by an embodiment of the present disclosure. Figure 4 As shown, before the prompt text data is encoded by the first text encoder, the method further includes steps S410 to S440.

[0057] In step S410, the real semantic text data of the historical image data is obtained.

[0058] In step S420, the real semantic text data of the historical image data is encoded by a first text encoder to obtain semantic features of the real semantic text data.

[0059] In step S430, the contrast loss is calculated based on the target semantic features of the historical image and the semantic features of the real semantic text data through a contrast loss function.

[0060] In step S440, parameters in the multi-granularity image encoder and parameters in the first text encoder are updated based on the contrast loss.

[0061] The method can obtain the real semantic text data of the historical image data and encode it using the first text encoder to obtain the semantic features of the text. The similarity between the target semantic features of the historical image and the semantic features of the real semantic text data is calculated by the contrast loss function, and then the parameters of the multi-granular image encoder and the first text encoder are updated. This process helps to optimize the correlation between the model and the text, and enhance its performance in multimodal learning tasks.

[0062] In some embodiments of the present disclosure, real semantic text data of historical image data is obtained. For example, the description or label provided manually for each image usually contains information such as objects, scenes, actions, etc. in the image. Speech-to-text data extracted from the video can also be used, especially the image sequence contained in the video can provide contextual information. Specifically, the specific method for obtaining real semantic text data includes: using a web crawler to crawl images and their accompanying text information from a website (such as Flickr, Pinterest). This needs to comply with the terms and conditions of use of the website. Use the API provided by the social media platform or data set (such as Twitter API, Flickr API) to obtain relevant image and text data. The collected text data usually needs to be preprocessed to improve its quality and usability. The acquired image and text data are stored in a database for subsequent access and processing. Commonly used databases include SQL and NoSQL databases. Select a suitable data format (such as JSON, CSV) to store the correspondence between images and text to ensure the readability and usability of the data.

[0063] In some embodiments of the present disclosure, the real semantic text data of the historical image data is encoded by the first text encoder to obtain the semantic features of the real semantic text data. For example, the first text encoder is a core component that converts text data into semantic feature representation. Common text encoders include: RNN (recurrent neural network) is suitable for processing sequence data and can capture contextual relationships in text; LSTM (long short-term memory network) can better capture long-distance dependencies and is suitable for complex text data. GRU (gated recurrent unit) is a simplified version of LSTM, with similar performance and faster training speed. Transformer processes text data through a self-attention mechanism, can process input sequences in parallel, is suitable for encoding long texts, and is often used in modern natural language processing tasks. BERT (Bidirectional Encoder Representation Transformer) is based on the Transformer architecture, can capture contextual information, and is suitable for a variety of text understanding tasks. Before the real semantic text data is input into the encoder, some preprocessing steps are required: remove irrelevant characters, punctuation marks, and noise to ensure the purity of the input. Decompose the text into words or subwords, usually using a word segmentation tool (such as BERT's WordPiece word segmenter). Convert the segmented text into a digital representation, usually using word embeddings (such as Word2Vec, GloVe) or encoder-specific embedding layers (such as BERT's input embedding). For variable-length input sequences, padding is performed to ensure that the input shape is consistent, and masking techniques are used when necessary to ensure that the model only focuses on the valid input parts. Input the processed text data into the first text encoder for encoding. After the input passes through the embedding layer, the text data is forward propagated through the layers of the encoder. For RNN and LSTM, the data is passed step by step; for Transformer, the data is processed in parallel in the self-attention mechanism. Calculate the relationship between each word in the input, generate context-related representations, and capture long-range dependencies. During the encoding process, RNN and LSTM generate hidden states, and Transformer generates contextual representations for each word, and finally outputs a fixed-size vector to represent the semantic features of the entire text. After being processed by the encoder, the output features are the semantic representation of the text. The output feature vector is usually a high-dimensional floating-point vector that can effectively represent the semantic information of the input text. In some cases, a pooling layer (such as maximum pooling or average pooling) can be used to convert sequence features into a fixed-length vector for subsequent processing. These semantic features can be used for downstream tasks such as classification, retrieval, or comparison with image features. The real semantic text data of the historical image data is encoded by the first text encoder, which can convert the text into a high-dimensional semantic feature representation. This process involves selecting a suitable encoder, preprocessing the input text, forward propagation through the encoder, and generating the final semantic feature vector.These features are crucial in multimodal learning and help improve the model's ability to understand the relationship between images and text.

[0064] In some embodiments of the present disclosure, a contrast loss is calculated based on the target semantic features of the historical image and the semantic features of the real semantic text data through a contrast loss function. The parameters in the multi-granularity image encoder and the parameters in the first text encoder are updated based on the contrast loss. For example, the text features obtained by the first text encoder and the image features obtained by the multi-dimensional image encoder are contrastively learned, and feature contrast loss (such as Contrastive Loss) is used to ensure that similar images and texts have similar feature representations. During training, the parameters of CLIP, VIT encoder, open target detector, and first text encoder are frozen, and only two one-dimensional convolutional layers and two MLP layers (i.e., the fourth multi-layer perceptron and the fifth multi-layer perceptron) are trained to learn the feature representation of image-text alignment while optimizing image feature fusion.

[0065] Figure 5 FIG. 1 is a flowchart of another method for generating text data provided by an embodiment of the present disclosure. Figure 5 As shown, before the prompt text data is encoded by the second text encoder, the method further includes steps S510 to S540.

[0066] In step S510, the output result of the cross-attention layer is enhanced based on the learnable query vector to obtain the semantic enhancement features of the historical image.

[0067] In step S520, historical prompt text data is obtained, and the historical prompt text data is encoded by a second text encoder to obtain semantic features of the historical prompt text data.

[0068] In step S530, the semantic enhancement features of the historical image and the semantic features of the historical prompt text data are fused to obtain fused features of the historical image.

[0069] In step S540, supervised learning is performed based on the fused features of the historical images and the loss function of the text generation task to update the parameters of the second text encoder.

[0070] This method can achieve the text generation task by combining the output of the multi-granular image encoder with the text features of the second text encoder, using a cross-attention mechanism and a learnable query vector for processing. This process not only improves the model's generation capabilities, but also enhances its understanding of multimodal data. Through supervised learning, the parameters of the second text encoder are further optimized to ensure that the generated text is closely related to the input image and prompt text.

[0071] In some embodiments of the present disclosure, after the training of the multi-granular image encoder is completed, it is necessary to combine the image features learned by the multi-granular image encoder with the text features output by the second encoder to generate text and enhance the generation ability of the model. The text generation refers to the question text before inputting the large language model, that is, the combination of the image and the prompt text. In this embodiment, the second text encoder is trained by the output (key-value pair) of the multi-granular image encoder, the output of the second text encoder, and the introduced learnable query vector (Q), and the second text encoder adopts the basic structure and weight of the first text encoder. Specifically, the output (key-value pair) of the multi-granular image encoder and the introduced learnable query vector (Q) need to be processed by the cross-attention layer. For example, the historical image data is processed by the trained multi-granular image encoder to obtain the corresponding semantic features. The semantic features are processed by the cross-attention layer, and then the output result of the cross-attention layer is enhanced based on the learnable query vector to obtain the semantic enhancement features of the historical image. The historical prompt text data is obtained, and the historical prompt text data is encoded by the second text encoder to obtain the semantic features of the historical prompt text data. The semantic enhancement features of the historical images and the semantic features of the historical prompt text data are fused to obtain the fused features of the historical images. Supervised learning is performed based on the fused features of the historical images and the loss function of the text generation task to update the parameters of the second text encoder. In this way, by combining the output of the multi-granularity image encoder with the text features of the second text encoder, a cross-attention mechanism and a learnable query vector are used for processing, and finally the text generation task is achieved. This process not only improves the generation ability of the model, but also enhances its understanding of multimodal data. Through supervised learning, the parameters of the second text encoder are further optimized to ensure that the generated text is closely related to the input image and prompt text.

[0072] In some embodiments of the present disclosure, the output result of the cross-attention layer is enhanced based on the learnable query vector to obtain the semantic enhancement features of the historical image. For example, the cross-attention layer combines features from different modalities (image and text) to generate a richer representation. Specifically, the image features are weighted according to the input query vector (here is the learnable query vector Q) to highlight the text-related parts of the image. The learnable query vector is an important component, which is usually learned by back propagation during the training process. Its function is to provide a kind of contextual information for the cross-attention layer so that the model can better understand the intention of the current task. The specific learnable query vector is usually randomly initialized at the beginning of training and then gradually optimized through training. The dimension of the query vector should match the output feature dimension of the cross-attention layer for efficient calculation. The output of the cross-attention layer is a weighted result based on the input image features (from the multi-granularity image encoder) and the learnable query vector. The image features are weighted summed using the calculated attention weights to obtain the output of the cross-attention layer. The output of the cross-attention layer is enhanced, for example, by introducing a learnable linear transformation or nonlinear activation function (such as ReLU, GELU, etc.) to further process the output of the cross-attention layer to enhance the expressiveness of the features. The enhanced features can be concatenated or weighted summed with the original image features to further enrich the feature representation. After the above enhancement processing, the semantically enhanced features of the historical image obtained can better represent the relationship between the image content and the prompt text. These enhanced features will be used as input to the subsequent text generation model to help generate more relevant and accurate text descriptions. By enhancing the output results of the cross-attention layer based on the learnable query vector, the semantic feature representation ability of the historical image can be effectively improved. This process not only improves the model's ability to understand the image content, but also enhances its performance in multimodal generation tasks.

[0073] In some embodiments of the present disclosure, historical prompt text data is obtained, and the historical prompt text data is encoded by the second text encoder to obtain the semantic features of the historical prompt text data. For example, the historical prompt text data refers to text information related to the historical image, which is usually used to guide the model to generate a description that matches the image content. The second text encoder is based on the structure and weights of the first text encoder, and usually adopts the Transformer architecture or other deep learning models. Its main function is to convert the input text data into a high-dimensional semantic feature representation. The pre-processed historical prompt text data is input into the second text encoder, usually in the form of word vectors. The text is first converted into a vector of fixed dimension by an embedding layer for each word. Pre-trained word vectors (such as Word2Vec, GloVe) or randomly initialized embeddings can be used. After the historical prompt text is input, the second text encoder will encode it. The self-attention mechanism in the encoder calculates the relationship between each word in the text and generates a contextual representation of each word. For each word, the attention weight with other words is calculated to capture the global information of the text. The output of the self-attention layer will be passed to a feedforward neural network, which usually consists of two linear transformations and an activation function. After each major step, layer normalization is usually performed to ensure numerical stability and accelerate training. After the above encoding process, the second text encoder will generate a semantic feature representation of the historical prompt text data: the final output is a high-dimensional vector or a set of vectors representing the semantic information of the entire text. These features can capture information such as the context, sentiment, and theme of the text. If the input historical prompt text is a sequence, a fixed-length feature vector is usually obtained by using the last hidden state of the sequence or through a pooling operation (such as maximum pooling or average pooling). The semantic features of the historical prompt text obtained will be used for subsequent tasks such as text generation, feature fusion, etc. These features can help the model better understand the text content, thereby generating a more accurate description related to the image. By obtaining the historical prompt text data and encoding it using the second text encoder, the semantic features of the text can be obtained. This process not only converts the text into a machine-understandable form, but also provides rich context information for subsequent generation tasks.

[0074] In some embodiments of the present disclosure, the semantic enhancement features of the historical image and the semantic features of the historical prompt text data are fused to obtain the fused features of the historical image. Supervised learning is performed based on the fused features of the historical image and the loss function of the text generation task to update the parameters of the second text encoder. For example, the main purpose of fusing the semantic enhancement features of the historical image and the semantic features of the historical prompt text data is to generate a comprehensive feature representation that can capture the information of the image and the text at the same time, thereby providing a richer context for the subsequent text generation task. The fusion process can adopt a variety of methods, for example, the semantic enhancement features of the historical image and the semantic features of the historical prompt text can be directly connected together to form a higher-dimensional feature vector. The two features can also be weighted and summed, and the weights can be learnable parameters or set according to prior knowledge. And the features can also be fused through interactive operations (for example, through an attention mechanism), which allows the model to adaptively learn how to combine image and text features. The above fusion method will generate a fused feature of the historical image, and this feature vector will be used as input in the subsequent text generation task. The fused feature not only contains the semantic information of the image, but also combines the contextual information of the text, making the generated text more relevant and accurate. In the text generation task, supervised learning methods are usually used to update model parameters. The following is a detailed description of this process: The goal of the text generation task: Generate text descriptions related to historical images based on the fused features of historical images.

[0075] Loss function definition: The loss function is used to measure the gap between the generated text and the real text. Common loss functions include: Cross entropy loss: used for classification tasks, calculates the difference between the probability distribution of each generated word and the target word. Parameter update: By calculating the gradient of the loss function, the parameters of the second text encoder are updated using an optimization algorithm (such as Adam, SGD, etc.). Through the above process, the parameters of the second text encoder will be adjusted according to the quality of the generated text. This supervised learning method enables the model to gradually optimize its understanding of the relationship between images and texts, thereby improving the relevance and quality of the generated text. By fusing the semantic enhancement features of historical images and the semantic features of historical prompt text data, a comprehensive historical image fusion feature can be obtained. Supervised learning based on this fusion feature and the loss function of the text generation task can effectively update the parameters of the second text encoder, thereby improving the performance of the model in the multimodal text generation task.

[0076] Figure 6 FIG. 1 is a flowchart of another method for generating text data provided by an embodiment of the present disclosure. Figure 6 As shown, before the semantic features of the image data are processed by the first multi-layer perceptron, the method further includes steps S610 to S650.

[0077] In step S610, the target semantic features of the historical image output by the multi-granularity image encoder are processed by a first multi-layer perceptron to obtain target semantic enhancement features of the historical image.

[0078] In step S620, the semantic features of the historical prompt text data output by the first text encoder are processed by a second multi-layer perceptron to obtain a first semantic enhancement feature of the historical prompt text data.

[0079] In step S630, the semantic features of the historical prompt text data output by the second text encoder are processed by a third multi-layer perceptron to obtain second semantic enhancement features of the historical prompt text data.

[0080] In step S640, the target semantic enhancement feature of the historical image, the first semantic enhancement feature of the historical prompt text data, and the second semantic enhancement feature of the historical prompt text data are fused to obtain the target semantic fusion feature.

[0081] In step S650, the target semantic fusion feature is input into the large language model, and the parameters of the first multilayer perceptron, the parameters of the second multilayer perceptron, and the parameters of the third multilayer perceptron are updated based on the target semantic fusion feature through the loss function in the large language model.

[0082] The method can fuse the target semantic enhancement features of the historical image, the first semantic enhancement features of the historical prompt text data, and the second semantic enhancement features. The target semantic fusion features obtained can make full use of multimodal information and enhance the performance of the model. After the feature is input into the large language model, the gradient calculation and parameter update are performed through the loss function, thereby improving the performance of the first multi-layer perceptron, the second multi-layer perceptron, and the third multi-layer perceptron, and finally achieving a more accurate multimodal learning task.

[0083] In some embodiments of the present disclosure, after the multi-granularity image encoder and the second text encoder are trained, the first multi-layer perceptron, the second multi-layer perceptron, and the third multi-layer perceptron can be introduced to fine-tune the entire model to further improve the accuracy of the model. For example, the target semantic features of the historical image output by the multi-granularity image encoder are processed by the first multi-layer perceptron to obtain the target semantic enhancement features of the historical image. The semantic features of the historical prompt text data output by the first text encoder are processed by the second multi-layer perceptron to obtain the first semantic enhancement features of the historical prompt text data. The semantic features of the historical prompt text data output by the second text encoder are processed by the third multi-layer perceptron to obtain the second semantic enhancement features of the historical prompt text data. The target semantic enhancement features of the historical image, the first semantic enhancement features of the historical prompt text data, and the second semantic enhancement features of the historical prompt text data are fused to obtain the target semantic fusion features. The target semantic fusion features are input into the large language model, and the parameters of the first multi-layer perceptron, the second multi-layer perceptron, and the third multi-layer perceptron are updated based on the target semantic fusion features through the loss function in the large language model.

[0084] In some embodiments of the present disclosure, the target semantic features of the historical image output by the multi-granularity image encoder are processed by the first multi-layer perceptron to obtain the target semantic enhancement features of the historical image. For example, the target semantic features of the historical image are feature vectors generated by the multi-granularity image encoder, which usually contain high-dimensional representations of the image and can capture important information and semantic content in the image. These features may be obtained after being processed by a convolution layer, a pooling layer, and an activation function, and have a certain degree of abstraction. The first multi-layer perceptron (MLP) is usually composed of multiple fully connected layers, each of which may include the following components: an input layer, the number of nodes in the input layer is equal to the dimension of the target semantic features of the historical image. A hidden layer, which may include one or more hidden layers, and a common structure is two to three layers. The number of nodes in each hidden layer can be adjusted according to the complexity of the task, and is usually reduced layer by layer. Activation function: A nonlinear activation function (such as ReLU, Leaky ReLU, or ELU) is used to introduce nonlinear feature transformations, and ReLU is the most commonly used choice. Output nodes, the number of nodes in the output layer can be set according to the required feature dimensions, and usually matches the dimension of the target semantic enhancement features. After inputting the target semantic features of the historical image, the processing flow of the first multi-layer perceptron is as follows: the input features of each layer are linearly transformed through the weight matrix, and the activation function is applied to the result of the linear transformation to obtain the activation output of the current layer. Through multi-layer nonlinear transformation, the first multi-layer perceptron can extract more discriminative features, so that the target semantic features of the historical image are enhanced. This enhanced feature can better represent the semantic information of the image. After being processed by the first multi-layer perceptron, the obtained feature vector is an enhanced version of the input feature, containing richer semantic information. These features will be used for subsequent feature fusion or input into other models (such as large language models). By processing the target semantic features of the historical image output by the multi-granularity image encoder through the first multi-layer perceptron, the semantic information of the image can be effectively extracted and enhanced. This process is implemented through a multi-layer fully connected structure and a nonlinear activation function. The target semantic enhanced features of the historical image finally obtained will provide more powerful feature support for subsequent multimodal learning tasks.

[0085] Based on the foregoing embodiment, the second multilayer perceptron and the third multilayer perceptron have a similar processing flow to the first multilayer perceptron, through which the first semantic enhancement feature of the historical prompt text data and the second semantic enhancement feature of the historical prompt text data can be obtained.

[0086] In some embodiments of the present disclosure, the target semantic enhancement feature of the historical image, the first semantic enhancement feature of the historical prompt text data, and the second semantic enhancement feature of the historical prompt text data are fused to obtain the target semantic fusion feature. The target semantic fusion feature is input into the large language model, and the parameters of the first multilayer perceptron, the second multilayer perceptron, and the third multilayer perceptron are updated based on the target semantic fusion feature through the loss function in the large language model. For example, the target semantic enhancement feature of the historical image is the image feature processed by the first multilayer perceptron, which contains rich semantic information of the image. The first semantic enhancement feature of the historical prompt text data is the text feature processed by the second multilayer perceptron, which can better represent the semantic information of the text. The second semantic enhancement feature of the historical prompt text data is the text feature processed by the third multilayer perceptron, which can better represent the semantic information of the text. Feature fusion is to combine features from different sources so that the model can obtain more comprehensive information from multimodal data. The steps of fusion processing are as follows: Splicing operation: splicing the three feature vectors to form a large feature vector. Linear transformation: the spliced ​​features can be transformed by a linear layer to better adapt to the input of the subsequent model. The fused target semantic features are input into a large language model (such as GPT, BERT, etc.) for further text generation or understanding tasks. The large language model calculates a loss function according to its tasks (such as text generation, classification, etc.). The loss function is usually based on the difference between the predicted output of the model and the true label. The model parameters are updated through the back propagation algorithm, and the specific steps are as follows: Calculate the gradient: Calculate the gradient of the loss function to each model parameter by the chain rule, including the parameters of the first multi-layer perceptron, the second multi-layer perceptron, and the third multi-layer perceptron. Parameter update: Use an optimization algorithm (such as SGD, Adam, etc.) to update the parameters of each multi-layer perceptron. By fusing the target semantic enhancement features of the historical image, the first semantic enhancement features of the historical prompt text data, and the second semantic enhancement features, the target semantic fusion features obtained can make full use of multimodal information and enhance the performance of the model. After the feature is input into the large language model, the gradient calculation and parameter update are performed through the loss function, thereby improving the performance of the first multi-layer perceptron, the second multi-layer perceptron, and the third multi-layer perceptron, and finally achieving a more accurate multimodal learning task.

[0087] Figure 7 It is a block diagram of a text data generating device provided by an embodiment of the present disclosure.

[0088] like Figure 7 As shown, the text data generating device 700 includes an acquisition module 710 , a first processing module 720 , a second processing module 730 , a third processing module 740 , a fourth processing module 750 , a splicing module 760 and a text generating module 770 .

[0089] Specifically, the acquisition module 710 is used to acquire image data and prompt text data.

[0090] The first processing module 720 is used to perform semantic processing on the image data through a multi-granularity image encoder to obtain semantic features of the image data. The multi-granularity image encoder is trained based on historical image data and real semantic text data of the historical image data.

[0091] The second processing module 730 is used to process the semantic features of the image data through a first multi-layer perceptron to obtain a first semantic feature.

[0092] The third processing module 740 is used to encode the prompt text data through the first text encoder, and process the result of the first text encoder through the second multi-layer perceptron to obtain a second semantic feature. The first text encoder is trained based on the real semantic text data of the historical image data and the result of the multi-granularity image encoder output.

[0093] The fourth processing module 750 is used to encode the prompt text data through the second text encoder, and process the result of the second text encoder through a third multi-layer perceptron to obtain a third semantic feature. The second text encoder is trained based on historical image data and historical prompt text data.

[0094] The splicing module 760 is used to splice the first semantic feature, the second semantic feature, and the third semantic feature to obtain target semantic data of the image data.

[0095] The text generation module 770 is used to process the target semantic data of the image data through a large language model to obtain feedback text data of the image data.

[0096] The text data generating device 700 can perform semantic processing on the image data through a multi-granularity image encoder, and can extract richer and more detailed semantic features. This training method based on historical image data and real semantic text data enables the model to better understand the image content, thereby improving the depth of understanding of complex visual scenes. By combining the output results of the first text encoder with the multi-granularity image encoder, the difference between different feature representations can be effectively reduced, and the alignment effect of image and text features is improved. This consistency helps the model to provide more accurate semantic feedback when processing multimodal data. By splicing the first, second, and third semantic features, effective integration of features of different dimensions is achieved. This multi-level feature fusion can more comprehensively reflect the target semantics of image data and improve the performance of the model in complex tasks. When processing low-resolution and multi-scale images, the present invention enhances the robustness of the model to image diversity and complexity through a multi-granularity image encoder. This enables the model to better adapt to image data under different scenes and conditions in practical applications. By processing the target semantic data of image data through a large language model, more accurate and relevant feedback text data can be generated. This high-quality text feedback not only improves the user experience, but also provides a more reliable information basis for subsequent applications.

[0097] In some embodiments of the present disclosure, before semantic processing is performed on image data through a multi-granularity image encoder, the text data generating device 700 is further used to: obtain historical image data and input the historical image data into the multi-granularity image encoder, wherein the multi-granularity image encoder includes a contrastive language-image pre-trained image encoder, a first convolutional layer, a visual image encoder, a second convolutional layer, an open target detector, and an average pooling layer; encode the historical image data through the contrastive language-image pre-trained image encoder, and convolve the result of the contrastive language-image pre-trained image encoder through the first convolutional layer to obtain a first convolutional semantic feature of the historical image; encode the historical image data through the visual image encoder, and convolve the result output by the visual image encoder through the second convolutional layer to obtain a second convolutional semantic feature of the historical image; detect the historical image data through an open target detector, and pool the result output by the open target detector through an average pooling layer to obtain at least one pooled semantic feature of the historical image; determine the target semantic feature of the historical image according to the first convolutional semantic feature of the historical image, the second convolutional semantic feature of the historical image, and the at least one pooled semantic feature of the historical image.

[0098] In some embodiments of the present disclosure, before encoding the prompt text data through the first text encoder, the text data generating device 700 is also used to: obtain the real semantic text data of the historical image data; encode the real semantic text data of the historical image data through the first text encoder to obtain the semantic features of the real semantic text data; calculate the contrast loss based on the target semantic features of the historical image and the semantic features of the real semantic text data through the contrast loss function; and update the parameters in the multi-granularity image encoder and the parameters in the first text encoder based on the contrast loss.

[0099] In some embodiments of the present disclosure, the multi-granularity image encoder also includes a cross-attention layer, a fourth multi-layer perceptron and a fifth multi-layer perceptron; determining the target semantic feature of the historical image according to the first convolutional semantic feature of the historical image, the second convolutional semantic feature of the historical image, and at least one pooled semantic feature of the historical image includes: processing the first convolutional semantic feature through the cross-attention layer, and processing the output result of the cross-attention layer through the fourth multi-layer perceptron to obtain the first historical target semantic feature; processing the second convolutional semantic feature through the cross-attention layer, and processing the output result of the cross-attention layer through the fourth multi-layer perceptron to obtain the second historical target semantic feature; splicing at least one pooled semantic feature, and processing the splicing result through the fifth multi-layer perceptron to obtain the third historical target semantic feature; splicing the first historical target semantic feature, the first historical target semantic feature, and the first historical target semantic feature to obtain the target semantic feature of the historical image.

[0100] In some embodiments of the present disclosure, before encoding the prompt text data through the second text encoder, the text data generating device 700 is also used to: enhance the output result of the cross-attention layer based on the learnable query vector to obtain the semantic enhancement features of the historical image; obtain the historical prompt text data, and encode the historical prompt text data through the second text encoder to obtain the semantic features of the historical prompt text data; fuse the semantic enhancement features of the historical image and the semantic features of the historical prompt text data to obtain the fused features of the historical image; and perform supervised learning based on the fused features of the historical image and the loss function of the text generation task to update the parameters of the second text encoder.

[0101] In some embodiments of the present disclosure, before processing the semantic features of image data by the first multi-layer perceptron, the text data generating device 700 is also used to: process the target semantic features of the historical image output by the multi-granularity image encoder by the first multi-layer perceptron to obtain the target semantic enhancement features of the historical image; process the semantic features of the historical prompt text data output by the first text encoder by the second multi-layer perceptron to obtain the first semantic enhancement features of the historical prompt text data; process the semantic features of the historical prompt text data output by the second text encoder by the third multi-layer perceptron to obtain the second semantic enhancement features of the historical prompt text data; fuse the target semantic enhancement features of the historical image, the first semantic enhancement features of the historical prompt text data, and the second semantic enhancement features of the historical prompt text data to obtain the target semantic fusion features; input the target semantic fusion features into the large language model, and update the parameters of the first multi-layer perceptron, the parameters of the second multi-layer perceptron, and the parameters of the third multi-layer perceptron based on the target semantic fusion features through the loss function in the large language model.

[0102] In some embodiments of the present disclosure, the first processing module 720 is configured to: encode the image data through a contrastive language-image pre-trained image encoder, and convolve the result of the contrastive language-image pre-trained image encoder through a first convolutional layer to obtain a first convolutional semantic feature of the image; encode the image data through a visual image encoder, and convolve the result output by the visual image encoder through a second convolutional layer to obtain a second convolutional semantic feature of the image; detect the image data through an open object detector, and pool the result output by the open object detector through an average pooling layer to obtain at least one pooled semantic feature of the image; determine the target semantic feature of the image based on the first convolutional semantic feature of the image, the second convolutional semantic feature of the image, and at least one pooled semantic feature of the image.

[0103] Figure 8 Schematic diagram of an electronic device 8 provided by an embodiment of the present disclosure. Figure 8 As shown, the electronic device 8 of this embodiment includes: a processor 801, a memory 802, and a computer program 803 stored in the memory 802 and executable on the processor 801. When the processor 801 executes the computer program 803, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 801 executes the computer program 803, the functions of the modules in the above-mentioned device embodiments are implemented.

[0104] The electronic device 8 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 8 may include, but is not limited to, a processor 801 and a memory 802. Those skilled in the art will appreciate that Figure 8The electronic device 8 is merely an example and does not limit the electronic device 8 . The electronic device 8 may include more or fewer components than those shown in the figure, or different components.

[0105] The processor 801 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0106] The memory 802 may be an internal storage unit of the electronic device 8, for example, a hard disk or memory of the electronic device 8. The memory 802 may also be an external storage device of the electronic device 8, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 8. The memory 802 may also include both an internal storage unit of the electronic device 8 and an external storage device. The memory 802 is used to store computer programs and other programs and data required by the electronic device.

[0107] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0108] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0109] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. A method for generating text data, characterized in that: include: Get image data and prompt text data; Performing semantic processing on the image data by a multi-granularity image encoder to obtain semantic features of the image data, wherein the multi-granularity image encoder is trained based on historical image data and real semantic text data of the historical image data; Processing the semantic features of the image data by a first multi-layer perceptron to obtain a first semantic feature; The prompt text data is encoded by a first text encoder, and a result of the first text encoder is processed by a second multi-layer perceptron to obtain a second semantic feature, wherein the first text encoder is trained based on the real semantic text data of the historical image data and the result output by the multi-granularity image encoder; The prompt text data is encoded by a second text encoder, and a result of the second text encoder is processed by a third multi-layer perceptron to obtain a third semantic feature, wherein the second text encoder is trained based on the historical image data and the historical prompt text data; Performing splicing processing on the first semantic feature, the second semantic feature, and the third semantic feature to obtain target semantic data of the image data; The target semantic data of the image data is processed by a large language model to obtain feedback text data of the image data.

2. The method according to claim 1, characterized in that Before the semantic processing of the image data by the multi-granularity image encoder, the method further comprises: Acquire the historical image data, and input the historical image data into the multi-granularity image encoder, wherein the multi-granularity image encoder includes a contrastive language-image pre-trained image encoder, a first convolutional layer, a visual image encoder, a second convolutional layer, an open object detector, and an average pooling layer; The historical image data is encoded by the contrastive language-image pre-trained image encoder, and the result of the contrastive language-image pre-trained image encoder is convolved by the first convolutional layer to obtain a first convolutional semantic feature of the historical image; The historical image data is encoded by the visual image encoder, and a result output by the visual image encoder is convolved by the second convolution layer to obtain a second convolution semantic feature of the historical image; Performing detection processing on the historical image data by the open object detector, and performing pooling processing on the result output by the open object detector by the average pooling layer to obtain at least one pooled semantic feature of the historical image; A target semantic feature of the historical image is determined according to a first convolutional semantic feature of the historical image, a second convolutional semantic feature of the historical image, and at least one pooled semantic feature of the historical image.

3. The method according to claim 2, characterized in that Before encoding the prompt text data by the first text encoder, the method further includes: Acquire real semantic text data of the historical image data; The real semantic text data of the historical image data is encoded by the first text encoder to obtain semantic features of the real semantic text data; Calculating contrast loss based on target semantic features of the historical image and semantic features of the real semantic text data through a contrast loss function; Parameters in the multi-granularity image encoder and parameters in the first text encoder are updated based on the contrast loss.

4. The method according to claim 2, characterized in that: The multi-granularity image encoder further comprises a cross attention layer, a fourth multi-layer perceptron and a fifth multi-layer perceptron; Determining a target semantic feature of the historical image according to the first convolution semantic feature of the historical image, the second convolution semantic feature of the historical image, and at least one pooling semantic feature of the historical image includes: Processing the first convolutional semantic feature through the cross attention layer, and processing the output result of the cross attention layer through the fourth multi-layer perceptron to obtain a first historical target semantic feature; Processing the second convolutional semantic feature through the cross attention layer, and processing the output result of the cross attention layer through the fourth multi-layer perceptron to obtain a second historical target semantic feature; Performing splicing processing on at least one of the pooled semantic features, and processing the splicing result through the fifth multi-layer perceptron to obtain a third historical target semantic feature; The first historical target semantic feature, the first historical target semantic feature, and the first historical target semantic feature are spliced ​​to obtain the target semantic feature of the historical image.

5. The method according to claim 4, characterized in that Before encoding the prompt text data by the second text encoder, the method further includes: Enhance the output result of the cross attention layer based on the learnable query vector to obtain the semantic enhancement feature of the historical image; Acquire historical prompt text data, and encode the historical prompt text data by the second text encoder to obtain semantic features of the historical prompt text data; fusing the semantic enhancement features of the historical image and the semantic features of the historical prompt text data to obtain fused features of the historical image; Supervised learning is performed based on the fused features of the historical images and the loss function of the text generation task to update the parameters of the second text encoder.

6. The method according to claim 5, characterized in that Before processing the semantic features of the image data by the first multi-layer perceptron, the method further includes: Processing the target semantic features of the historical image output by the multi-granularity image encoder by the first multi-layer perceptron to obtain the target semantic enhancement features of the historical image; Processing the semantic features of the historical prompt text data output by the first text encoder through the second multi-layer perceptron to obtain a first semantic enhancement feature of the historical prompt text data; Processing the semantic features of the historical prompt text data output by the second text encoder by the third multi-layer perceptron to obtain a second semantic enhancement feature of the historical prompt text data; fusing the target semantic enhancement feature of the historical image, the first semantic enhancement feature of the historical prompt text data, and the second semantic enhancement feature of the historical prompt text data to obtain a target semantic fusion feature; The target semantic fusion feature is input into the large language model, and the parameters of the first multilayer perceptron, the parameters of the second multilayer perceptron, and the parameters of the third multilayer perceptron are updated based on the target semantic fusion feature through the loss function in the large language model.

7. The method according to claim 2, characterized in that The image data is semantically processed by a multi-granularity image encoder to obtain semantic features of the image data including: The image data is encoded by the contrastive language-image pre-trained image encoder, and the result of the contrastive language-image pre-trained image encoder is convolved by the first convolutional layer to obtain a first convolutional semantic feature of the image; Encoding the image data by the visual image encoder, and convolving the result output by the visual image encoder by the second convolution layer to obtain a second convolution semantic feature of the image; Performing detection processing on the image data by using the open object detector, and performing pooling processing on the result output by the open object detector by using an average pooling layer to obtain at least one pooled semantic feature of the image; A target semantic feature of the image is determined according to the first convolution semantic feature of the image, the second convolution semantic feature of the image, and at least one pooling semantic feature of the image.

8. A text data generating device, characterized in that: include: An acquisition module, used for acquiring image data and prompt text data; A first processing module, configured to perform semantic processing on the image data through a multi-granularity image encoder to obtain semantic features of the image data, wherein the multi-granularity image encoder is trained based on historical image data and real semantic text data of the historical image data; A second processing module, configured to process the semantic features of the image data through a first multi-layer perceptron to obtain a first semantic feature; A third processing module is used to encode the prompt text data through a first text encoder, and process the result of the first text encoder through a second multi-layer perceptron to obtain a second semantic feature, wherein the first text encoder is trained based on the real semantic text data of the historical image data and the result output by the multi-granularity image encoder; a fourth processing module, configured to encode the prompt text data through a second text encoder, and process the result of the second text encoder through a third multi-layer perceptron to obtain a third semantic feature, wherein the second text encoder is trained based on the historical image data and the historical prompt text data; a splicing module, configured to splice the first semantic feature, the second semantic feature, and the third semantic feature to obtain target semantic data of the image data; The text generation module is used to process the target semantic data of the image data through a large language model to obtain feedback text data of the image data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images