Text generation methods, apparatuses, electronic devices, computer-readable storage media, and computer program products
By performing multi-scale image processing and feature fusion, the problem of low answer accuracy in single-scale image question answering tasks is solved, achieving higher text generation accuracy.
Patent Information
- Application Number
- CN202510603033.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-08
- Publication Date
- 2026-11-10
AI Technical Summary
Existing single-scale image question answering tasks result in low accuracy of generated answers and fail to effectively utilize image information.
By performing multi-scale scaling on the image, features at multiple scales are extracted, and the features are then processed by channel shuffling and fusion, combined with weighted fusion to generate text.
It improves the accuracy of image content understanding and text generation, captures multi-granular information of images, enhances feature representation capabilities, and improves the accuracy of cross-modal text reasoning.
Smart Images

Figure CN123068699A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a text generation method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology
[0002] Related technologies connect a single image to a large language model through visual encoding and multilayer perceptrons, thereby fusing image information with the language model to achieve better document understanding and generation; however, single-scale image question answering tasks can only generate answers based on image information at the current scale, resulting in low accuracy of the generated answers. Summary of the Invention
[0003] This application provides a text generation method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can effectively improve the accuracy of image content understanding and thus improve the accuracy of text generation.
[0004] The technical solution of this application embodiment is implemented as follows:
[0005] This application provides a text generation method, including:
[0006] The first image is subjected to multi-scale scaling to obtain multiple second images, and feature extraction is performed on each second image to obtain the features of each second image;
[0007] Channel shuffling is performed on the channels of the first feature to obtain the first processing result of the first feature, and channel shuffling is performed on the channels of the second feature to obtain the second processing result of the second feature.
[0008] Wherein, the first feature is any feature of the second image, and the second feature is a feature of the second image that is different from the first feature;
[0009] The first processing result and the second processing result are fused to obtain a third feature, and the third feature is mapped to the same feature space as the first feature to obtain a fourth feature corresponding to the first feature.
[0010] Based on the weight of the first feature, the fourth feature corresponding to the first feature is fused to obtain the fifth feature;
[0011] Based on the fifth feature and the first text corresponding to the first image, the second text is generated.
[0012] This application provides a text generation apparatus, including:
[0013] The first processing module is used to perform multi-scale scaling processing on the first image to obtain multiple second images, and to perform feature extraction processing on each second image to obtain the features of each second image;
[0014] The second processing module is used to perform channel shuffling on the channels of the first feature to obtain a first processing result of the first feature, and to perform channel shuffling on the channels of the second feature to obtain a second processing result of the second feature; wherein, the first feature is any feature of the second image, and the second feature is a feature of the second image that is different from the first feature;
[0015] The third processing module is used to fuse the first processing result and the second processing result to obtain a third feature, and to map the third feature to the same feature space as the first feature to obtain a fourth feature corresponding to the first feature;
[0016] The fusion module is used to perform fusion processing on the fourth feature corresponding to the first feature based on the weight of the first feature to obtain the fifth feature;
[0017] The generation module is used to generate second text based on the fifth feature and the first text corresponding to the first image.
[0018] This application provides an electronic device, including:
[0019] Memory is used to store executable instructions or computer programs.
[0020] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the text generation method provided in the embodiments of this application.
[0021] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the text generation method provided in this application.
[0022] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the text generation method provided in this application.
[0023] The embodiments of this application have the following beneficial effects:
[0024] By scaling the first image at different ratios, second images at multiple scales are generated. Feature extraction is performed on each second image to obtain its features, capturing multi-granular information and avoiding information loss at a single scale. Next, channel shuffling is performed on the channels of the first feature to obtain a first processing result. Similarly, channel shuffling is performed on the channels of the second feature to obtain a second processing result. Here, the first feature is any feature of the second image, and the second feature is a feature in the second image that differs from the first feature. By shuffling the channel order of the features, the independence of features between channels can be broken, allowing for greater differentiation between features in different channels. Feature information can be exchanged and fused to uncover potential information between features. Subsequently, the results of the first and second processing steps are fused to obtain a third feature, which more comprehensively and accurately describes the features, enhancing their expressive power. This third feature is then mapped to the same feature space as the first feature, resulting in a fourth feature corresponding to the first feature, ensuring feature consistency and compatibility. Next, based on the weights of each first feature, the fourth features corresponding to the first feature are fused to obtain a fifth feature, integrating cross-scale information association and improving the feature's information-expressive power. Finally, a more reasonable and accurate second text is generated based on the fifth feature and the first text. In summary, through multi-scale processing, cross-scale association, and weight-based fusion, the ability to capture image details in text generation tasks is significantly improved, thereby further enhancing the accuracy of cross-modal text reasoning. Attached Figure Description
[0025] Figure 1 This is a schematic diagram of the structure of the document large model provided in the embodiments of this application;
[0026] Figure 2 This is a schematic diagram of the architecture of the text generation system 100 provided in an embodiment of this application;
[0027] Figure 3 This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application;
[0028] Figure 4 This is a flowchart illustrating the text generation method provided in an embodiment of this application;
[0029] Figure 5 This is a flowchart illustrating the text generation method provided in an embodiment of this application;
[0030] Figure 6 This is a flowchart illustrating the text generation method provided in an embodiment of this application;
[0031] Figure 7 This is a model framework diagram of the text generation method provided in the embodiments of this application;
[0032] Figure 8 This is an architecture diagram of the feature fusion module provided in the embodiments of this application;
[0033] Figure 9 This is an architecture diagram of the mutual information module provided in the embodiments of this application. Detailed Implementation
[0034] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0035] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0036] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.
[0037] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.
[0038] In the following description, the terms “first, second, ...” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first, second, ...” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.
[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0040] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.
[0041] 1) Document Big Model: The document big model is an advanced artificial intelligence model that integrates Natural Language Processing (NLP) and Computer Vision (CV) technologies. This model is specifically designed to understand and process information in documents, whether scanned paper documents or electronic documents. The document big model can comprehensively analyze the text, images, tables, and layout in a document, thereby achieving the understanding, extraction, and generation of document content.
[0042] 2) Vision Transformer (ViT): A vision transformer is a deep learning model that applies the Transformer architecture to computer vision tasks. It transforms images into sequential data and uses a self-attention mechanism to capture global information.
[0043] 3) Multi-Layer Perceptron (MLP): MLP is a supervised learning model based on artificial neural networks, used to solve complex nonlinear problems. It can be widely applied to classification and regression tasks. A multi-layer perceptron is a feedforward artificial neural network composed of multiple neurons (neural nodes) arranged in a hierarchical structure, including an input layer, hidden layers, and an output layer. Neurons between layers are connected by weights, and information propagates sequentially from the input layer to the output layer without feedback connections.
[0044] In related technologies, see Figure 1 , Figure 1 This is a schematic diagram of the structure of the document large model provided in the embodiments of this application, such as... Figure 1 As shown, the document big model directly processes the image through the visual encoding module of the Vision Transformer (ViT), and then connects the processing result of the video encoding module with the big language model through a multilayer perceptron to obtain the document big model. However, directly inputting the image into the ViT model can only obtain information about the image at the current scale. Moreover, if only a single-scale image is input, the model may miss some information, resulting in lower accuracy of the generated answer.
[0045] Based on this, embodiments of this application provide a text generation method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can effectively improve the accuracy of image content analysis and text content. The electronic device provided in this application embodiment can be implemented as a server, or implemented collaboratively by a server and a terminal. The following description uses an example of a text generation method provided in this application embodiment implemented collaboratively by a server and a terminal.
[0046] For example, see Figure 2 , Figure 2 This is a schematic diagram of the architecture of the text generation system 100 provided in this application embodiment. To support a text generation application, such as... Figure 2 As shown, the text generation system 100 includes: a server 200, a network 300, and a terminal 400. The terminal 400 is connected to the server 200 through the network 300. The network 300 can be a local area network (LAN), a wide area network (WAN), or a combination of both.
[0047] In some embodiments, a user inputs a first image and corresponding first text through a terminal 400, and transmits the first image and corresponding first text to a server 200 via a network 300. Subsequently, the server 200 performs multi-scale scaling on the first image to obtain multiple second images, and performs feature extraction on each second image to obtain the features of each second image. Then, the server 200 generates second text based on the features of each second image and the corresponding first text. Finally, the server 200 can send the generated second text to the terminal 400 via the network 300 and display it on the terminal 400.
[0048] The text generation method provided in this application can be applied to various scenarios that require text information generation, such as the following scenarios:
[0049] 1) In the field of education, taking interactive teaching aids as an example, students upload plant pictures and ask questions (such as "What is the name of the plant in the picture?"). The system combines visual and text analysis to generate answers, which can enhance the interactivity of teaching. It can also recognize experimental equipment through images and answer operational questions (such as "How to connect the circuit?"). Taking personalized learning as an example, based on the handwritten images of solutions uploaded by students, the system can identify incorrect steps and provide targeted analysis. It can also generate lesson plans that combine text and images to adapt to different learning stages of students.
[0050] 2) In the field of healthcare, taking medical image analysis as an example, cell microscopy can automatically identify cell types, numbers, and distribution. It can also generate diagnostic reports based on CT / MRI images to help doctors determine the location and nature of lesions. In the field of health monitoring, it can identify skin lesion images and provide preliminary diagnostic suggestions (such as rash classification). It can also analyze infant behavioral images (such as crying postures) to infer needs (such as hunger, discomfort, etc.).
[0051] 3) In daily life consumption, taking smart clothing and shopping as an example, users can upload clothing pictures to generate matching suggestions (such as "What clothes are suitable to match this earring?"), or identify product attributes and answer usage questions based on product images; in home security, users can identify abnormal behavior (such as an elderly person falling) through home detection images and trigger alarms, or obtain device repair guidance by taking pictures of device status.
[0052] 4) In industrial manufacturing, taking production process optimization as an example, it can identify images of production line equipment, automatically generate fault diagnosis reports and maintenance plans, and detect defects (such as part size deviations) through product images in the quality inspection process; taking autonomous driving as an example, it can analyze road images in real time, answer complex road conditions (such as "type of obstacle ahead"), and predict pedestrian or vehicle behavior to improve decision-making safety.
[0053] 5) Public services, taking government affairs and law as an example, can be used for document image verification (such as ID card authenticity identification) and answering questions about the application process, as well as extracting the content of legal document images and reviewing compliance; taking scientific research data analysis as an example, can be used to analyze the charts in academic papers (such as "experimental data trend descriptions"), as well as to analyze astronomical or geological images (such as star classification, rock composition identification, etc.).
[0054] Example, Figure 2 The server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal 400 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, in-vehicle terminal, etc., but is not limited to these. The terminal 400 and server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.
[0055] The structure of the electronic device provided in the embodiments of this application will be described below. Taking the electronic device as a server as an example, see... Figure 3 , Figure 3 This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application. Figure 3 The illustrated electronic device 500 includes at least one processor 510, a memory 540, and at least one network interface 520. The various components in the electronic device 500 are coupled together via a bus system 530. It is understood that the bus system 530 is used to implement communication between these components. In addition to a data bus, the bus system 530 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 3 The general labeled all buses as Bus System 530.
[0056] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.
[0057] The memory 540 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 540 may optionally include one or more storage devices physically located away from the processor 510.
[0058] The memory 540 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 540 described in this application embodiment is intended to include any suitable type of memory.
[0059] In some embodiments, memory 540 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.
[0060] Operating system 541 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks;
[0061] The network communication module 542 is used to reach other computing devices via one or more (wired or wireless) network interfaces 620, exemplary network interfaces 620 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.
[0062] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 3 A text generation device 543 stored in memory 540 is shown. It can be software in the form of programs and plug-ins, including the following software modules: a first processing module 5431, a second processing module 5432, a third processing module 5433, a fusion module 5434, and a generation module 5435. These modules are logically related and can therefore be arbitrarily combined or further split according to the functions they implement.
[0063] The text generation method provided in this application will be specifically described below with reference to the exemplary application and implementation of the server provided in the embodiments of this application.
[0064] See Figure 4 , Figure 4 This is a flowchart illustrating the text generation method provided in this application embodiment, which will be combined with... Figure 4 The steps shown are explained.
[0065] In step 101, the first image is subjected to multi-scale scaling to obtain multiple second images.
[0066] It should be noted that multi-scale scaling of images can be achieved by using dedicated image processing software (such as Photoshop), by using related programming code (such as Python + OpenCV), or by using custom scaling algorithms (such as nearest neighbor interpolation). No specific limitations are made here.
[0067] As an example, assuming the resolution of the first image is 1024*768, and the scaling scales are 1.0, 0.5, and 0.25 respectively, scaling the first image according to the scaling scale will result in a second image with a scaling scale of 1.0 having a resolution of 1024*768, a second image with a scaling scale of 0.5 having a resolution of 512*384, and a second image with a scaling scale of 0.25 having a resolution of 256*192.
[0068] In step 102, feature extraction processing is performed on each of the second images to obtain the features of each of the second images.
[0069] It should be noted that a single network can be used to adjust and extract features for each second image separately to obtain the features of each second image, or different networks can be used for feature extraction for each second image to obtain the features of each second image. No specific limitation is made here.
[0070] As an example, taking a single network as an example, suppose the resolutions of the second images generated by multi-scale scaling are 256*256, 512*512, and 1024*1024, respectively. The second images of different sizes can be directly input into the same network (e.g., ResNet-50) for feature extraction, thereby obtaining the features of each second image. Taking multiple networks as an example, suppose the resolutions of the second images generated by multi-scale scaling are 256*256, 512*512, and 1024*1024, respectively. A lightweight network (e.g., MobileNetV3) can be used to extract features from the second image with a resolution of 256*256, a medium-depth network (e.g., ResNet-50) can be used to extract features from the second image with a resolution of 512*512, and a deep network (e.g., ResNet-152) can be used to extract features from the second image with a resolution of 1024*1024, respectively, thereby obtaining the features of each second image.
[0071] In some embodiments, step 102 described above can be implemented as follows: for each of the second images, a first neural network corresponding to the size of the second image is invoked to perform feature extraction processing on the second image to obtain the features of the second image; wherein the spatial complexity of the first neural network is negatively correlated with the size of the second image. Thus, by designing the spatial complexity of the neural network to be negatively correlated with the size of the second image, a balance is achieved between computational resources, feature quality, and hardware adaptability. This approach is effectively applicable to scenarios with mixed multi-scale data and limited resources, providing an efficient and flexible feature extraction method for image processing tasks.
[0072] It should be noted that the space complexity of a neural network is used to characterize the number of parameters in the model. That is, the more parameters a model has, the more data is required to train the model, and the greater the space complexity of the corresponding neural network.
[0073] As an example, suppose there are two second images of different sizes: a small image (resolution 256*256) and a large image (resolution 512*512). Based on the size, the small image corresponds to a neural network with high spatial complexity (e.g., ResNet-152), while the large image corresponds to a neural network with low spatial complexity (e.g., MobileNetV3). Features are then extracted from the images of different sizes using their respective neural networks to obtain the features of the second image.
[0074] In step 103, the channels of the first feature are subjected to channel shuffling to obtain the first processing result of the first feature.
[0075] Here, the first feature is the feature of any second image.
[0076] It's important to note that channel shuffling is a key operation in the ShuffleNet network architecture, designed to optimize computational efficiency and feature fusion. ShuffleNet reduces the computational cost of 1×1 convolutions and improves model efficiency and performance through grouped convolutions and channel shuffling. The main purpose of channel shuffling is to allow features from different convolutional groups to be alternately used as input features for subsequent networks, thereby enhancing the overall representability of the network, reducing computational complexity, and ensuring network quality. Through channel shuffling, features from different groups can be fused together in subsequent convolutional operations, improving feature diversity and expressive power.
[0077] Here, the specific implementation process of channel shuffling includes: First, rearranging the dimensions of the input channels to divide them into multiple groups; then, transposing the features of each group and flattening the transposed features to form a new channel order.
[0078] In some embodiments, see Figure 5 , Figure 5 This is a flowchart illustrating the text generation method provided in an embodiment of this application, such as... Figure 5 As shown, Figure 4 Step 103 shown can be achieved through... Figure 5 Steps 1031 to 1032 shown are implemented, and will be combined with Figure 5 The steps shown are explained.
[0079] In step 1031, the first feature is subjected to group regularization to obtain the first feature after group regularization.
[0080] It should be noted that group normalization is a regularization technique in machine learning. It refers to constraining parameters during model training to prevent overfitting while learning the group structure information between features.
[0081] As an example, suppose the number of channels for the first feature is C. We perform group regularization on the first feature, dividing channel C into G groups. Each group contains C / G channels. For each channel group, we calculate the mean μ and variance σ of all positional feature values within the group. 2 Then, normalization is performed on each feature value within the group to obtain the normalized feature values. Where X is the eigenvalue, and channel-level scaling factor γ and offset β are introduced to normalize the eigenvalue. Perform a linear transformation to obtain the eigenvalues after the linear transformation. This completes the group regularization process, arranging each linearly transformed feature value according to its corresponding position to obtain the first feature after group regularization. The number of channels of the first feature after group regularization is still C.
[0082] In step 1032, the channels of the first feature that has undergone group regularization are subjected to channel shuffling to obtain the first processing result of the first feature.
[0083] Continuing with the example above, channel shuffling is performed on the channels of the first feature after group regularization. This involves regrouping the number of channels of the first feature after group regularization into K groups, changing the channel dimension C of the first feature after group regularization to [K, C / K]. The first feature of [K, C / K] is then transposed to obtain the transposed result of [C / K, K]. The transposed result is then flattened to re-flatten the feature into C channels, thus completing the channel shuffling process and obtaining the first feature after channel shuffling, which is the first processing result of the first feature.
[0084] Thus, by using group regularization, the feature distribution can be standardized, the internal covariate shift can be reduced, and the information isolation between channels can be broken by shuffling the channel order, which can effectively enhance the diversity of feature expression.
[0085] In step 104, the channels of the second feature are subjected to channel shuffling to obtain the second processing result of the second feature.
[0086] Here, the second feature is a feature in the second image that is different from the first feature.
[0087] In some embodiments, step 104 described above can be implemented as follows: performing group regularization on the second feature to obtain a group-regularized second feature; performing channel shuffling on the channels of the group-regularized second feature to obtain a second processing result of the second feature. Thus, by using group regularization, the feature distribution can be standardized, internal covariate shifts can be reduced, and by shuffling the channel order, information isolation between channels can be broken, effectively enhancing the diversity of feature representation.
[0088] It should be noted that the implementation of group regularization and channel shuffling for the second feature in the above steps is similar to the implementation of steps 1031 and 1032 above. For details, please refer to the implementation of steps 1031 and 1032 above, and will not be repeated here.
[0089] In some embodiments, before performing step 104, the following processing may also be performed: based on the dimensions of the multiple second images, the features of the multiple second images are sorted to obtain a feature sequence; if the first feature is at the beginning of the feature sequence, then the features adjacent to the first feature and the feature at the end of the feature sequence are taken as the second feature; if the first feature is at the end of the feature sequence, then the features adjacent to the first feature and the feature at the beginning of the feature sequence are taken as the second feature; if the first feature is neither at the end nor at the beginning of the feature sequence, then the features adjacent to the first feature are taken as the second feature. Thus, establishing a feature sequence based on the dimensions of the multiple second images allows for efficient operation based on the feature sequence, thereby improving the efficiency of subsequent processing; by reasonably selecting the second feature, rich and representative feature information is provided for subsequent processing tasks, thereby improving the accuracy and reliability of subsequent processing.
[0090] As an example, if the size of the second image corresponding to feature E1 is 1024*1024, the size of the second image corresponding to feature E2 is 512*512, and the size of the second image corresponding to feature E3 is 128*128, then the feature sequence obtained by sorting the features of the second image according to the size of the second image is E1, E2, and E3. Assuming that the first feature is at the beginning of the feature sequence, such as E1, then the second features are E2 (the feature adjacent to the first feature) and E3 (the feature at the end of the feature sequence). Assuming that the first feature is at the end of the feature sequence, such as E3, then the second features are E1 (the feature at the beginning of the feature sequence) and E2 (the feature adjacent to the first feature). Assuming that the first feature is neither at the end nor at the beginning of the feature sequence, such as E2, then the second features are E1 (the feature adjacent to the first feature) and E3 (the feature adjacent to the first feature).
[0091] In some embodiments, before performing step 104, the following processing may also be performed: based on the dimensions of the multiple second images, the features of the multiple second images are sorted to obtain a feature sequence; if the first feature is at the beginning of the feature sequence, then two features in the feature sequence other than those adjacent to the first feature or located at the end of the feature sequence are selected as the second feature; if the first feature is at the end of the feature sequence, then two features in the feature sequence other than those adjacent to the first feature or located at the beginning of the feature sequence are selected as the second feature; if the first feature is neither at the end nor at the beginning of the feature sequence, then two features in the feature sequence other than those adjacent to the first feature are selected as the second feature. Thus, by introducing non-adjacent feature information, global context information can be constructed, avoiding the omission of important information due to local window limitations, thereby overcoming locality limitations and capturing long-distance dependencies between features.
[0092] As an example, suppose the feature sequence is {E1, E2, E3, E4, E5}. If the first feature is E1, then the second feature can be E3 and E4; if the first feature is E5, then the second feature can be E1 and E2; if the first feature is E2, then the second feature can be E4 and E5.
[0093] In step 105, the first processing result and the second processing result are fused to obtain the third feature.
[0094] It should be noted that the fusion of the first and second processing results can be achieved by directly concatenating the two results or by adding the feature values; no specific limitation is made here.
[0095] As an example, suppose the feature size of the first processing result is 4096*768 and the feature size of the second processing result is 4096*768. Taking feature concatenation as an example, the two features are concatenated to obtain a third feature with a feature size of 9192*768.
[0096] In some embodiments, step 105 described above can be implemented as follows: if there are multiple second features, the first processing result is fused with each of the second processing results to obtain a third feature corresponding to each second feature. Thus, by fusing different processing results, different feature information can be summarized, thereby making each third feature contain more comprehensive and richer information; by fusing different second features separately, the diversity of feature information can be captured more comprehensively, so as to make fuller use of feature diversity.
[0097] As an example, suppose the feature sequence is {E1, E2, E3, E4, E5}, the second features corresponding to the first feature E1 are E2 and E5, the first processing result of the first feature E1 is E1_1, the second processing result of the second feature E2 is E2_1, and the second processing result of the second feature E5 is E5_1. By fusing the first processing result E1_1 and the second processing result E2_1, we can obtain the third feature E1_new1 of the second feature. By fusing the first processing result E1_1 and the second processing result E5_1, we can obtain the third feature E1_new2 of the second feature. Similarly, we can obtain the third feature of the second feature corresponding to each first feature.
[0098] In step 106, the third feature is mapped to the same feature space as the first feature to obtain the fourth feature corresponding to the first feature.
[0099] In some embodiments, step 106 described above can be implemented as follows: mapping the third feature to obtain a first mapping result; performing dimension repair on the first mapping result to obtain a dimension-repaired result; performing channel shuffling on the dimension-repaired result to obtain a channel-shuffling-processed mapping result; and performing group regularization on the channel-shuffling-processed mapping result to obtain a fourth feature corresponding to the first feature. Thus, by mapping the feature to a new feature space through a function or transformation, the nonlinear expressive power of the feature can be increased, thereby capturing more complex feature relationships and patterns. Dimension repair transforms the mapped feature back to its original space, helping to convert complex feature relationships into more easily interpretable association results, facilitating subsequent processing and analysis. Grouping and transpose operations offset some of the randomness of the first channel shuffling, restoring cross-channel correlation, making the channel distribution more consistent with the grouping requirements of subsequent group regularization. Group regularization eliminates the distribution shift caused by channel shuffling and intermediate operations, thereby restoring the scale consistency of the feature.
[0100] As an example, assuming we obtain a third feature with a size of 9192*768, we flatten it, reshaping it to arrange the features at each spatial location sequentially, resulting in a feature with a size of 7059456*1. Next, we input this 7059456*1 feature into an MLP for mapping, obtaining two features with a size of 3529728*1, which is the first mapping result. Based on the dimensionality of the first feature, we perform dimensionality correction on the first mapping result, resulting in two features of 4096*768, which are the dimensionality-corrected results. The feature corresponding to the dimension repair result has C channels. Channel C is divided into M groups, changing the channel number of the feature from C to [C, C / M]. The feature in [C, C / M] is transposed to obtain the transposed result of [C / M, C]. This transposed result is then flattened to restore the feature to its original channel dimension C, thus obtaining the feature with restored channel order, i.e., the dimension flattened result. Next, the feature with restored channel order is re-divided into N groups, each containing C / N channels. For each channel group, the mean μ and variance σ of all positional feature values within the group are calculated. 2 Then, normalization is performed on each feature value within the group to obtain the normalized feature values. Where X represents the feature value, and a learnable scaling factor γ is used. recover and offset β recover The features are scaled and evaluated to minimize the damage to the original features, thus obtaining the features after dimension reduction, which is the fourth feature corresponding to the first feature.
[0101] In some embodiments, if there are multiple second features, step 106 described above can be implemented by mapping a third feature corresponding to each second feature to the same feature space as the first feature, thereby obtaining a fourth feature corresponding to each second feature of the first feature. In this way, by mapping the third feature to the same feature space as the first feature, it can better adapt to subsequent task processing.
[0102] It should be noted that the implementation method for mapping the third feature for each second feature is similar to the implementation method of step 106 above. For details, please refer to the implementation method of step 106 above, and it will not be repeated here.
[0103] In step 107, based on the weight of the first feature, the fourth feature corresponding to the first feature is fused to obtain the fifth feature.
[0104] As an example, suppose the weight of the first feature E1 is w1, the weight of the first feature E2 is w2, the fourth feature corresponding to the first feature E1 is E1_new, and the fourth feature corresponding to the first feature E2 is E2_new. Based on the weight of the first feature, the fourth feature corresponding to the first feature is weighted and fused to obtain the fifth feature E as w1*E1_new+w2*E2_new.
[0105] In some embodiments, before performing step 107, the following processing may also be performed: fusing multiple first features to obtain a sixth feature; mapping the sixth feature to obtain a first mapping result; and normalizing the first mapping result to obtain the weight corresponding to each first feature. Thus, the mapping process enhances the differences between features, making similar features cluster more closely together while dispersing different features, which helps to enhance the expressiveness of the features; the normalization process ensures that each feature has the same importance to the model's weights, avoiding disproportionate impacts on the model results due to the large numerical range of some features.
[0106] It should be noted that fusion processing can be achieved by operating on basic features, such as additive fusion, multiplicative fusion, concatenation fusion, weighted fusion, etc.; it can also be driven by advanced mechanisms, such as attention mechanism fusion, pyramid pooling fusion, deconvolution fusion, etc.; it can also be achieved through mathematical transformations and interactions, such as linear combination fusion, nonlinear combination fusion, feature interaction fusion, etc.; it can also be achieved through hierarchical strategy fusion, such as early fusion, late fusion, intermediate fusion, etc.; and it can also be achieved through engineering optimization methods, such as aggregation function fusion, embedding fusion, etc., without specific limitations here.
[0107] As an example, suppose there are five first features, E1, E2, E3, E4, and E5, each with a size of 4096*768. Concatenating and fusing these five first features yields a sixth feature with a size of 20480*768. Flattening this sixth feature results in a flattened version with a size of 15728640*1. Mapping this flattened version yields a first mapping result, a feature with a size of 5*1. Normalizing this first mapping result gives the probability of the 5*1 feature as [0.4, 0.2, 0.15, 0.01, 0.24]. This corresponds to a weight of 0.4 for the first feature E1, 0.2 for E2, 0.15 for E3, 0.01 for E4, and 0.25 for E5.
[0108] In some embodiments, step 107 described above can be implemented by fusing the fourth feature corresponding to the first feature with the first feature to obtain a ninth feature corresponding to the first feature; and by weighting the ninth feature corresponding to the first feature based on the weight of the first feature to obtain the fifth feature. Thus, fusing the fourth feature corresponding to the first feature with the first feature integrates feature information obtained from different sources or through different processing methods, providing a more comprehensive and detailed description of the image's features, thereby enhancing the expressive power of the features.
[0109] As an example, suppose the weight of the first feature E1 is w1, the weight of the first feature E2 is w2, the fourth feature corresponding to the first feature E1 is E1_new, and the fourth feature corresponding to the first feature E2 is E2_new. By fusing the first feature and the fourth feature corresponding to the first feature, we can obtain the ninth feature E1_inter of the first feature E1 as E1+E1_new, and the ninth feature E2_inter of the first feature E2 as E2+E2_new. Based on the weight of the first feature, we can perform a weighted fusion process on the ninth feature corresponding to the first feature to obtain the fifth feature E as w1*E1_inter+w2*E2_inter.
[0110] In some embodiments, if there are multiple second features, step 107 described above can be implemented as follows: the fourth feature corresponding to each second feature of the first feature is fused with the first feature to obtain a ninth feature; based on the weight of the first feature, the ninth feature corresponding to the first feature is weighted to obtain the fifth feature. Thus, fusing the fourth feature of each second feature corresponding to the first feature with the first feature integrates feature information obtained from different sources or through different processing methods, providing a more comprehensive and detailed description of the image's features, thereby enhancing the expressive power of the features.
[0111] As an example, suppose the weight of the first feature E1 is w1, the weight of the first feature E2 is w2, the fourth feature of the second feature E3 corresponding to the first feature E1 is E1_new1, the fourth feature of the second feature E4 corresponding to the first feature E1 is E1_new2, the fourth feature of the second feature E5 corresponding to the first feature E2 is E2_new1, and the fourth feature of the second feature E6 corresponding to the first feature E2 is E2_new2. By fusing the fourth feature of each second feature corresponding to the first feature E1 with the first feature (taking the feature value as an example), we can obtain the ninth feature E1_inter corresponding to the first feature E1 as E1+E1_new1+E1_new2; similarly, the ninth feature E2_inter corresponding to the first feature E2 is E2+E2_new1+E2_new2. Finally, according to the weight of the first feature, the ninth feature corresponding to the first feature is weighted and fused to obtain the fifth feature E as w1*E1_inter+w2*E2_inter.
[0112] In step 108, a second text is generated based on the fifth feature and the first text corresponding to the first image.
[0113] In some embodiments, see Figure 6 , Figure 6 This is a flowchart illustrating the text generation method provided in an embodiment of this application, such as... Figure 6 As shown, Figure 4 Step 108 shown can be achieved through... Figure 6 Steps 1081 to 1082 shown are implemented, and will be combined with Figure 6 The steps shown are explained.
[0114] In step 1081, the fifth feature is mapped to obtain a second mapping result, and the first text is vectorized to obtain a seventh feature.
[0115] As an example, suppose the first image is a picture containing a brown fox. The fifth feature obtained through processing is f1, and the second mapping result obtained by mapping the fifth feature f1 is f2. The first text is "What color is the animal in this picture?", and the seventh feature obtained by vectorizing the first text is f3.
[0116] In step 1082, the second mapping result and the seventh feature are fused to obtain the eighth feature, and the second text is generated based on the eighth feature.
[0117] Following the example above, the second mapping result f2 and the seventh feature f3 are fused to obtain the eighth feature f4. Finally, the answer is predicted based on the eighth feature f4, generating the second text "The animal in this picture is brown".
[0118] In this way, by combining features from different sources, multi-source information is fully utilized, providing strong support for generating accurate and relevant text content.
[0119] In some embodiments, taking an image generation scenario as an example, suppose the first image is an abstract painting with simple geometric shapes (circles, triangles, and squares). The fifth feature g1 is obtained through processing. The fifth feature g1 is then mapped to obtain a second mapping result g2. The first text is "Generate a painting containing these shapes with more vibrant colors." The first text is vectorized, converting the semantic information in the first text into vector form, resulting in a seventh feature g3. The second mapping result g2 and the seventh feature g3 are then fused, combining the image features and the semantic features of the text description to obtain an eighth feature g4. Based on the eighth feature g4, the image generation model generates a new image that conforms to the text description according to the fused features. The generated second text can be a simple description of the generated image, such as "An abstract painting containing brightly colored circles, triangles, and squares."
[0120] In some embodiments, taking a music recommendation scenario as an example, suppose the first image is a photo of a singer performing on stage. After feature processing, the fifth feature is obtained as h1. Mapping the fifth feature h1 yields the second mapping result h2. The first text is "Recommend music similar to this singer's style." Vectorizing the first text transforms the user's recommendation request into a vector, resulting in the seventh feature h3. The second mapping result h2 and the seventh feature h3 are then fused, combining the singer's style features in the image with the user's recommendation request to obtain the eighth feature h4. Based on the eighth feature h4, the music recommendation system recommends suitable music to the user according to the fused features. The generated second text can be a list of recommended music, such as "Recommend singer A's 'Song 1,' singer B's 'Song 2,' etc."
[0121] In some embodiments, taking a product recommendation scenario as an example, assuming the first image is a picture of a fashionable sneaker, the fifth feature is obtained after processing, k1. The fifth feature k1 is mapped to obtain the second mapping result k2. The first text is "Recommend similar styles of sneakers". The first text is vectorized to transform the user's product recommendation needs into vectors, resulting in the seventh feature k3. Subsequently, the second mapping result k2 and the seventh feature k3 are fused. Combining the features of the sneakers and the user's recommendation needs, the eighth feature k4 is obtained. Based on the eighth feature k4, the product recommendation system will recommend similar styles of sneakers to the user according to the fused features. The generated second text can be a list of recommended products, such as "Recommend sneakers 1 from brand X, sneakers 2 from brand Y, etc."
[0122] In some embodiments, the above-described text generation method is implemented by calling a first model. Before executing step 101, the following processing may also be performed: multi-scale scaling of the third image to obtain multiple fourth images; the following processing is performed through the first model: feature extraction processing is performed on each of the fourth images to obtain features of each of the fourth images; channel shuffling processing is performed on the channels of the tenth feature to obtain the third processing result of the tenth feature; and channel shuffling processing is performed on the channels of the eleventh feature to obtain the fourth processing result of the eleventh feature; wherein, the tenth feature is a feature of any of the fourth images, and the eleventh feature is a feature of the fourth image. The first model identifies features that differ from the tenth feature. The third and fourth processing results are fused to obtain a twelfth feature, which is then mapped to the same feature space as the tenth feature to obtain a thirteenth feature. Based on the weights of the tenth feature, the thirteenth feature is fused to obtain a fourteenth feature. Based on the fourteenth feature and the third text corresponding to the third image, predicted text is generated. Based on the predicted text and the real text corresponding to the third image, a first loss is determined. Based on the first loss, the parameters of the first model are updated to obtain an updated first model. Thus, through multi-scale feature analysis and fusion, the model can more accurately understand image content and generate more accurate text, effectively improving the text generation capability of the first model. Multi-scale processing and feature fusion help the model adapt to different images and problem types, thereby enhancing the model's generalization ability. Through complex multi-scale feature analysis and fusion, and model training based on weight optimization, the accuracy of the model's generated text content can be significantly improved.
[0123] As an example, assuming the third image contains multiple objects, firstly, the third image is subjected to multi-scale scaling to obtain three fourth images at different scales. The scaling scale for fourth image P1 is 1, for fourth image P2 it is 0.5, and for fourth image P3 it is 0.25. Next, the first model is used to extract features from P1, P2, and P3 respectively, obtaining the corresponding features E1, E2, and E3. Then, when the tenth feature is E1 and the eleventh features are E2 and E3, channel shuffling is performed on the tenth feature E1 to obtain the third processing result. E1_new1, the eleventh features E2 and E3 are subjected to channel shuffling processing to obtain the fourth processing results E2_new1 and E3_new1 respectively. Similarly, when the tenth feature is E2, the corresponding third processing result is E2_new2 and the fourth processing result is E1_new2 and E3_new2, and when the tenth feature is E3, the corresponding third processing result is E3_new3 and the fourth processing result is E1_new3 and E2_new3. Then, the third processing result E1_new1 and the fourth processing result E2_new1 and E3_new1 are processed. E1 is fused to obtain the twelfth feature E1_inter, and similarly, the twelfth features E2_inter and E3_inter are obtained. Next, the twelfth feature E1_inter is mapped to the same feature space as the tenth feature to obtain the thirteenth feature E1_lim, and similarly, the thirteenth features E2_lim and E3_lim are obtained. Then, assuming the weight of the tenth feature E1 is w1, the weight of E2 is w2, and the weight of E3 is w3, based on the weights of the tenth feature, the corresponding thirteenth feature is fused to obtain the thirteenth feature E1_inter. The fourteenth feature E is w1*E1_lim+w2*E2_lim+w3*E3_lim; then, assuming the third text corresponding to the third image is "What animals are in this picture?", the predicted text is generated by combining the fourteenth feature E and the third text as "There are dogs and cats in this picture". Assuming the corresponding real text is "There are dogs and cats in this picture", the first loss L can be determined by formula (1) based on the predicted text and the real text; finally, based on the first loss L, the parameters of the first model are updated using gradient descent or other optimization algorithms to obtain the updated first model. In this way, by continuously optimizing the model parameters, multi-scale image information can be used to more comprehensively understand the image content, and more accurate and relevant text content can be generated through feature association and weight fusion.
[0124] L=-∑ i y true,i log(y pred,i (1)
[0125] Among them, y true,i Let y be the real text corresponding to the i-th image.pred,i Let be the predicted probability of each word in the predicted text corresponding to the i-th image.
[0126] In practical applications, when the first loss reaches the parameter threshold, backpropagation is performed in the first model based on the first loss, and the model parameters of the first model are updated during the propagation process.
[0127] Here's an explanation of backpropagation: The third image and third text are input into the input layer of the neural network model, pass through the hidden layer, and finally reach the output layer to output the result. This is the forward propagation process of the neural network model. Since there is an error between the output result of the neural network model and the actual result, the error between the result and the actual value is calculated and backpropagated from the output layer to the hidden layer until it reaches the input layer. During the backpropagation process, the values of the model parameters are adjusted according to the error. The above process is iterated continuously until convergence.
[0128] Taking the loss function L of the first model in this embodiment as an example, the server determines the first loss based on the loss function, and propagates the first loss backward from the output layer in the first model. The first loss is propagated backward layer by layer. When the first loss reaches each layer, the gradient (that is, the partial derivative of the loss function with respect to the parameters of each layer) is solved in combination with the propagated first loss, and the corresponding gradient value of the parameters of each layer is updated.
[0129] The following describes an exemplary application of the embodiments of this application in a real-world application scenario. This exemplary application describes the specific implementation process of the text generation method in an intelligent question-answering scenario.
[0130] In related technologies, the document big model directly processes images through the visual encoding module of a VisionTransformer (ViT), and then connects the processing results of the video encoding module with a large language model through a multilayer perceptron to obtain the document big model. However, directly inputting images into the ViT model can only obtain information about the current scale of the image, and with only a single scale of image input, the model may miss some information, resulting in lower accuracy of the generated answer.
[0131] In some embodiments, see Figure 7 , Figure 7 This is a model framework diagram of the text generation method provided in the embodiments of this application. The following will combine... Figure 7 The text generation method provided in this application is described in detail.
[0132] First, the acquired document image is scaled while maintaining its aspect ratio, ensuring the total pixel count does not exceed 1024*1024, resulting in image Pmax. Then, image Pmax is scaled at different scales (e.g., 0.25, 0.38, 0.50, 0.75, and 1.0), resulting in the corresponding images P1, P2, P3, P4, and P5. This allows the model to learn multi-scale image information, providing more accurate features (large, medium, and small images), which, through the feature fusion module, enables the model to achieve better question-answering results.
[0133] Subsequently, image P5 is processed through the RepVGG-A0 backbone network to extract features, followed by adaptive pooling and a fully connected layer to obtain the corresponding feature matrix E5 (size 4096*768); image P4 is processed through the ResNet18 backbone network to extract features, followed by adaptive pooling and a fully connected layer to obtain the corresponding feature matrix E4 (size 4096*768); image P3 is processed through the ViT-small backbone network to extract features, obtaining the corresponding feature matrix E3 (size 4096*768); image P2 is processed through the ViT-base backbone network to extract features, obtaining the corresponding feature matrix E2 (size 4096*768); and image P1 is processed through the ViT-large backbone network to extract features, obtaining the corresponding feature matrix E1 (size 4096*768). Thus, by using different network architectures such as RepVGG, ResNet, and ViT to process images of different scales, different output information is obtained, enabling the model to have the ability to fuse models. Here, model fusion involves training multiple models using different backbone networks and fusing the output results of the trained models to achieve higher performance than a single model. In addition, the complexity of RepVGG-A0, ResNet18, ViT-small, ViT-base, and ViT-large networks increases sequentially, while the complexity of images P5, P4, P3, P2, and P1 decreases sequentially. By designing a structure that uses small networks for large images and large networks for small images, the feature extraction speed of the visual encoding module is faster, resulting in a faster model response speed and less resource consumption.
[0134] See afterward. Figure 8 , Figure 8 This is an architecture diagram of the feature fusion module provided in the embodiments of this application, such as... Figure 8As shown, two feature matrices are selected sequentially from feature matrices E1, E2, E3, E4, and E5 and mutual information learning is performed using a mutual information module to obtain E1_new1, E1_new2, E2_new1, E2_new2, E3_new1, E3_new2, E4_new1, E4_new2, E5_new1, and E5_new2, respectively. Then, E1_new1 and E1_new2, E2_new1 and E2_new2, E3_new1 and E3_new2, E4_new1 and E4_new2, and E5_new1 and E5_new2 are fused to obtain E1_inter, E2_inter, E3_inter, and E4_inter. The feature fusion module merges feature matrices E1, E2, E3, E4, and E5 to obtain a feature matrix of size 20480*768. This feature matrix is then flattened to obtain a feature matrix of size 15728640*1, and mapped using an MLP to obtain a feature matrix of size 5*1. This 5*1 feature matrix is then subjected to a softmax algorithm for probability prediction, resulting in 5*1 feature probabilities. These five feature probabilities are used as weights for E1_inter, E2_inter, E3_inter, E4_inter, and E5_inter. The five feature matrices are then weighted and summed with their respective weights to obtain the fused feature E_last. In this way, the feature fusion module merges multiple feature matrices, generates weight probabilities based on the feature matrices, and thus determines the weight proportion of each feature matrix.
[0135] In some embodiments, see Figure 9 , Figure 9 This is an architecture diagram of the mutual information module provided in the embodiments of this application, such as... Figure 9As shown, taking feature matrices E5 (size 4096*768) and E4 (size 4096*768) as inputs, the inputs are processed through two branches. Each branch undergoes group regularization and channel shuffling to shuffle the dimensions, resulting in two shuffled feature matrices. These feature matrices are then concatenated to obtain a concatenated feature matrix of size 9192*768. The concatenated feature matrix is then flattened to obtain a flattened feature matrix of size 7059456*1. This flattened feature matrix is then input into the MLP module for mapping, resulting in two feature matrices of size 3529728*1. Dimensional repair is then performed on these two feature matrices to obtain two feature matrices of size 4096*768. Finally, these two feature matrices are processed through a channel shuffling module and a group regularization module to restore their features to the same feature space as E5 and E4, resulting in new feature matrices E5_new and E4_new. In this way, the mutual information module is used to break down the channel barrier between two feature matrices, enabling the two feature matrices to learn from and influence each other in the channel dimension.
[0136] See also Figure 7 Finally, the fused feature E_last is processed by MLP to obtain the visual feature E_visual, and the user input question is processed by the tokenizer to obtain the text feature E_text. The two features are concatenated and input into the large language model (Qwen2.5-0.5B) for prediction processing to obtain the predicted answer that responds to the question and matches the image content.
[0137] It should be noted that the network architecture and process flow are similar in the training and inference phases of this application. During the model training process, it is necessary to fix the parameters of the large language model and update the parameters of other modules in the model except for the large language model based on the first error between the predicted answer and the real answer.
[0138] The following description continues to illustrate the exemplary structure of the text generation device 543 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 3 As shown, the software modules stored in the text generation device 543 in the memory 540 may include: a first processing module 5431, a second processing module 5432, a third processing module 5433, a fusion module 5434, and a generation module 5435.
[0139] The first processing module 5431 is used to perform multi-scale scaling processing on the first image to obtain multiple second images, and to perform feature extraction processing on each second image to obtain features of each second image; the second processing module 5432 is used to perform channel shuffling processing on the channels of the first feature to obtain a first processing result of the first feature, and to perform channel shuffling processing on the channels of the second feature to obtain a second processing result of the second feature; wherein, the first feature is a feature of any second image, and the second feature is a feature of the second image that is different from the first feature; the third processing module 5433 is used to perform fusion processing on the first processing result and the second processing result to obtain a third feature, and to map the third feature to the same feature space as the first feature to obtain a fourth feature corresponding to the first feature; the fusion module 5434 is used to perform fusion processing on the fourth feature corresponding to the first feature based on the weight of the first feature to obtain a fifth feature; the generation module 5435 is used to generate second text based on the fifth feature and the first text corresponding to the first image.
[0140] In some embodiments, the first processing module 5431 is further configured to, for each of the second images, invoke a first neural network corresponding to the size of the second image to perform feature extraction processing on the second image to obtain a first feature of the second image; wherein, the space complexity of the first neural network is negatively correlated with the size of the second image.
[0141] In some embodiments, the second processing module 5432 is further configured to sort the features of the multiple second images based on their sizes to obtain a feature sequence; if the first feature is at the beginning of the feature sequence, then the features adjacent to the first feature and the features at the end of the feature sequence are used as the second feature; if the first feature is at the end of the feature sequence, then the features adjacent to the first feature and the features at the beginning of the feature sequence are used as the second feature; if the first feature is neither at the end nor at the beginning of the feature sequence, then the features adjacent to the first feature are used as the second feature.
[0142] In some embodiments, the second processing module 5432 is further configured to perform group regularization processing on the first feature to obtain a first feature after group regularization processing; and to perform channel shuffling processing on the channels of the first feature after group regularization processing to obtain a first processing result of the first feature.
[0143] In some embodiments, the third processing module 5433 is further configured to, if the number of the second features is multiple, fuse the first processing result with each of the second processing results respectively to obtain a third feature corresponding to each of the second features; the third processing module 5433 is further configured to map the third feature corresponding to each of the second features to the same feature space as the first feature to obtain a fourth feature corresponding to each of the second features of the first feature.
[0144] In some embodiments, the fusion module 5434 is further configured to perform fusion processing on multiple first features to obtain a sixth feature; perform mapping processing on the sixth feature to obtain a first mapping result; and perform normalization processing on the first mapping result to obtain a weight corresponding to each first feature.
[0145] In some embodiments, the fusion module 5434 is further configured to fuse the fourth feature corresponding to the first feature with the first feature to obtain a ninth feature corresponding to the first feature; and to perform weighted processing on the ninth feature corresponding to the first feature based on the weight of the first feature to obtain the fifth feature.
[0146] In some embodiments, the generation module 5435 is further configured to perform mapping processing on the fifth feature to obtain a second mapping result, and to perform vectorization processing on the first text to obtain a seventh feature; to perform fusion processing on the second mapping result and the seventh feature to obtain an eighth feature, and to generate the second text based on the eighth feature.
[0147] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, therefore, it will not be repeated. For any technical details not covered in the text generation apparatus provided in this application embodiment, please refer to... Figure 4 , Figure 5 ,or Figure 6 The meaning is understood in accordance with the description of any of the accompanying drawings.
[0148] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the text generation method described above in this application.
[0149] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the text generation method provided in this application. For example, ... Figure 4 , Figure 5 ,or Figure 6 The text generation method is shown.
[0150] In some embodiments, the computer-readable storage medium may be a memory such as ferroelectric random access memory (FRAM), ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.
[0151] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.
[0152] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).
[0153] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.
[0154] In summary, by scaling the first image at different ratios to generate second images at multiple scales through the embodiments of this application, and extracting features from each second image, the multi-granularity information of the image can be captured, avoiding the loss of information at a single scale. Next, channel shuffling is performed on the channels of the first feature to obtain a first processing result for the first feature, and channel shuffling is performed on the channels of the second feature to obtain a second processing result for the second feature. Here, the first feature is a feature of any second image, and the second feature is a feature of the second image that differs from the first feature. By shuffling the channel order of the features, the independence of features between channels can be broken, allowing different... Feature information between channels can be exchanged and fused to uncover potential relationships between features. Subsequently, the first and second processing results are fused to obtain a third feature, which more comprehensively and accurately describes the features, enhancing their expressive power. This third feature is then mapped to the same feature space as the first feature, resulting in a fourth feature corresponding to the first feature, ensuring feature consistency and compatibility. Next, based on the weights of each first feature, the fourth features corresponding to the first feature are fused to obtain a fifth feature, integrating cross-scale information association and improving the feature's information expressive power. Finally, a more reasonable and accurate second text is generated based on the fifth feature and the first text. In summary, through multi-scale processing, cross-scale association, and weight-based fusion, the ability to capture image details in complex visual question answering tasks is significantly improved, thereby further enhancing the accuracy of cross-modal reasoning.
[0155] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.
Claims
1. A text generation method, characterized in that, The method includes: The first image is subjected to multi-scale scaling to obtain multiple second images, and feature extraction is performed on each second image to obtain the features of each second image; Channel shuffling is performed on the channels of the first feature to obtain the first processing result of the first feature, and channel shuffling is performed on the channels of the second feature to obtain the second processing result of the second feature. Wherein, the first feature is any feature of the second image, and the second feature is a feature of the second image that is different from the first feature; The first processing result and the second processing result are fused to obtain a third feature, and the third feature is mapped to the same feature space as the first feature to obtain a fourth feature corresponding to the first feature. Based on the weight of the first feature, the fourth feature corresponding to the first feature is fused to obtain the fifth feature; Based on the fifth feature and the first text corresponding to the first image, the second text is generated.
2. The method according to claim 1, characterized in that, The step of performing feature extraction processing on each of the second images to obtain the features of each of the second images includes: For each of the second images, a first neural network corresponding to the size of the second image is invoked to perform feature extraction processing on the second image to obtain the features of the second image; The space complexity of the first neural network is negatively correlated with the size of the second image.
3. The method according to claim 1, characterized in that, Before performing channel shuffling on the channels of the second feature to obtain the second processing result of the second feature, the method further includes: Based on the dimensions of multiple second images, the features of the multiple second images are sorted to obtain a feature sequence; If the first feature is located at the beginning of the feature sequence, then the features adjacent to the first feature and the feature located at the end of the feature sequence are taken as the second feature. If the first feature is located at the end of the feature sequence, then the features adjacent to the first feature in the feature sequence and the feature located at the beginning of the feature sequence are taken as the second feature; If the first feature is not at the end of the feature sequence and is not at the beginning of the feature sequence, then the feature adjacent to the first feature in the feature sequence is taken as the second feature.
4. The method according to claim 1, characterized in that, The process of performing channel shuffling on the channels of the first feature to obtain the first processing result of the first feature includes: The first feature is subjected to group regularization to obtain the first feature after group regularization. The channels of the first feature that have undergone group regularization are subjected to channel shuffling to obtain the first processing result of the first feature.
5. The method according to claim 1, characterized in that, Before fusing the fourth feature corresponding to the first feature based on the weight of the first feature to obtain the fifth feature, the method further includes: The first feature is fused together to obtain the sixth feature; The sixth feature is mapped to obtain the first mapping result; The first mapping result is normalized to obtain the weight corresponding to each of the first features.
6. The method according to claim 1, characterized in that, The step of generating second text based on the fifth feature and the first text corresponding to the first image includes: The fifth feature is mapped to obtain a second mapping result, and the first text is vectorized to obtain a seventh feature; The second mapping result and the seventh feature are fused together to obtain the eighth feature, and the second text is generated based on the eighth feature.
7. The method according to claim 1, characterized in that, The process of fusing the fourth feature corresponding to the first feature based on the weight of the first feature to obtain the fifth feature includes: The fourth feature corresponding to the first feature is fused with the first feature to obtain the ninth feature corresponding to the first feature; Based on the weight of the first feature, the ninth feature corresponding to the first feature is weighted to obtain the fifth feature.
8. The method according to claim 1, characterized in that, The process of fusing the first processing result of the first feature with the second processing result of the second feature to obtain the third feature includes: If there are multiple second features, the first processing result is fused with each of the second processing results to obtain a third feature corresponding to each second feature; The step of mapping the third feature to the same feature space as the first feature to obtain the fourth feature corresponding to the first feature includes: Map the third feature corresponding to each second feature to the same feature space as the first feature to obtain the fourth feature corresponding to each second feature of the first feature.
9. A text generation device, characterized in that, The device includes: The first processing module is used to perform multi-scale scaling processing on the first image to obtain multiple second images, and to perform feature extraction processing on each second image to obtain the features of each second image; The second processing module is used to perform channel shuffling on the channels of the first feature to obtain a first processing result of the first feature, and to perform channel shuffling on the channels of the second feature to obtain a second processing result of the second feature; wherein, the first feature is any feature of the second image, and the second feature is a feature of the second image that is different from the first feature; The third processing module is used to fuse the first processing result and the second processing result to obtain a third feature, and to map the third feature to the same feature space as the first feature to obtain a fourth feature corresponding to the first feature; The fusion module is used to perform fusion processing on the fourth feature corresponding to the first feature based on the weight of the first feature to obtain the fifth feature; The generation module is used to generate second text based on the fifth feature and the first text corresponding to the first image.
10. An electronic device, characterized in that, include: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the text generation method according to any one of claims 1 to 8.
11. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the text generation method according to any one of claims 1 to 8 is implemented.
12. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the text generation method according to any one of claims 1 to 8 is implemented.
