Image-text processing method and device, electronic equipment and readable storage medium
By performing vector transformation and feature extraction on text and image placeholders in the large-scale graph-text model, the processing latency problem of the VLM model is solved, enabling real-time feature extraction and segmented inference, thus improving processing efficiency.
Patent Information
- Application Number
- CN202511701404.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-19
- Publication Date
- 2026-02-24
AI Technical Summary
Existing VLM models require waiting for a certain number of images to accumulate from the video stream when processing tasks, which increases processing latency.
By performing vector transformation on the text and image placeholders to be processed, the text embedding vector sequence is segmented, and the image feature vector is extracted in real time using a visual encoder to replace the word embedding vectors corresponding to the image placeholders. Language model inference is then performed in segments to shorten the processing latency.
It enables feature extraction upon receiving the image to be processed, shortening the processing latency of large image-text models and improving processing efficiency.
Smart Images
Figure CN121564503A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and specifically to a graphic processing method, apparatus, electronic device, and readable storage medium. Background Technology
[0002] The Visual Language Model (VLM) is a multimodal artificial intelligence model that combines visual and language processing capabilities. VLM models can understand and generate natural language related to visual content. Their core goal is to achieve cross-modal interaction and reasoning, solving complex tasks that traditional computer vision and natural language processing models cannot accomplish. VLM models can be applied in many fields, such as autonomous driving, remote sensing image analysis, and cross-modal understanding. Taking autonomous driving as an example, VLM models can be used to identify semantic and visual information in complex traffic scenes for environmental perception and decision support.
[0003] A VML model primarily consists of a visual encoder, a language model, and a multimodal fusion module connecting the two. The visual encoder extracts image features, transforming the image into high-dimensional feature vectors. These high-dimensional feature vectors capture the key visual features of the image for use by the subsequent multimodal fusion module. Commonly used visual encoders include Convolutional Neural Networks (CNNs) (e.g., ResNet) or Visual Transformer models (e.g., the Transformer-based computer vision model ViT). The language model processes text input and output, generating natural language descriptions or performing reasoning based on text instructions. The language model is typically a pre-trained Large Language Model (LLM), such as the Generative Pre-trained Transformer (GPT) or the Bidirectional Encoder Representations from Transformers (BERT). The multimodal fusion module converts the visual features (image embeddings) generated by the visual encoder into a format that the language model can understand through a cross-modal attention mechanism. It then uses a projection layer to project the visual features into the same semantic space as the text word embeddings, thereby achieving alignment and joint inference between visual features and text semantics.
[0004] The inference process of the VLM model mainly includes the visual encoding stage and the LLM inference stage. This will be illustrated using a ViT model as the visual encoder and a Large Language Model (LLM) as the language model. Figure 1 The reasoning process of the VLM model is illustrated. For example... Figure 1 As shown, the user inputs an image (or, if the user inputs a video, the video is first processed into an image by extracting frames) and text data (e.g., including prompts and conversation history). For the input image, the ViT model extracts image features ( Figure 1 ViT (Vision Extraction) is a feature extraction technique used in Natural Language Processing (LLM) to extract image feature vectors. The dimensions of these vectors are then aligned with the dimensions of the text tokens. For example, the ViT model segments an image into fixed-size "image patches" and transforms these patches into semantic units similar to "tokens" in Natural Language Processing. For the input text data (including prompts and historical conversations), a word segmenter divides the input text data into multiple words. Then, the word embedding layer of the LLM model performs vector transformation on each word to obtain the corresponding text embedding vector. To enable the LLM model to distinguish and understand image information, the image feature vectors are inserted into specific positions in the text embedding vector sequence. Typically, special tokens are used as placeholders to represent images or videos, such as image placeholders.<img_pad> Representing an image, using video placeholders<video_pad> This represents a video. Therefore, placeholders are used to represent images in the input text data, and word segmentation and vector transformation are performed to obtain text embedding vectors (which include the vectors corresponding to the input text data and the vectors corresponding to the image placeholders). Then, the aligned image feature vectors and text embedding vectors are fused, that is, the image feature vectors extracted by the ViT model are used to replace the vectors corresponding to the image placeholders, resulting in a fused feature vector sequence, which is the final input of the LLM model.
[0005] The inference phase of the LLM model includes Prefill and Decode. Prefill is the first core phase in the large language model's text generation process. It involves the model performing attention calculations, linear transformations, and other operations on the received feature vector sequence to generate and cache the key and value information (KV Cache) for each token, preparing for subsequent token-by-token text generation. Decode is the second core phase, immediately following Prefill. This phase utilizes the KV Cache cached in the Prefill phase, combined with the already generated token sequence, and predicts probabilities through attention mechanisms and the Softmax function (which transforms a vector containing arbitrary real numbers into a probability distribution). It then generates the next token one by one, repeating this process until a complete text that meets the "stopping condition" (such as generating a period or reaching the maximum length) is generated.
[0006] Depend on Figure 1 It can be seen that the image features extracted by the ViT model and the text features extracted by the LLM model are fused within the LLM model based on the positions of image or video placeholders to ensure smooth and efficient inference in the VLM model. Since the image features extracted by the ViT model are needed in the LLM inference stage, the execution result of the visual encoder is a prerequisite for language inference. Therefore, as... Figure 2 As shown, the VLM model needs to wait for a period of time to accumulate a certain number of images from the video stream when processing tasks, and this waiting time increases the processing latency. Summary of the Invention
[0007] To solve the above-mentioned technical problems, or at least partially solve them, embodiments of this application provide a text and image processing method, a text and image processing apparatus, an electronic device, and a readable storage medium.
[0008] In a first aspect, embodiments of this application provide a method for processing images and text, including: Perform vector transformation on the text and image placeholders to be processed to obtain the first text embedding vector sequence; The first text embedding vector sequence is segmented to obtain multiple word embedding sub-vectors; In response to acquiring the image to be processed, a visual encoder based on a large image-text model performs feature extraction processing on the image to be processed to obtain the visual feature vector of the image to be processed. The image placeholders in the word embedding subvector corresponding to the image to be processed are replaced using the visual feature vector of the image to be processed. In response to the completion of the replacement of the word embedding vector corresponding to the image placeholder in the word embedding sub-vector, the language model based on the large graph-text model performs inference on the replaced word embedding sub-vector.
[0009] Optionally, the step of segmenting the first text embedding vector sequence to obtain multiple word embedding sub-vectors includes: segmenting the first text embedding vector sequence according to the number of image placeholders to obtain multiple word embedding sub-vectors, each word embedding sub-vector including a word embedding vector corresponding to an image placeholder.
[0010] Optionally, before replacing the image placeholder in the word embedding sub-vector corresponding to the image to be processed with the visual feature vector of the image to be processed, the method further includes: aligning the dimension of the visual feature vector of the image to be processed with the dimension of the word embedding sub-vector.
[0011] Optionally, in response to the completion of the word embedding vector replacement corresponding to the image placeholder in the word embedding sub-vector, the language model based on the large graph-text model performs inference on the replaced word embedding sub-vector, including: In response to the completion of the replacement of the word embedding vector corresponding to the image placeholder in the word embedding sub-vector, the language model based on the large graph-text model performs prefill processing on the replaced word embedding sub-vector; In response to the completion of image placeholder replacement for the last word embedding sub-vector among the plurality of word embedding sub-vectors, the Prefill processing results corresponding to all word embedding sub-vectors are decoded to obtain the processing results of the large image-text model.
[0012] Secondly, embodiments of this application provide a graphic processing apparatus, including: The conversion module is used to perform vector conversion on the text and image placeholders to be processed, and obtain the first text embedding vector sequence. The segmentation module is used to segment the first text embedding vector sequence to obtain multiple word embedding sub-vectors; The visual feature extraction module is used to extract features from the image to be processed based on the large image-text model in response to the acquisition of the image to be processed, and to obtain the visual feature vector of the image to be processed. The replacement module is used to replace the image placeholder in the word embedding sub-vector corresponding to the image to be processed using the visual feature vector of the image to be processed; The inference module is used to infer the replaced word embedding vector based on the language model of the large graph-text model in response to the completion of the replacement of the word embedding vector corresponding to the image placeholder in the word embedding sub-vector.
[0013] Optionally, the segmentation module is used to: segment the first text embedding vector sequence according to the number of image placeholders to obtain multiple word embedding sub-vectors, each word embedding sub-vector including a word embedding vector corresponding to an image placeholder.
[0014] Optionally, the replacement module is further configured to: align the dimension of the visual feature vector of the image to be processed with the dimension of the word embedding sub-vector.
[0015] Optionally, the inference module is configured to: in response to the completion of the replacement of the word embedding vector corresponding to the image placeholder in the word embedding sub-vector, perform prefill processing on the replaced word embedding sub-vector based on the language model of the large graph-text model; in response to the completion of the replacement of the image placeholder in the last word embedding sub-vector among the plurality of word embedding sub-vectors, decode the prefill processing results corresponding to all word embedding sub-vectors to obtain the processing result of the large graph-text model.
[0016] Thirdly, embodiments of this application provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the image and text processing method provided in any embodiment of this application.
[0017] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, and when the program or instructions are executed by a processor, they implement the image and text processing method provided in any embodiment of this application.
[0018] Fifthly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a computer, cause the computer to execute the graphic processing method provided in any embodiment of this application.
[0019] Sixthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the image and text processing method provided in any embodiment of this application.
[0020] The technical solution provided in this application has at least the following beneficial effects: The image-text processing method provided in this application embodiment, upon acquiring an image to be processed, performs feature extraction processing on the image based on a large image-text model to obtain a visual feature vector of the image to be processed. That is, this application embodiment can call the visual encoder for feature extraction processing for each image to be processed received, without accumulating multiple images, thus shortening the processing latency. Upon receiving text to be processed, vector conversion is performed on the text to be processed and image placeholders to obtain a first text embedding vector sequence. The first text embedding vector sequence is segmented to obtain multiple word embedding sub-vectors. The visual feature vector of the image to be processed is then used to replace... The image placeholders in the word embedding sub-vectors corresponding to the image to be processed are replaced. In response to the completion of the replacement of the image placeholders in the word embedding sub-vectors, the language model based on the large graph-text model performs inference on the replaced word embedding sub-vectors. That is, according to the number of image placeholders, the first text embedding vector sequence is divided into multiple word embedding sub-vectors, and the vectors corresponding to the image placeholders in the word embedding sub-vectors are replaced respectively. Then, the vectors corresponding to the image placeholders in the first text embedding vector sequence are replaced in segments. The Prefill process can be performed in segments by making full use of the time waiting for subsequent images, which further shortens the processing latency of the large graph-text model. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.
[0022] Figure 1 A schematic diagram illustrating the reasoning process of a VLM model in prior art is shown; Figure 2 A schematic diagram illustrating the reasoning process of a VLM model in prior art is shown; Figure 3 A schematic flowchart of a text processing method according to an embodiment of this application is shown; Figure 4 A schematic flowchart of another embodiment of the image processing method of this application is shown; Figure 5 A schematic flowchart of a text processing method according to another embodiment of this application is shown; Figure 6 A schematic diagram of the structure of a graphic processing apparatus according to an embodiment of this application is shown; Figure 7 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation
[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0024] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0025] Figure 3 A schematic flowchart of a text processing method according to an embodiment of this application is shown. Figure 3 As shown, this image and text processing method includes: Step S301: Perform vector transformation on the text to be processed and the image placeholder to obtain the first text embedding vector sequence. The text to be processed is the text data of the input image-text model, which may include, for example, prompts and historical conversations. The image placeholder is, for example, img_pad.
[0026] In this step, the text to be processed and the image placeholders are input into the language processing model (e.g., LLM model) of the large graph-text model. The word segmenter in the language processing model segments the text to be processed and the image placeholders into words. Then, the word embedding layer in the language processing model performs vector transformation on each word to obtain the text embedding vector corresponding to each word, thus obtaining the first text embedding vector sequence corresponding to the text to be processed and the image placeholders.
[0027] In an optional embodiment, this image-text processing method can also be used to process video data. The input to the large image-text model includes the text to be processed and video placeholders. The video placeholder is, for example, `video_pad`. In this step, the text to be processed and the video placeholders are input as input data to the language processing model (e.g., an LLM model) of the large image-text model. The word segmenter in the language processing model segments the text to be processed and the video placeholders into multiple words. Then, the word embedding layer in the language processing model performs vector transformation on each word to obtain the text embedding vector corresponding to each word, thereby obtaining the first text embedding vector sequence corresponding to the text to be processed and the video placeholders.
[0028] Step S302: Segment the first text embedding vector sequence to obtain multiple word embedding sub-vectors. The first text embedding vector sequence includes the text embedding vector corresponding to the text to be processed and the text embedding vector corresponding to the image placeholder. Optionally, the first text embedding vector sequence can be segmented according to the image placeholder, for example, according to the number of image placeholders, to obtain multiple word embedding sub-vectors, each word embedding sub-vector including a word embedding vector corresponding to an image placeholder.
[0029] As an optional example, the first text embedding vector sequence includes: TTTT TT TT. "T" represents the text embedding vector corresponding to the text to be processed. "" represents the text embedding vector corresponding to the image placeholder. TTTT TT TT is segmented to obtain word embedding subvectors TTTT TT and word embedding subvectors TT.
[0030] Step S303: In response to acquiring the image to be processed, the visual encoder based on the large image-text model performs feature extraction processing on the image to be processed to obtain the visual feature vector of the image to be processed.
[0031] The image to be processed is either an image input to the large-scale image-text model or a video frame extracted from the video stream of the input large-scale image-text model. In response to receiving the image to be processed, the large-scale image-text model's visual encoder, such as a ViT model, directly performs feature extraction processing on the image to obtain its visual feature vector. It is worth noting that in this embodiment, feature extraction processing is performed on each image to be processed received, eliminating the need to wait for the accumulation of multiple images before performing feature extraction, thereby shortening the inference latency of the large-scale image-text model.
[0032] Figure 4 This diagram illustrates the reasoning process of the large-scale graphic model according to an embodiment of this application. For example... Figure 4 As shown, image 1 is input into the image-text big data model, and the visual encoder of the big data model, such as the ViT model, extracts the visual feature vector of image 1. Image 2 is input into the big data model, and the visual encoder of the big data model, such as the ViT model, extracts the visual feature vector of image 2. Image 3 is input into the big data model, and the visual encoder of the big data model, such as the ViT model, extracts the visual feature vector of image 3. And so on, image N is input into the big data model, and the visual encoder of the big data model, such as the ViT model, extracts the visual feature vector of image N. In this embodiment, feature extraction processing is performed on each image to be processed upon receipt, without waiting for the accumulation of multiple images before feature extraction processing. Compared to... Figure 2 The prior art shown uses 4 frames per second (which can be changed to other values, such as 2 or 6) and takes 12 images for 3 consecutive seconds (which can be changed to other values, such as 4 or 5) to extract visual features. This application can extract visual features when it receives the first image, which effectively reduces the processing latency of large image and text models.
[0033] In an optional embodiment, if multiple images to be processed are simultaneously input into the large image-text model, the visual encoder can perform feature extraction processing on the multiple images to be processed in parallel.
[0034] Step S304: Use the visual feature vector of the image to be processed to replace the vector corresponding to the image placeholder in the word embedding sub-vector of the image to be processed. For example, the word embedding sub-vector includes TTTT. TT、 TT. The word embedding sub-vector corresponding to the image to be processed is TTTT. TT, then the word embedding sub-vector is replaced with TTTT using the visual feature vector of the image to be processed. Vectors corresponding to image placeholders in TT Optionally, the image-text big data model can determine the word embedding sub-vector corresponding to the image to be processed based on the image's identifier (e.g., number) and / or cue word text.
[0035] Step S305: In response to the completion of image placeholder replacement in the word embedding sub-vector, the language model based on the large graph-text model performs inference on the replaced word embedding sub-vector.
[0036] For example, the visual feature vector III of the image to be processed is replaced with the word embedding subvector TTTT. The vector corresponding to the image placeholder in TT is used to obtain TTTTIIITT. The large image-text model is used for inference based on TTTTIIITT to obtain the inference result.
[0037] like Figure 4 As shown, the visual feature vector of image 1 is used to replace the vector corresponding to the image placeholder in the word embedding sub-vector, and a prefill process is performed on the replaced word embedding sub-vector. The visual feature vector of image 2 is used to replace the vector corresponding to the image placeholder in the word embedding sub-vector, and a prefill process is performed on the replaced word embedding sub-vector. The visual feature vector of image 3 is used to replace the vector corresponding to the image placeholder in the word embedding sub-vector, and a prefill process is performed on the replaced word embedding sub-vector. This process is repeated, and after the prefill process is completed, the decoding is performed based on the prefill processing result to obtain the output result of the image-text large model (i.e., the inference result).
[0038] The image-text processing method provided in this application embodiment, upon acquiring an image to be processed, performs feature extraction processing on the image based on a large image-text model to obtain a visual feature vector of the image to be processed. That is, this application embodiment can call the visual encoder for feature extraction processing for each image to be processed received, without accumulating multiple images, thus shortening the processing latency. Upon receiving text to be processed, vector conversion is performed on the text to be processed and image placeholders to obtain a first text embedding vector sequence. The first text embedding vector sequence is segmented to obtain multiple word embedding sub-vectors. The visual feature vector of the image to be processed is then used to replace... The image placeholders in the word embedding sub-vectors corresponding to the image to be processed are replaced. In response to the completion of the replacement of the image placeholders in the word embedding sub-vectors, the language model based on the large graph-text model performs inference on the replaced word embedding sub-vectors. That is, according to the number of image placeholders, the first text embedding vector sequence is divided into multiple word embedding sub-vectors, and the vectors corresponding to the image placeholders in the word embedding sub-vectors are replaced respectively. Then, the vectors corresponding to the image placeholders in the first text embedding vector sequence are replaced in segments. The Prefill process can be performed in segments by making full use of the time waiting for subsequent images, which further shortens the processing latency of the large graph-text model.
[0039] The image and text processing method provided in this application embodiment can be used in car sentry scenarios to monitor and provide intelligent early warnings through images or videos collected by vehicle-mounted cameras. For example, it can monitor for behaviors such as vehicle collisions, door opening accidents, personnel scraping, object scraping, kicking, smashing, pulling door handles, and unplugging charging guns.
[0040] In an optional embodiment, before replacing the image placeholders in the corresponding word embedding sub-vectors of the image to be processed using the visual feature vectors of the image to be processed, the image-text processing method further includes: aligning the dimensions of the visual features of the image to be processed with the dimensions of the word embedding sub-vectors. Vector dimension alignment refers to making vectors of different dimensions consistent through linear transformation or mapping, so that subsequent model processing can proceed.
[0041] In an optional embodiment, the process of reasoning about the replaced word embedding sub-vectors using a language model based on a large graph-text model includes: In response to the completion of image placeholder replacement in the word embedding sub-vector, the language model based on the large graph-text model performs prefill processing on the replaced word embedding sub-vector; In response to the completion of image placeholder replacement for the last word embedding sub-vector among multiple word embedding sub-vectors, the Prefill processing results corresponding to all word embedding sub-vectors are decoded to obtain the processing results of the large image-text model.
[0042] Prefill and Decode are two key stages in language model inference, corresponding to input processing and output generation, respectively. Independent scheduling is achieved through a decoupled architecture (such as DistServe) to improve efficiency. The Prefill stage generates a key-value cache by embedding the replaced words into sub-vectors. The Decode stage generates the next token based on the cache, progressively outputting the inference result.
[0043] Figure 5 A schematic flowchart of another embodiment of the graphic processing method of this application is shown. Figure 5 The image processing method shown is used to process video streams. For example... Figure 5 As shown, the method includes: (1) Input the text to be processed and the video to be processed into the large image model. The text to be processed may include prompts and may also include historical conversations. Perform the following processes (2)-(4) on the text to be processed.
[0044] (2) The text to be processed and the video placeholders are segmented to obtain multiple words.
[0045] (3) Perform vector transformation on multiple words to obtain the first text embedding vector sequence corresponding to the multiple words.
[0046] (4) Based on the video placeholders, the first text embedding vector sequence is segmented to obtain multiple word embedding sub-vectors. The number of word embedding sub-vectors is the same as the number of video placeholders.
[0047] (5) Perform frame extraction on the video stream (e.g., extract one frame at a specified interval) to obtain multiple images to be processed, such as N images to be processed. N is a positive integer greater than or equal to 1.
[0048] (6) Input the images to be processed into the large image model respectively, or input multiple images to be processed into the large image model at the same time.
[0049] (7) The visual encoder in the large image-text model receives the image to be processed and directly performs feature extraction processing on the image to be processed to obtain the visual feature vector corresponding to the image to be processed. Or, the visual encoder in the large image-text model receives multiple images to be processed and performs feature extraction processing on the multiple images to be processed in batches to obtain the visual feature vector corresponding to each image to be processed.
[0050] (8) Align the dimensions of the visual feature vector corresponding to the image to be processed with the word embedding sub-vector so that the language model in the large image-text model can process it.
[0051] (9) Replace the vector corresponding to the video placeholder in the corresponding word embedding sub-vector with the visual feature vector corresponding to the image to be processed. For example, replace the first word embedding sub-vector in the first text embedding vector sequence with the visual feature vector corresponding to the first image to be processed in the input image-text big data model.
[0052] (10) Prefill the replaced word embedding subvector.
[0053] (11) Decode the result of the Prefill process to obtain the reasoning result of the large graphic model.
[0054] Compared to prior art, the image processing method provided in this application fully utilizes the time previously required to accumulate multiple images. It directly performs feature extraction upon receiving the image to be processed. Specifically, this application can call the visual encoder for feature extraction as soon as it receives each image to be processed, eliminating the need to accumulate multiple images and shortening processing latency. Upon receiving the text to be processed, it performs vector conversion between the text to be processed and video placeholders to obtain a first text embedding vector sequence. The first text embedding vector sequence is then segmented to obtain multiple word embedding sub-vectors. Finally, the visual feature vectors of the image to be processed are used to replace the text to be processed. The video placeholders in the word embedding sub-vectors corresponding to the image are replaced. In response to the completion of the replacement of the image placeholders in the word embedding sub-vectors, the language model based on the large graph-text model performs inference on the replaced word embedding sub-vectors. That is, according to the number of video placeholders, the first text embedding vector sequence is divided into multiple word embedding sub-vectors, and the vectors corresponding to the image placeholders in the word embedding sub-vectors are replaced respectively. Then, the vectors corresponding to the image placeholders in the first text embedding vector sequence are replaced in segments. The prefill process can be performed in segments by making full use of the time waiting for subsequent images, which further shortens the processing latency of the large graph-text model.
[0055] Figure 6 A schematic diagram of the graphic processing apparatus provided in an embodiment of this application is shown. Figure 6 As shown, the image processing apparatus 600 includes: The conversion module 601 is used to perform vector conversion on the text to be processed and the image placeholders to obtain a first text embedding vector sequence; The segmentation module 602 is used to segment the first text embedding vector sequence to obtain multiple word embedding sub-vectors; The visual feature extraction module 603 is used to perform feature extraction processing on the image to be processed based on the large image-text model in response to the acquisition of the image to be processed, so as to obtain the visual feature vector of the image to be processed. Replacement module 604 is used to replace the image placeholder in the word embedding sub-vector corresponding to the image to be processed using the visual feature vector of the image to be processed; The inference module 605 is used to infer the replaced word embedding vector based on the language model of the large graph-text model in response to the completion of the replacement of the word embedding vector corresponding to the image placeholder in the word embedding sub-vector.
[0056] In an optional embodiment, the segmentation module is used to: segment the first text embedding vector sequence according to the number of image placeholders to obtain multiple word embedding sub-vectors, each word embedding sub-vector including a word embedding vector corresponding to an image placeholder.
[0057] In an optional embodiment, the replacement module is further configured to: align the dimension of the visual feature vector of the image to be processed with the dimension of the word embedding sub-vector.
[0058] In an optional embodiment, the inference module is configured to: in response to the completion of word embedding vector replacement corresponding to the image placeholder in the word embedding sub-vector, perform prefill processing on the replaced word embedding sub-vector based on the language model of the large graph-text model; in response to the completion of image placeholder replacement in the last word embedding sub-vector among the plurality of word embedding sub-vectors, decode the prefill processing results corresponding to all word embedding sub-vectors to obtain the processing result of the large graph-text model.
[0059] The above-described apparatus can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.
[0060] It should be noted that the image and text processing method provided in this application embodiment can be executed by an image and text processing device, or by a control module in the image and text processing device for executing the loading image and text processing method.
[0061] The image and text processing device in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.
[0062] The image processing device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.
[0063] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.
[0064] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In this application, the electronic device includes various types of devices such as terminal devices and servers (clusters).
[0065] The embodiments of this disclosure can be implemented as an apparatus configured as desired using any suitable hardware, firmware, software, or any combination thereof, including electronic devices such as terminal devices and servers (clusters). Figure 7 Exemplary electronic devices that can be used to implement the various embodiments described in this application are shown. Figure 7 As shown, the electronic device includes: The system includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704. The processor 701, communication interface 702, and memory 703 communicate with each other via the communication bus 704. Memory 703 is used to store computer programs; When processor 701 executes a program stored in memory 703, it performs the following steps: Perform vector transformation on the text and image placeholders to be processed to obtain the first text embedding vector sequence; The first text embedding vector sequence is segmented to obtain multiple word embedding sub-vectors; In response to acquiring the image to be processed, a visual encoder based on a large image-text model performs feature extraction processing on the image to be processed to obtain the visual feature vector of the image to be processed. The image placeholders in the word embedding subvector corresponding to the image to be processed are replaced using the visual feature vector of the image to be processed. In response to the completion of the replacement of the word embedding vector corresponding to the image placeholder in the word embedding sub-vector, the language model based on the large graph-text model performs inference on the replaced word embedding sub-vector.
[0066] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.
[0067] The communication interface is used for communication between the aforementioned terminal and other devices.
[0068] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0069] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0070] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the graphic processing methods described in the above embodiments.
[0071] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).
[0072] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0073] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0074] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.
Claims
1. A method for processing images and text, characterized in that, include: Perform vector transformation on the text and image placeholders to be processed to obtain the first text embedding vector sequence; The first text embedding vector sequence is segmented to obtain multiple word embedding sub-vectors; In response to acquiring the image to be processed, a visual encoder based on a large image-text model performs feature extraction processing on the image to be processed to obtain the visual feature vector of the image to be processed. The image placeholders in the word embedding subvector corresponding to the image to be processed are replaced using the visual feature vector of the image to be processed. In response to the completion of the replacement of the word embedding vector corresponding to the image placeholder in the word embedding sub-vector, the language model based on the large graph-text model performs inference on the replaced word embedding sub-vector.
2. The method according to claim 1, characterized in that, The step of segmenting the first text embedding vector sequence to obtain multiple word embedding sub-vectors includes: Based on the number of image placeholders, the first text embedding vector sequence is segmented to obtain multiple word embedding sub-vectors, each of which includes a word embedding vector corresponding to an image placeholder.
3. The method according to claim 1, characterized in that, Before replacing the image placeholder in the word embedding sub-vector corresponding to the image to be processed with the visual feature vector of the image to be processed, the method further includes: aligning the dimension of the visual feature vector of the image to be processed with the dimension of the word embedding sub-vector.
4. The method according to any one of claims 1-3, characterized in that, The replacement of the word embedding vector corresponding to the image placeholder in the word embedding sub-vector is completed. Based on the language model of the large graph-text model, inference is performed on the replaced word embedding sub-vector, including: In response to the completion of the replacement of the word embedding vector corresponding to the image placeholder in the word embedding sub-vector, the language model based on the large graph-text model performs prefill processing on the replaced word embedding sub-vector; In response to the completion of image placeholder replacement for the last word embedding sub-vector among the plurality of word embedding sub-vectors, the Prefill processing results corresponding to all word embedding sub-vectors are decoded to obtain the processing results of the large image-text model.
5. A graphic processing device, characterized in that, include: The conversion module is used to perform vector conversion on the text and image placeholders to be processed, and obtain the first text embedding vector sequence. The segmentation module is used to segment the first text embedding vector sequence to obtain multiple word embedding sub-vectors; The visual feature extraction module is used to extract features from the image to be processed based on the large image-text model in response to the acquisition of the image to be processed, and to obtain the visual feature vector of the image to be processed. The replacement module is used to replace the image placeholder in the word embedding sub-vector corresponding to the image to be processed using the visual feature vector of the image to be processed; The inference module is used to infer the replaced word embedding vector based on the language model of the large graph-text model in response to the completion of the replacement of the word embedding vector corresponding to the image placeholder in the word embedding sub-vector.
6. The apparatus according to claim 5, characterized in that, The segmentation module is used for: Based on the number of image placeholders, the first text embedding vector sequence is segmented to obtain multiple word embedding sub-vectors, each of which includes a word embedding vector corresponding to an image placeholder.
7. The apparatus according to claim 5, characterized in that, The replacement module is further configured to: align the dimension of the visual feature vector of the image to be processed with the dimension of the word embedding sub-vector.
8. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the method as described in any one of claims 1-4.
9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the method as described in any one of claims 1-4.
10. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-4.