Data processing method, electronic device, storage medium, and computer program product

Abstract

The application relates to the technical field of artificial intelligence, and discloses a data processing method, an electronic device, a storage medium and a computer program product. The method comprises the following steps: an electronic device acquires visual data to be inferred, and the visual data comprises one or more first images. Then, each first image is split into a plurality of first sub-images, and each first sub-image is subjected to image coding by using a first visual encoder to obtain a plurality of first visual tokens. The size of each acquired first sub-image matches the processing resolution of the first visual encoder. Next, the plurality of first visual tokens can be compressed based on a preset compression rate to obtain a plurality of second visual tokens, wherein the number of the second visual tokens is less than the number of the first visual tokens. Finally, the plurality of second visual tokens after compression can be used to perform image inference by using an MLLM to obtain an inference result. In this way, the power consumption of the electronic device when running the MLLM can be reduced, and the inference efficiency of the MLLM can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, electronic device, storage medium, and computer program product. Background Technology

[0002] In recent years, with the significant improvement in the ability of multimodal large language models (MLLMs) to understand and process various modalities of data such as text, video, and audio, MLLMs have been more widely used in fields such as intelligent question answering and image recognition. For example, MLLMs can analyze image data and generate corresponding text descriptions.

[0003] MLLM consumes a significant amount of power during operation. Therefore, deploying MLLM on electronic devices with limited storage space or computing power (such as mobile terminals like smartphones) will lead to a decrease in MLLM inference efficiency.

[0004] Therefore, how to reduce the computational power consumption of MLLM so that it can still have high inference efficiency when deployed on electronic devices with limited storage space or computing power is an urgent problem to be solved. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a data processing method, an electronic device, a storage medium, and a computer program product to reduce power consumption and improve inference efficiency of MLLM when the electronic device is running.

[0006] Firstly, this application provides a data processing method applied to an electronic device, the electronic device including an MLLM (Multi-Layered Machine), the MLLM being used to process visual data. The data processing method may include: acquiring visual data to be inferred, the visual data including one or more first images; splitting each of the one or more first images into multiple first sub-images, and performing image encoding on the multiple first sub-images using a first visual encoder to obtain multiple first visual feature tokens; wherein the size of each of the multiple first sub-images matches the processing resolution of the first visual encoder. Next, compressing the multiple first visual tokens based on a preset compression rate to obtain multiple compressed second visual tokens, the number of compressed second visual tokens being less than the number of uncompressed first visual tokens; and obtaining the inference result corresponding to the visual data using the MLLM based on the multiple second visual tokens.

[0007] Thus, by using the above method, the first visual token corresponding to the visual data can be compressed based on a preset compression rate, thereby obtaining a smaller number of second visual tokens. This reduces the computational process and storage requirements when performing inference based on visual tokens, improves the inference efficiency of electronic devices when performing inference on visual data using MLLM, and reduces the power consumption of electronic devices running MLLM.

[0008] In one possible implementation of the first aspect described above, splitting each of the first images in one or more first images into multiple first sub-images includes: determining a dynamic patch size based on the base block patch size of the first visual encoder and a preset compression rate; adjusting the size of at least one of the first images in one or more first images based on the dynamic patch size to obtain a corresponding second image, wherein the size of the second image is greater than or equal to the size of the corresponding first image, and the size of the second image is an integer multiple of the dynamic patch size. This ensures that the size of the second image can be divided into multiple dynamic patch sizes without repetition or redundancy, thereby ensuring that the visual encoder does not omit the content of the second image when encoding in patches as the smallest unit, and thus guaranteeing the integrity of the visual tokens generated based on each patch.

[0009] Then, after acquiring the second image, the electronic device can determine the overlap size based on the size of the second image and the processing resolution of the first visual encoder. Based on the overlap size and the processing resolution of the first visual encoder, the second image is segmented into multiple first sub-images. The overlap size includes an overlap height and / or an overlap width; and, in the height direction of the second image, there is an overlap region with a height equal to the overlap height between the bottom of the i-th first sub-image and the top of the adjacent (i+1)-th first sub-image, where i is a positive integer; and / or, in the width direction of the second image, there is an overlap region with a width equal to the overlap width between the right side of the j-th first sub-image and the left side of the adjacent (j+1)-th first sub-image, where j is a positive integer.

[0010] In this way, by calculating the overlap size, the second image can be segmented into multiple first sub-images without adjusting the size of the second image, and the size of each first sub-image can be matched with the processing resolution of the first visual encoder.

[0011] In one possible implementation of the first aspect described above, the dimensions of the second image include the height and width of the second image.

[0012] For example, the height of the second image can satisfy the following formula:

[0013] ;

[0014] Where H2 represents the height of the second image, H1 represents the height of the first image, and N token Indicates the dynamic patch size. This indicates rounding to the nearest integer.

[0015] As another example, the width of the second image can satisfy the following formula:

[0016] ;

[0017] Where W2 represents the width of the second image, W1 represents the width of the first image, and N token Indicates the dynamic patch size. This indicates rounding to the nearest integer.

[0018] In one possible implementation of the first aspect above, the overlap height can satisfy the following formula:

[0019] ;

[0020] Where H3 represents the overlap height, H0 represents the height in the processing resolution, and H2 represents the height of the second image. This indicates rounding to the nearest integer.

[0021] For another example, the overlap width can satisfy the following formula:

[0022] ;

[0023] Where W3 represents the overlap width, W0 represents the width in the processing resolution, and W2 represents the width of the second image. This indicates rounding to the nearest integer.

[0024] In one possible implementation of the first aspect above, the above-mentioned image encoding of multiple first sub-images by a first visual encoder to obtain multiple first visual tokens includes: image encoding of multiple first sub-images by a first visual encoder to obtain a visual token corresponding to each first sub-image, wherein the visual token of each first sub-image includes a visual token corresponding to an overlapping region and a visual token corresponding to a non-overlapping region; when n first sub-images include the same first overlapping region, saving the visual token corresponding to the first overlapping region in one of the n first sub-images and deleting the visual tokens corresponding to the first overlapping region in the other n-1 first sub-images, where n is a positive integer greater than 1; and obtaining multiple first visual tokens based on the saved visual tokens corresponding to each overlapping region and the visual tokens corresponding to the non-overlapping regions in each first sub-image.

[0025] Thus, by removing duplicate visual tokens, data redundancy can be reduced. This, in turn, reduces the computational or storage requirements for inference based on visual tokens, improves the inference efficiency of electronic devices using MLLM to infer visual data, and lowers the power consumption of electronic devices running MLLM.

[0026] In one possible implementation of the first aspect above, the above-mentioned image encoding of multiple first sub-images by a first visual encoder to obtain a visual token corresponding to each first sub-image includes: in the process of image encoding of multiple first sub-images by a first visual encoder, dividing each first sub-image into multiple patches based on a dynamic patch size, and encoding the multiple patches to obtain a visual token corresponding to each first sub-image.

[0027] Thus, compared to encoding each first sub-image based on a base patch of the first visual encoder, in this embodiment, when each first sub-image is divided into multiple patches based on a larger dynamic patch size, the first visual encoder can segment the received first sub-image into fewer patches. Therefore, the number of visual tokens can be reduced when mapping each patch to a fixed-dimensional visual token, resulting in lower power consumption when the electronic device performs inference on visual data based on visual tokens.

[0028] In one possible implementation of the first aspect above, the visual data is a video comprising multiple first images, with a preset compression rate of a first compression rate; or, the visual data is an image set comprising multiple first images, with a preset compression rate of a second compression rate; or, the visual data is a single first image, with a preset compression rate of a third compression rate; wherein the first compression rate is greater than the second compression rate, and the second compression rate is greater than the third compression rate.

[0029] Thus, when the first compression ratio is high, the number of second visual tokens for a single first image in the video will be less, and consequently, the number of second visual tokens for multiple first images in the video will also be less. Similarly, when the second compression ratio is greater than the third compression ratio, the number of second visual tokens for a single first image in the image set will be less, and consequently, the number of second visual tokens for multiple first images in the video will also be less. Furthermore, this reduces the computational process and storage requirements when performing inference based on visual tokens for various types of visual data.

[0030] In one possible implementation of the first aspect described above, compressing multiple first visual tokens based on a preset compression ratio to obtain multiple compressed second visual tokens includes: dividing the multiple first visual tokens into one or more groups of first visual tokens. Each group of first visual tokens comprises k×k adjacent first visual tokens, and each k×k first visual token comprises data of k×k×r dimensions, where k is the preset compression ratio, r is the dimension of the first visual token, k is a positive integer greater than 1, and r is a positive integer. The k×k×r dimension data corresponding to each group of first visual tokens is compressed to 1×r dimension data to obtain multiple second visual tokens.

[0031] In this way, by compressing k×k×r dimension data into 1×r dimension data, the number of compressed second visual tokens can be reduced. This, in turn, reduces the computational process and storage requirements when performing inference based on visual tokens, improves the inference efficiency of electronic devices when using MLLM to infer visual data, and reduces the power consumption of electronic devices running MLLM.

[0032] In one possible implementation of the first aspect described above, the electronic device includes a plurality of visual encoders, wherein a first visual encoder among the plurality of visual encoders satisfies one or more of the following first conditions: the size of the first visual encoder is smaller than the size of the other visual encoders among the plurality of visual encoders; the base patch size of the first visual encoder is smaller than the base patch size of the other visual encoders; and the processing resolution of the first visual encoder is greater than the processing resolution of the other visual encoders.

[0033] Thus, before encoding the visual data, a first visual encoder can be selected from multiple visual encoders based on one or more of the aforementioned first conditions, in order to reduce the power consumption of the electronic device when running the first visual encoder to encode the visual data, and to improve the accuracy of the visual data inference results.

[0034] In one possible implementation of the first aspect described above, before encoding the multiple first sub-images using the first visual encoder, the method further includes: obtaining one or more of the processing resolution, scale, or basic patch size of each visual encoder among the multiple visual encoders; arranging the visual encoders in descending order according to their processing resolution, and assigning a first number to each visual encoder in ascending order; and / or arranging the visual encoders in ascending order according to their scale, and assigning a second number to each visual encoder in ascending order; and / or arranging the visual encoders in ascending order according to their basic patch size, and assigning a third number to each visual encoder in ascending order; and / or obtaining a total number corresponding to each visual encoder based on one or more of the first number, second number, or third number corresponding to each visual encoder; and selecting the visual encoder with the smallest total number as the first visual encoder from among the total numbers corresponding to each visual encoder.

[0035] In this way, a first visual encoder can be selected from multiple visual encoders to reduce the power consumption of electronic devices when running the first visual encoder to encode visual data, and to improve the accuracy of visual data inference results.

[0036] In one possible implementation of the first aspect above, selecting the visual encoder with the smallest total number from the total numbers corresponding to each visual encoder as the first visual encoder includes: if there are at least two visual encoders whose total numbers are the smallest total number, the visual encoder that satisfies the first condition the most is selected as the first visual encoder.

[0037] In this way, the optimal first visual encoder can be selected from multiple visual encoders with the same total number.

[0038] In one possible implementation of the first aspect above, the method further includes: obtaining text data to be inferred; obtaining the inference result corresponding to the visual data through a multimodal large language model based on multiple second visual tokens includes: mapping the second visual tokens in the visual space to a third visual token in the text space; obtaining the inference result corresponding to the visual data and the text data through MLLM based on the third visual token and the first text token corresponding to the text data.

[0039] Thus, the above methods can be used to perform inference on multimodal data such as visual and textual data. Furthermore, it can improve inference efficiency and reduce the power consumption of electronic devices.

[0040] In one possible implementation of the first aspect above, the above-mentioned reasoning result obtained from the visual data and text data based on the third visual token and the first text token corresponding to the text data through MLLM includes: obtaining the first text token corresponding to the text data through a text segmenter; inputting the third visual token and the first text token into the large language neural network submodule in the MLLM to obtain the reasoning result corresponding to the visual data and text data.

[0041] In a second aspect, this application also provides an electronic device, comprising: at least one memory and at least one processor, wherein the memory is coupled to the processor; the memory is used to store computer program code / instructions; when the computer program code / instructions are executed by the processor, the electronic device performs the data processing method mentioned in the first aspect and any possible implementation thereof.

[0042] Thirdly, this application also provides a readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the data processing methods mentioned in the first aspect and any possible implementation thereof.

[0043] Fourthly, this application also provides a computer program product, comprising: computer instructions that, when executed on an electronic device, cause the electronic device to perform the data processing method mentioned in the first aspect and any possible implementation thereof.

[0044] Fifthly, this application also provides a chip including a processor for reading and executing computer code / instructions stored in a memory to perform the data processing methods mentioned in the first aspect and any possible implementation thereof.

[0045] The beneficial effects of the second to fifth aspects mentioned above can be referred to the relevant descriptions in the first aspect and any possible implementation of the first aspect, which will not be repeated here. Attached Figure Description

[0046] Figure 1A According to some embodiments, a schematic diagram of an application scenario for the first type of MLLM is shown;

[0047] Figure 1B According to some embodiments, a schematic diagram of a second type of MLLM application scenario is shown;

[0048] Figure 1C According to some embodiments, a schematic diagram of a third application scenario of MLLM is shown;

[0049] Figure 2 According to some embodiments of this application, a schematic flowchart of a data processing method is shown;

[0050] Figure 3A According to some embodiments of this application, a flowchart illustrating the process of splitting a first image into multiple first sub-images is shown;

[0051] Figure 3B According to some embodiments of this application, a schematic diagram of converting a first image into a second image is shown;

[0052] Figure 3C According to some embodiments of this application, another schematic diagram of converting a first image into a second image is shown;

[0053] Figure 3D According to some embodiments of this application, a schematic diagram of a scenario for determining an overlapping region is shown;

[0054] Figure 4 According to some embodiments of this application, a schematic diagram of a scenario for compressing visual tokens is shown;

[0055] Figure 5 According to some embodiments of this application, a schematic diagram of a scenario for reasoning on visual data and text data is shown;

[0056] Figure 6 According to some embodiments of this application, a schematic flowchart for selecting a first visual encoder is shown;

[0057] Figure 7A According to some embodiments of this application, a power consumption diagram of an MLLM employing the data processing method of this application is shown;

[0058] Figure 7B According to some embodiments of this application, the test performance of an MLLM using the data processing method of this application on various datasets is shown;

[0059] Figure 8 According to some embodiments of this application, a schematic diagram of the hardware structure of an electronic device is shown. Detailed Implementation

[0060] The illustrative embodiments of this application include, but are not limited to, a data processing method, an electronic device, a storage medium, and a computer program product.

[0061] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0062] As mentioned earlier, with the significant improvement in MLLM's ability to understand and process various modal data such as text, video, and audio, MLLM is being used more widely in fields such as intelligent question answering and image recognition.

[0063] For example, Figures 1A to 1C This demonstrates various application scenarios for MLLM.

[0064] For example, refer to Figure 1A As shown, MLLM can analyze a single image and extract key information such as names, ID numbers, or addresses. For example, see reference... Figure 1B As shown, MLLM can also analyze multiple images simultaneously and provide comparative descriptions of them. For example, refer to... Figure 1C As shown, MLLM can also be used to analyze videos and output corresponding text descriptions. Therefore, MLLM has wide applications in fields such as intelligent question answering, image recognition, and video recognition.

[0065] However, MLLM consumes a significant amount of power during operation. Therefore, deploying MLLM on electronic devices with limited storage space or computing power (such as mobile terminals like smartphones) will lead to a decrease in MLLM inference efficiency.

[0066] To address the aforementioned problems, this application provides a data processing method. Specifically, an electronic device acquires visual data to be inferred, wherein the visual data may include one or more first images. The electronic device then divides each first image into multiple first sub-images and performs image encoding on each first sub-image using a first visual encoder, thereby obtaining multiple first visual feature tokens. The size of each acquired first sub-image matches the processing resolution of the first visual encoder so that it can be input into the first visual encoder for image encoding. Next, the electronic device compresses the multiple first visual tokens based on a preset compression rate to obtain multiple compressed second visual tokens, such that the number of compressed second visual tokens is less than the number of uncompressed first visual tokens. Finally, the electronic device performs image inference using MLLM based on the compressed multiple second visual tokens, thereby obtaining the inference result corresponding to the visual data.

[0067] It should be noted that compressing the first visual token can involve deleting redundant data (such as duplicate texture data), fusing similar data (such as clustering similar information), or deleting data that has little impact on the inference result (such as image background). This allows the compressed second visual token to retain data that has a significant impact on the accuracy of the inference result, thereby ensuring the accuracy of the inference result.

[0068] It's also important to note that MLLM is not a single neural network model, but rather an integrated system composed of multiple components, modules, or models. It typically includes modules such as a visual encoder and a large language model for encoding or reasoning about visual data. For example, ChatGPT-4V... TM It is simply an MLLM. Therefore, the data processing method performed by the electronic device mentioned in this application can also refer to the data processing method performed by the electronic device through an MLLM.

[0069] Thus, in the method provided in this application, the first visual token corresponding to the visual data can be compressed based on a preset compression ratio, thereby obtaining a smaller number of second visual tokens. This reduces the computational process and storage requirements when performing inference based on visual tokens, improves the inference efficiency of electronic devices when performing inference on visual data using MLLM, and reduces the power consumption of electronic devices running MLLM. Furthermore, the compressed second visual tokens can retain data that significantly affects the accuracy of the inference results, thus ensuring the accuracy of the inference results.

[0070] In some embodiments, a visual token can refer to a fixed-dimensional vector that maps image features.

[0071] In some embodiments, when the visual data is video, it includes more first images, resulting in a greater total number of first visual tokens before compression. Alternatively, a collection of images comprising multiple first images will also generate more first visual tokens compared to a single first image. Therefore, when compressing the first visual tokens based on a preset compression rate, the preset compression rate can vary depending on whether the visual data is video, a single first image, or an image collection comprising multiple first images, allowing different types of visual data to have fewer second visual tokens.

[0072] For example, when the visual data is a video comprising multiple first images, the corresponding preset compression rate can be a first compression rate. When the visual data is an image set comprising multiple first images, the corresponding preset compression rate can be a second compression rate. When the visual data is a single first image, the corresponding preset compression rate can be a third compression rate. The first compression rate is greater than the second compression rate, and the second compression rate is greater than the third compression rate.

[0073] Thus, when compressing multiple first visual tokens corresponding to a single first image using a third compression rate, or compressing multiple first visual tokens corresponding to an image set using a larger second compression rate, or compressing multiple first visual tokens corresponding to a video using an even larger first compression rate, various types of visual data can have a smaller number of second visual tokens. Consequently, when performing inference on various types of visual data, the computational process and storage requirements based on visual tokens can be reduced, improving the inference efficiency of electronic devices when using MLLM to infer visual data, and reducing the power consumption of electronic devices running MLLM.

[0074] It is understood that, since the data processing method provided in this application has relatively low requirements for the computing and storage capabilities of electronic devices, it can be applied to any electronic device. These electronic devices include, but are not limited to, mobile stations (MS) and mobile terminals (MT). For example, electronic devices can be mobile phones, smart TVs, wearable devices, tablets, desktop computers, laptops, virtual reality (VR) devices, augmented reality (AR) devices, terminals in industrial control, self-driving, remote medical surgery, smart grids, transportation safety, smart cities, and smart homes. This application does not limit the specific form of the electronic device.

[0075] The following is based on Figure 2 The flowchart shown illustrates the data processing method mentioned in the embodiments of this application. This data processing method can be applied to electronic devices, such as any electronic device like the mobile phone mentioned above. The electronic device may include an MLLM (Multi-Level Machine), which can be used to process visual data. Figure 2 As shown, the specific data processing method is as follows:

[0076] S21: Obtain the visual data to be inferred.

[0077] In some embodiments, visual data may include one or more first images. For example, visual data may be a single first image; or, for another example, visual data may be a collection of images including multiple first images; or, for yet another example, visual data may be a video including multiple first images.

[0078] S22: Divide each of the first images in one or more first images into multiple first sub-images.

[0079] The size of each first sub-image is matched with the processing resolution of the first visual encoder.

[0080] In some embodiments, when performing inference on visual data, the visual data first needs to be encoded by a first visual encoder. The processing resolution of the first visual encoder (i.e., the size of the input image it can receive) is fixed. Therefore, when the size of the first image in the visual data matches the processing resolution of the first visual encoder—for example, when the size of the first image is the same as the processing resolution of the first visual encoder—the first image can be directly used as the first sub-image and input into the first visual encoder for encoding.

[0081] In other embodiments, when the size of the first image in the visual data does not match the processing resolution of the first visual encoder (e.g., the size of the first image is larger or smaller than the processing resolution of the first visual encoder), the electronic device needs to process the first image to obtain a first sub-image that matches the processing resolution of the first visual encoder. Then, the electronic device can input the obtained first sub-image into the first visual encoder for encoding. Specifically, the process by which the electronic device processes the first image to obtain a first sub-image that matches the processing resolution of the first visual encoder can be referred to... Figure 3A As shown in S221 to S224.

[0082] S221: Determine the dynamic patch size based on the base block (patch) size of the first visual encoder and the preset compression rate.

[0083] In some embodiments, the preset compression rate can be the compression rate mentioned above when compressing the first visual token. For example, when the visual data is a video including multiple first images, the corresponding preset compression rate can be a first compression rate. When the visual data is an image set including multiple first images, the corresponding preset compression rate can be a second compression rate. When the visual data is a single first image, the corresponding preset compression rate can be a third compression rate. The first compression rate is greater than the second compression rate, and the second compression rate is greater than the third compression rate. For details, please refer to the foregoing description, which will not be repeated here.

[0084] In some embodiments, the first visual encoder can segment the received first sub-image into multiple patches and map each patch to a fixed-dimensional visual token. To reduce the number of visual tokens generated by the visual encoder, the electronic device can increase the base patch size based on a preset compression ratio. Thus, after increasing the base patch size based on the preset compression ratio to obtain a larger dynamic patch size, the first visual encoder can segment the received first sub-image into fewer patches, thereby reducing the number of visual tokens when mapping each patch to a fixed-dimensional visual token. Consequently, the electronic device can generate less power consumption when performing inference on visual data based on visual tokens.

[0085] In some embodiments, the electronic device may use the product of the base patch size and the preset compression rate as the dynamic patch size. For example, the calculation process of the dynamic patch size can be referred to formula (1).

[0086] (1);

[0087] Where, N token N represents the dynamic patch size. patch R represents the base patch size of the first visual encoder. psf This indicates the preset compression ratio.

[0088] Thus, the dynamic patch size can be calculated using the above formula (1). For example, if the basic patch size of the first visual encoder is 16 and the preset compression rate is 2, the electronic device can determine the dynamic patch size as 32 based on the above formula (1).

[0089] S222: Adjust the size of at least one of the first images based on the dynamic patch size to obtain the corresponding second image.

[0090] In some embodiments, the size of the second image is greater than or equal to the size of the corresponding first image, and the size of the second image is an integer multiple of the dynamic patch size. That is, if the size of the first image is exactly an integer multiple of the dynamic patch size, the electronic device can directly use the first image as the second image without adjusting its size. Alternatively, if the size of the first image is not an integer multiple of the dynamic patch size, the electronic device needs to adjust the size of the first image to obtain the corresponding second image.

[0091] In some embodiments, the size of the second image is an integer multiple of the dynamic patch size, ensuring that the size of the second image can be divided into multiple dynamic patch sizes without repetition or redundancy. Specifically, when the first visual encoder encodes visual data, it can use the patch as the smallest unit, mapping each patch to a visual token of a fixed dimension. Therefore, in this embodiment, by ensuring that the size of the second image can be divided into multiple dynamic patch sizes without repetition or redundancy, it can be ensured that the visual encoder does not omit the content of the second image when encoding in patches as the smallest unit, thereby guaranteeing the integrity of the visual tokens generated based on each patch.

[0092] In some embodiments, the size of the second image includes the height and width of the second image, and the electronic device can calculate the height and width of the second image based on the following formulas (2) and (3), respectively.

[0093] (2);

[0094] Where H2 represents the height of the second image, H1 represents the height of the first image, and N token This represents the dynamic patch size determined based on the formula (1) above. This indicates rounding to the nearest integer.

[0095] Thus, the height of the second image can be calculated based on the above formula (2). For example, if the height H1 of the first image is 700, and the dynamic patch size determined based on the above formula (1) is 32, then the electronic device can determine the height of the second image as 704 based on the above formula (2). Furthermore, the height of the second image, 704, is an integer multiple of the dynamic patch size, 32.

[0096] As another example, the electronic device can also calculate the width of the second image based on the following formula (3).

[0097] (3);

[0098] Where W2 represents the width of the second image, W1 represents the width of the first image, and N token This represents the dynamic patch size determined based on the formula (1) above. This indicates rounding to the nearest integer.

[0099] Thus, the width of the second image can be calculated based on the above formula (3). For example, if the width W1 of the first image is 1000, and the dynamic patch size determined based on the above formula (1) is 32, then the electronic device can determine the width of the second image as 1024 based on the above formula (3). Furthermore, the width of the second image, 1024, is an integer multiple of the dynamic patch size, 32.

[0100] Thus, the size of the second image can be determined using formulas (2) and (3) above, and the size of the first image can be adjusted to match the size of the second image. For example, as... Figure 3B As shown, the electronic device can directly enlarge the first image 301 by stretching or other methods to obtain the second image 302. For example, as... Figure 3C As shown, the electronic device can also obtain the second image 303 by adding blank areas around the edges of the first image 301. This application does not limit this.

[0101] S223: Determine the overlap size based on the size of the second image and the processing resolution of the first visual encoder.

[0102] In some embodiments, before segmenting the second image into multiple first sub-images, it can be determined whether there are overlapping regions between the first sub-images. For example, as... Figure 3D As shown, the second image 304 can have a width of 1024 × a height of 704, while the dimensions of each first sub-image (such as image A1, image A2, image A3, image A4, image A5, and image A6) matching the processing resolution of the first visual encoder (e.g., a processing resolution of 384) can be 384 × 384. Since the width 1024 of the second image 304 is not an integer multiple of 384, and the height 704 of the second image 304 is also not an integer multiple of 384, the second image 304 cannot be segmented into multiple first sub-images without overlap or remainder. Therefore, the electronic device can also first calculate the overlap size between each first sub-image, such as calculating the overlap height and overlap width, so as to segment the second image 304 into multiple first sub-images based on the overlap size. Figure 3D The shaded areas in the diagram represent overlapping areas.

[0103] In some embodiments, when the electronic device calculates the overlap height, it means that in the height direction of the second image, there is an overlap region with a height equal to the overlap height between the bottom of the i-th (i is a positive integer) first sub-image and the top of the adjacent (i+1)-th first sub-image. For example, as... Figure 3DAs shown, in the height direction of the second image 304, there is an overlapping region with a height equal to the overlap height between the bottom of the first sub-image (e.g., image A1) and the top of the adjacent second sub-image (e.g., image A4). For example, as... Figure 3D As shown, in the height direction of the second image 304, there is an overlapping area with a height equal to the overlap height between the bottom of the first sub-image (e.g., image A2) and the top of the adjacent second sub-image (e.g., image A5).

[0104] For example, the electronic device can calculate the overlap height based on the following formula (4).

[0105] (4);

[0106] Where H3 represents the overlap height, H0 represents the height in the processing resolution, and H2 represents the height of the second image. This indicates rounding to the nearest integer.

[0107] Thus, based on the above formula (4), the height of the overlapping area between two adjacent first sub-images in the height direction can be calculated.

[0108] In other embodiments, when the electronic device calculates the overlap width, it means that in the width direction of the second image, there is an overlap region with a width equal to the overlap width between the right side of the j-th (j is a positive integer) first sub-image and the left side of the adjacent (j+1)-th first sub-image. For example, as Figure 3D As shown, in the width direction of the second image 304, there is an overlapping region with a width equal to the overlap width between the right side of the first sub-image (e.g., image A1) and the left side of the adjacent second sub-image (e.g., image A2). For example, as... Figure 3D As shown, in the width direction of the second image 304, there is an overlapping region with a width equal to the overlap width between the right side of the second first sub-image (e.g., image A2) and the left side of the adjacent third first sub-image (e.g., image A3). For example, as... Figure 3D As shown, in the width direction of the second image 304, there is an overlapping region with a width equal to the overlap width between the right side of the first sub-image (e.g., image A4) and the left side of the adjacent second sub-image (e.g., image A5).

[0109] For example, the electronic device can calculate the overlap width based on the following formula (5).

[0110] (5);

[0111] Where W3 represents the overlap width, W0 represents the width in the processing resolution, and W2 represents the width of the second image. This indicates rounding to the nearest integer.

[0112] Thus, based on the above formula (5), the width of the overlapping region between two adjacent first sub-images in the width direction can be calculated.

[0113] In some embodiments, if the electronic device determines that the width of the second image is an integer multiple of the width of the processing resolution of the first visual encoder, and the height of the second image is also an integer multiple of the height of the processing resolution, the electronic device may not calculate the overlap size. That is, if both the width and height of the second image are integer multiples of the processing resolution, the electronic device may not execute S223.

[0114] In some embodiments, the overlap size may include overlap height and / or overlap width. That is, when the electronic device performs S223 to calculate the overlap size, it may calculate the overlap height and overlap width, or it may calculate only one of the overlap height or overlap width. For example, if the electronic device determines that the width and height of the second image are not integer multiples of the processing resolution, the electronic device may calculate the overlap height and overlap width using the above formulas (4) and (5). Alternatively, if the electronic device determines that the width of the second image is an integer multiple of the width of the processing resolution, but the height of the second image is not an integer multiple of the height of the processing resolution, the electronic device may calculate only the overlap height using the above formula (4) without calculating the overlap width. Alternatively, if the electronic device determines that the height of the second image is an integer multiple of the height of the processing resolution, but the width of the second image is not an integer multiple of the width of the processing resolution, the electronic device may calculate only the overlap width using the above formula (5) without calculating the overlap height. This application does not limit this.

[0115] S224: Based on the overlap size and the processing resolution of the first visual encoder, the second image is segmented into multiple first sub-images.

[0116] In some embodiments, after determining the overlap size through S223, the electronic device can segment the second image into multiple first sub-images based on the overlap size. Furthermore, the size of each of the multiple first sub-images matches the processing resolution of the first visual encoder; for example, the size of each first sub-image is the same as the processing resolution of the first visual encoder. For example, such as... Figure 3D As shown, after calculating the overlap height and overlap width, the electronic device can segment the second image 304 into multiple first sub-images (such as images A1 to A6) based on the overlap height and overlap width.

[0117] In summary, through the above... Figure 3AThe method shown allows processing of the first image to obtain a first sub-image that matches the processing resolution of the first visual encoder. Then, the electronic device can execute step S23 below to input each of the first sub-images into the first visual encoder for encoding.

[0118] S23: Encode multiple first sub-images using a first visual encoder to obtain multiple first visual tokens.

[0119] In some embodiments, when reasoning about visual data, the visual data first needs to be encoded by a first visual encoder to obtain multiple encoded first visual tokens.

[0120] In some embodiments, an electronic device may typically include multiple visual encoders; however, in order to reduce power consumption when the electronic device runs a first visual encoder to encode visual data or to improve the inference accuracy of the electronic device for visual data, the first visual encoder may be a lightweight visual encoder among multiple visual encoders.

[0121] For example, the first visual encoder can satisfy at least one or more of the following first conditions: (1) The size of the first visual encoder is smaller than the size of other visual encoders in the electronic device. For example, the size of the visual encoder can be evaluated by the amount of computation and / or the number of parameters. That is, if the amount of computation and / or the number of parameters of the first visual encoder is smaller than the amount of computation and / or the number of parameters of other visual encoders, it can be said that the size of the first visual encoder is smaller than the size of other visual encoders. Alternatively, the size of the visual encoder can also be evaluated directly by the size of the memory space occupied by the visual encoder in the electronic device. (2) The size of the basic block (patch) of the first visual encoder is smaller than the size of the basic patch of other visual encoders. In some embodiments, the visual encoder can divide the received image into multiple patches and map each patch to a visual token of a fixed dimension. Therefore, when the patch size of the first visual encoder is smaller than the patch size of other visual encoders, the first visual encoder can divide the received image into more patches, and thus the first visual encoder can output more visual tokens to ensure the integrity of image feature encoding, thereby ensuring the accuracy of the results of visual data inference by the electronic device. (3) The processing resolution of the first visual encoder is greater than the processing resolution of other visual encoders. In some embodiments, if the size of the first image in the visual data is larger than the processing resolution of the visual encoder, the first image needs to be segmented into multiple sub-images matching the processing resolution when encoding the visual data, so that the sub-images can be input into the visual encoder for encoding. Therefore, if the processing resolution of the first visual encoder is larger than the processing resolution of other visual coding rates, the size of the segmented sub-images can be larger when the first image is segmented into multiple sub-images, thereby reducing the number of segmentations of the first image and saving power consumption of the electronic device.

[0122] In summary, before encoding visual data, a first visual encoder can be selected from multiple visual encoders based on one or more of the aforementioned first conditions. This can reduce power consumption when the electronic device runs the first visual encoder to encode the visual data or improve the accuracy of visual data inference results. Specifically, the process by which the electronic device selects the first visual encoder from multiple visual encoders can be found in the following text. Figure 6 The details will not be elaborated here.

[0123] In some embodiments, the operation of selecting the first visual encoder described above can be implemented during the design phase of the MLLM. That is, after the first visual encoder is selected and the MLLM is designed, the MLLM with the first visual encoder module can be directly deployed on multiple clients. When the clients run the MLLM to perform inference on visual data, they can directly use the first visual encoder in the MLLM to encode the visual data without performing the operation of selecting the first visual encoder described above.

[0124] In some embodiments, during the process of encoding multiple first sub-images by a first visual encoder, the electronic device can divide each first sub-image into multiple patches based on the dynamic patch size determined by the above formula (1). Then, the electronic device can encode the divided multiple patches to obtain the visual token corresponding to each first sub-image through linear transformation or convolution operations in the first visual encoder. That is, the first visual encoder can map the image features in the patch into a fixed-dimensional vector (i.e., a visual token) through linear transformation or convolution operations. In this way, when each first sub-image is divided into multiple patches based on a larger dynamic patch size, the first visual encoder can divide the received first sub-image into fewer patches, thereby reducing the number of visual tokens when mapping each patch to a fixed-dimensional visual token. Consequently, the electronic device can generate less power consumption when performing inference on visual data based on visual tokens.

[0125] In some embodiments, after the electronic device encodes multiple first sub-images using a first visual encoder to obtain a visual token corresponding to each first sub-image, if there are overlapping regions among the first sub-images, the visual token of the first sub-image may also include the visual token corresponding to the overlapping region and the visual token corresponding to the non-overlapping region. For example, in the above... Figure 3D In the first sub-image A1 and the first sub-image A2, there is an overlapping area. After the first visual encoder encodes the first sub-image A1 and the first sub-image A2 respectively to obtain the visual token of each first sub-image, it will also obtain the corresponding overlapping area of ​​the repeated visual token.

[0126] Therefore, in this embodiment, to save power consumption, the electronic device can also delete duplicate visual tokens, retaining only one of the multiple identical visual tokens. Specifically, when n (n is a positive integer greater than 1) first sub-images include the same first overlapping region, the electronic device can save the visual token corresponding to the first overlapping region in one of the n first sub-images. Furthermore, the electronic device can delete the visual tokens corresponding to the first overlapping regions in the other n-1 first sub-images. For example, in the above... Figure 3D In the image, first sub-images A1, A2, A4, and A5 share the same first overlapping region A11. Therefore, the electronic device can delete the visual token corresponding to the first overlapping region A11 in three of the first sub-images, retaining only the visual token corresponding to the first overlapping region A11 in one of the first sub-images.

[0127] Then, the electronic device can obtain multiple first visual tokens based on the stored visual tokens corresponding to each overlapping region and the visual tokens corresponding to the non-overlapping regions in each first sub-image.

[0128] Thus, by removing duplicate visual tokens, data redundancy can be reduced. This, in turn, reduces the computational and storage requirements for inference based on visual tokens, improves the inference efficiency of electronic devices using MLLM to infer visual data, and reduces the power consumption of electronic devices running MLLM.

[0129] In some embodiments, if there are no overlapping areas between the first sub-images, the electronic device does not need to delete the overlapping visual tokens, and can directly use the visual tokens of the first sub-images output by the first visual encoder as multiple first visual tokens.

[0130] S24: Compress multiple first visual tokens based on a preset compression rate to obtain multiple compressed second visual tokens.

[0131] In some embodiments, the number of compressed second visual tokens is less than the number of uncompressed first visual tokens.

[0132] In some embodiments, when the visual data is a video comprising multiple first images, the corresponding preset compression ratio can be a first compression ratio. When the visual data is an image set comprising multiple first images, the corresponding preset compression ratio can be a second compression ratio. When the visual data is a single first image, the corresponding preset compression ratio can be a third compression ratio. The first compression ratio is greater than the second compression ratio, and the second compression ratio is greater than the third compression ratio. For example, the first compression ratio can be 8, the second compression ratio can be 4, and the third compression ratio can be 2. Thus, when the first compression ratio is higher, the number of second visual tokens in a single first image in the video will be less, and therefore the number of second visual tokens in multiple first images in the video will also be less. Similarly, when the second compression ratio is greater than the third compression ratio, the number of second visual tokens in a single first image in the image set will be less, and therefore the number of second visual tokens in multiple first images in the video will also be less.

[0133] The process of compressing multiple first-view tokens is described below.

[0134] In some embodiments, when compressing multiple first visual tokens, the electronic device can first divide the multiple first visual tokens into one or more groups (e.g., two or more groups). Each group of first visual tokens can include k×k adjacent first visual tokens, where k is a preset compression rate and k is a positive integer greater than 1. If each first visual token includes r (where r is a positive integer) dimensional data, then each group of first visual tokens (i.e., k×k first visual tokens) also includes k×k×r dimensional data. Then, the electronic device can compress the k×k×r dimensional data corresponding to each group of first visual tokens into 1×r dimensional data, that is, compress k×k first visual tokens (total dimension k×k×r) into one second visual token (dimension r). This ensures that the number of second visual tokens obtained is less than the number of first visual tokens.

[0135] For example, the following uses a first compression ratio of 8, a second compression ratio of 4, and a third compression ratio of 2 as examples, combined with... Figure 4 As shown, the process of compressing multiple first-visual tokens by an electronic device is described.

[0136] like Figure 4 As shown, after the electronic device obtains multiple first visual tokens corresponding to visual data 401, the electronic device can first perform modal judgment on visual data 401 to determine whether the visual data is a video, a set of images, or a single first image (also known as a single image).

[0137] For example, when the electronic device determines that visual data 401 is video, it can group multiple first visual tokens based on the first compression ratio 8 corresponding to the video, thus dividing the multiple first visual tokens into P1 groups (P1 being a positive integer). Each group of first visual tokens in P1 can include 8×8 adjacent first visual tokens. If each first visual token includes 1024 dimensions of data, then each group of first visual tokens in P1 can include 8×8×1024 (i.e., 64×1024) dimensions of data. At this point, the electronic device can compress the 64×1024 dimensions of data in each group of first visual tokens into 1×1024 dimensions of data. That is, the electronic device can compress the 8×8 first visual tokens (total dimension 64×1024) in each group of first visual tokens into 1×1 second visual tokens (dimension 1×1024), thus obtaining P1 second visual tokens. Thus, compared to the 8×8 first visual tokens in group P1 before compression, there are only P1 second visual tokens after compression, which shows that the number of second visual tokens is greatly reduced after compression.

[0138] For example, when the electronic device determines that visual data 401 is an image set, it can group multiple first visual tokens based on the second compression ratio 4 corresponding to the image set, thus dividing the multiple first visual tokens into P2 groups (P2 being a positive integer). Each group of first visual tokens in P2 groups can include 4×4 adjacent first visual tokens. If each first visual token includes 1024 dimensions of data, then each group of first visual tokens in P2 groups can include 4×4×1024 (i.e., 16×1024) dimensions of data. At this point, the electronic device can compress the 16×1024 dimensions of data in each group of first visual tokens into 1×1024 dimensions. That is, the electronic device can compress the 4×4 first visual tokens (total dimension 16×1024) in each group of first visual tokens into 1×1 second visual tokens (dimension 1×1024), thus obtaining P2 second visual tokens. Thus, compared to the 4×4 first visual tokens in group P2 before compression, there are only P2 second visual tokens after compression, which shows that the number of second visual tokens is greatly reduced after compression.

[0139] For example, when the electronic device determines that visual data 401 is a single first image, it can group multiple first visual tokens based on the third compression ratio 2 corresponding to the single first image, thus dividing the multiple first visual tokens into P3 groups (P3 being a positive integer). Each group of first visual tokens in P3 groups can include 2×2 adjacent first visual tokens. If each first visual token includes 1024 dimensions of data, then each group of first visual tokens in P3 groups can include 2×2×1024 (i.e., 4×1024) dimensions of data. At this point, the electronic device can compress the 4×1024 dimensions of data in each group of first visual tokens into 1×1024 dimensions. That is, the electronic device can compress the 2×2 first visual tokens (total dimension 4×1024) in each group of first visual tokens into 1×1 second visual tokens (dimension 1×1024), thus obtaining P3 second visual tokens. Thus, compared to the 2×2 first visual tokens in the P3 group before compression, there are only P3 second visual tokens after compression, which shows that the number of second visual tokens is greatly reduced after compression.

[0140] In summary, by compressing multiple first visual tokens, the number of compressed second visual tokens can be reduced. This, in turn, reduces the computational and storage requirements for inference based on visual tokens, improves the inference efficiency of electronic devices using MLLM for visual data reasoning, and lowers the power consumption of electronic devices running MLLM.

[0141] In some embodiments, electronic devices can compress k×k×r-dimensional data into 1×r-dimensional data by deleting redundant data (such as deleting duplicate texture data), fusing similar data (such as clustering similar information), deleting data that has little impact on the inference result (such as deleting image backgrounds), and taking values ​​based on a preset step size interval (such as retaining data of 1 dimension every 2 dimensions of data). This application does not limit the scope of this method. In this way, the compressed second visual token can still retain data that has a significant impact on the accuracy of the inference result, thereby ensuring the accuracy of the inference result.

[0142] S25: Based on multiple second visual tokens, obtain the inference results corresponding to the visual data through MLLM.

[0143] In some embodiments, after acquiring multiple second visual tokens, the electronic device can use MLLM to infer the second visual tokens and obtain the inference result. For example, the electronic device can use a large language neural network submodule in MLLM (such as the lightweight large language model Qwen3.06B) to infer the second visual tokens and obtain the inference result.

[0144] In other embodiments, the MLLM may include multiple sub-modules, such as a first visual encoder module, an image segmentation module, and a large language neural network sub-module. Therefore, the methods shown in S21 to S25 above can all be executed by the MLLM, and this application does not limit this.

[0145] Thus, the above method can be used to perform inference on visual data. Furthermore, while ensuring the accuracy of the inference results, it can also improve inference efficiency and reduce the power consumption of electronic devices.

[0146] Furthermore, in some embodiments, in addition to inferring from visual data, embodiments of this application can also infer from received visual data and multiple modalities such as text data. For example, an electronic device can map a second visual token in visual space to a third visual token in text space. Then, the electronic device can obtain the inference results corresponding to the visual data and text data through MLLM based on the third visual token and the first text token corresponding to the text data. See details for further information. Figure 5 As stated above.

[0147] For example, such as Figure 5 As shown, after acquiring the first image 501, the electronic device can split the first image 501 into multiple first sub-images based on the splitting sub-module. For example, the electronic device can adjust the size of the first image 501 to obtain a second image 502. Then, the electronic device can determine the overlap size based on the size of the second image 502 and the processing resolution of the first visual encoder, and segment the second image 502 into multiple first sub-images based on the overlap size and the processing resolution. Next, the electronic device can input the multiple first sub-images into the first visual encoder for encoding to obtain multiple first visual tokens. Then, the electronic device can compress the multiple first visual tokens into multiple second visual tokens using the token compression sub-module and the method shown in S24 above. Next, the electronic device can map the second visual tokens in visual space to third visual tokens in text space using the multilayer perceptron (MLP) projection sub-module.

[0148] Furthermore, after acquiring text data, the electronic device can input the text data into a text segmenter to obtain the first text token corresponding to the text data. Then, the electronic device can input the first text token output by the text segmenter and the third visual token output by the MLP projection submodule into the large language neural network submodule in MLLM (such as the lightweight large language model Qwen3.06B) to obtain the inference results corresponding to the visual data and text data.

[0149] Thus, the above method enables inference on multimodal data such as visual and textual data. Furthermore, while ensuring the accuracy of the inference results, it can also improve inference efficiency and reduce the power consumption of electronic devices.

[0150] In some embodiments, an electronic device may typically include multiple visual encoders; however, in order to reduce power consumption when the electronic device runs a first visual encoder to encode visual data or to improve the inference accuracy of the electronic device for visual data, the first visual encoder may be a lightweight visual encoder among multiple visual encoders.

[0151] The following is combined with Figure 6 The flowchart shown illustrates the process of selecting a first visual encoder from multiple visual encoders as described in this application. This method can be applied to electronic devices, such as any electronic device like the computer mentioned above. Figure 6 As shown, specifically, the method is as follows:

[0152] S601: Obtain one or more of the processing resolution, scale, or base patch size of each of the multiple visual encoders.

[0153] In some embodiments, processing resolution, scale, and patch size are inherent parameters of the visual encoder. Processing resolution can refer to the size of the image that the visual encoder can receive; scale can refer to the computational cost or number of parameters of the visual encoder, or it can refer to the amount of memory space occupied by the visual encoder in the electronic device; the basic patch size refers to the size of the patches into which the visual encoder divides the received image. For example, the electronic device may have three visual encoders as shown in Table 1.

[0154] Table 1:

[0155]

[0156] As shown in Table 1, the electronic device can include three types of SigLIP-2-base visual encoders. The electronic device can obtain the processing resolution, basic patch size, and scale of each of the three visual encoders. For example, the first type of SigLIP-2-base visual encoder has a processing resolution of 224, a basic patch size of 16, and requires 93MB of memory space from the electronic device. Another example is the second type of SigLIP-2-base visual encoder, which has a processing resolution of 256, a basic patch size of 16, and requires 93MB of memory space from the electronic device. Yet another example is the third type of SigLIP-2-base visual encoder, which has a processing resolution of 256, a basic patch size of 32, and requires 95MB of memory space from the electronic device.

[0157] S602: Arrange each visual encoder in descending order according to its processing resolution, and assign a first number to each visual encoder in ascending order after the descending order.

[0158] In some embodiments, if there are n (n is a positive integer) visual encoders with different processing resolutions, the n visual encoders can be arranged in descending order of processing resolution, and each visual encoder can be assigned a first number 1, 2, 3...n-2, n-1, n in ascending order of sorting number.

[0159] In some embodiments, after arranging multiple visual encoders in descending order of processing resolution, if multiple visual encoders have the same processing resolution, the same number can be assigned to visual encoders with the same processing resolution. For example, as shown in Table 1 above, after arranging the three encoders in descending order of processing resolution, the visual encoders arranged in ascending order of their sorting numbers (e.g., the second type SigLIP-2-base, the third type SigLIP-2-base, and the first type SigLIP-2-base) can be assigned a first number (e.g., 1, 1, 2) sequentially. That is, the first number of the second type SigLIP-2-base visual encoder is 1, the first number of the third type SigLIP-2-base visual encoder is 1, and the first number of the third type SigLIP-2-base visual encoder is 2. For another example, the electronic device may also include four visual encoders. Furthermore, after arranging the four visual encoders in descending order of processing resolution, if the second and third visual encoders have the same processing resolution, the electronic device can also set the first code 1, 2, 2, 3 to the four visual encoders in ascending order of sorting number.

[0160] S603: Arrange each visual encoder in ascending order according to its size, and assign a second number to each visual encoder in ascending order.

[0161] In some embodiments, if there are n (n is a positive integer) visual encoders of different sizes, the n visual encoders can be arranged in ascending order of size from smallest to largest, and the second numbers 1, 2, 3...n-2, n-1, n can be set in sequence.

[0162] In some embodiments, after arranging multiple visual encoders in ascending order of size, if multiple visual encoders have the same size, the same number can be assigned to visual encoders with the same size. For example, as shown in Table 1 above, after arranging the three encoders in ascending order of size, the second number (e.g., the first SigLIP-2-base, the second SigLIP-2-base, and the third SigLIP-2-base) can be assigned sequentially in ascending order of their sorting numbers (e.g., the first SigLIP-2-base, the second SigLIP-2-base, and the third SigLIP-2-base). That is, the second number of the first SigLIP-2-base visual encoder is 1, the second number of the second SigLIP-2-base visual encoder is 1, and the first number of the third SigLIP-2-base visual encoder is 2. For another example, the electronic device may also include four visual encoders. Furthermore, after arranging the four visual encoders in ascending order of size, if the first and second visual encoders have the same size, the electronic device can assign the second code 1, 1, 2, and 3 to the four visual encoders sequentially.

[0163] S604: Arrange each vision encoder in ascending order according to the basic patch size of each vision encoder, and set a third number for each vision encoder after ascending order in ascending order.

[0164] In some embodiments, if there are n (n is a positive integer) visual encoders with different base patch sizes, the n visual encoders can be arranged in ascending order according to their base patch sizes, and the third numbers 1, 2, 3...n-2, n-1, n can be set sequentially.

[0165] In some embodiments, after arranging multiple visual encoders in ascending order of their basic patch size, if multiple visual encoders have the same basic patch size, the same number can be assigned to the visual encoders with the same basic patch size. For example, the three encoders shown in Table 1 above, after arranging them in ascending order of their basic patch size, can be assigned a third number (e.g., 1, 1, 2) sequentially according to their ascending order of sorting number (e.g., the first type SigLIP-2-base, the second type SigLIP-2-base, and the third type SigLIP-2-base). That is, the second number of the first type SigLIP-2-base visual encoder is 1, the second number of the second type SigLIP-2-base visual encoder is 1, and the first number of the third type SigLIP-2-base visual encoder is 2. For another example, the electronic device may also include four visual encoders. Furthermore, after arranging the four visual encoders in ascending order of their base patch size, if the base patch size of the third and fourth visual encoders is the same, the electronic device can sequentially set the third code 1, 2, 3, 3 for the four visual encoders.

[0166] It is understood that in the embodiments of this application, the electronic device may execute one or more of S602 to S604 described above. For example, the electronic device may execute S602 to S604; or, for example, the electronic device may execute only S602, without executing S603 and S604. This application does not limit this.

[0167] It is also understood that when the electronic device executes two or three of S602 to S604 above, this application does not limit the execution order. For example, when the electronic device executes S602 and S603, it may execute S602 first and then S603, or it may execute S603 first and then S602, or it may execute S602 and S603 simultaneously. This application does not limit this.

[0168] S605: Based on one or more of the first number, second number, or third number corresponding to each visual encoder, obtain the total number corresponding to each visual encoder.

[0169] For example, the sum of the numbers of each visual encoder obtained above can be used as the total number of each visual encoder. For instance, the first number, the second number, and the third number can be added together sequentially to obtain the total number of the corresponding visual encoder. Alternatively, the electronic device can also perform a weighted summation of the numbers of each visual encoder obtained above to obtain the total number of the corresponding visual encoder. This application does not limit this approach.

[0170] It is understood that in the embodiments of this application, the electronic device can execute any one or more of S602 to S604 described above. Therefore, when determining the total number of each visual encoder, the total number can also be obtained based on one or more of the first number, the second number, or the third number. For example, if the electronic device only executes S602 and S603 to obtain the first number and the second number, but does not execute S604, then the electronic device can obtain the total number of the corresponding visual encoder based solely on the first number and the second number. This application does not limit this.

[0171] S606: Among the total numbers corresponding to each vision encoder, select the vision encoder with the smallest total number as the first vision encoder.

[0172] For example, if the first number, second number, and third number of the three types of visual encoders shown in Table 1 are determined by S602 to S603 respectively, and the total number is determined by S605 based on the first number, second number, and third number of each visual encoder, then the electronic device can use the visual encoder corresponding to the smallest total number (e.g., the second type of SigLIP-2-base visual encoder SigLIP-2-base-93M-256-16) as the first visual encoder.

[0173] In some embodiments, the smallest total number indicates a larger processing resolution, smaller scale, or smaller patch size. A larger processing resolution of the visual encoder means that when the first image is segmented into multiple sub-images, the size of the segmented sub-images can be larger, thereby reducing the number of segmentations of the first image and saving power consumption of the electronic device. A smaller scale of the visual encoder means lower power consumption when the electronic device runs the visual encoder. A smaller patch size of the visual encoder means that the image received by the first visual encoder can be divided into more patches, allowing the first visual encoder to output more visual tokens to ensure the integrity of image feature encoding, thereby improving the accuracy of the results of visual data inference by the electronic device.

[0174] Therefore, by selecting the visual encoder corresponding to the smallest total number as the first visual encoder, the power consumption of the electronic device when running the first visual encoder to encode visual data can be reduced or the accuracy of visual data inference results can be improved.

[0175] In some embodiments, if at least two visual encoders among the visual encoders have a total number corresponding to the minimum total number, the electronic device may also select the visual encoder that satisfies the first condition shown in S22 the most times as the first visual encoder. For example, if the total numbers corresponding to visual encoder A and visual encoder B are both the minimum total number, visual encoder A satisfies two of the first conditions shown in S22, and visual encoder B satisfies one of the first conditions shown in S22, then the electronic device may select visual encoder A, which satisfies the first condition the most times, as the first visual encoder.

[0176] In some embodiments, the operation of selecting the first visual encoder described above can be implemented during the design phase of the MLLM. That is, after the first visual encoder is selected and the MLLM is designed, the MLLM with the first visual encoder module can be directly deployed on multiple clients. When the clients run the MLLM to perform inference on visual data, they can directly use the first visual encoder in the MLLM to encode the visual data without performing the operations shown in S601 to S606 above to select the first visual encoder.

[0177] Thus, using the method described above, a first visual encoder can be determined for encoding the visual data. Furthermore, the electronic device can use this first visual encoder to encode the visual data to obtain a first visual token corresponding to the visual data.

[0178] In summary, in this application, the electronic device can use a lightweight first visual encoder to encode visual data, thereby saving power consumption during the encoding process. Furthermore, in this embodiment, the first visual token corresponding to the visual data can be compressed based on a preset compression rate, resulting in a smaller number of second visual tokens. This reduces the computational process and storage requirements for inference based on visual tokens, improves the inference efficiency of the electronic device using MLLM for visual data inference, and reduces the power consumption of the electronic device running MLLM. Moreover, the compressed second visual tokens can retain data that significantly impacts the accuracy of the inference results, thus ensuring the accuracy of the inference results.

[0179] The following is combined with Figure 7A and Figure 7B The advantages of MLLM using the data processing method of this application are introduced.

[0180] For example, such as Figure 7AAs shown, compared to the multimodal large language model InternVL2.5-2B, the multimodal large language model MagicVL-2B using the data processing method of this application generates lower power consumption when running the visual encoder and the large language neural network submodule. Furthermore, the total power consumption of the multimodal large language model MagicVL-2B using the data processing method of this application is also lower.

[0181] For example, such as Figure 7B As shown, various MLLMs can be tested using a variety of existing large datasets. For example, MLLMs can be tested using datasets such as MMBeencV1.1(en), RealWorldQA, HallusionBench5, SEED-2Plus, AI2D, OCRBench, DocVQA, CRPE, and MMStar5. When testing on the same dataset, MLLMs closer to the outer edge of the dataset generally perform better on that dataset. (Reference) Figure 7B As can be seen, compared with multimodal large language models such as Phi-3.5-Vision-4B, InternVL2.5-2B, SmolVLM2, Qwen2-Vl-2B and MiniCPM-V2, the multimodal large language model Magic-VL (Qwen3-1.7B) using the data processing method of this application has better reasoning performance (such as reasoning efficiency or reasoning accuracy) on various datasets.

[0182] In summary, the MLLM using the data processing method of this application can generate lower power consumption and achieve better inference performance, such as higher inference efficiency or accuracy, when performing inference on visual data.

[0183] Furthermore, in some embodiments, this application also provides a readable storage medium. The readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the data processing method mentioned in this application.

[0184] Furthermore, in other embodiments, this application also provides a computer program product, wherein the computer program product includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device can implement the data processing method mentioned in this application.

[0185] Furthermore, in some other embodiments, this application also provides a chip including a processor. The processor is used to read and execute computer code / instructions stored in memory to perform the data processing methods mentioned in this application.

[0186] Furthermore, in other embodiments, this application also provides an electronic device. The electronic device includes at least one memory and at least one processor, with the memory coupled to the processor. The memory stores computer program code / instructions. When the computer program code / instructions are executed by the processor, the electronic device can implement the data processing methods mentioned in this application.

[0187] like Figure 8 The diagram illustrates the hardware structure of an electronic device 100 according to an embodiment of this application. Figure 8 As shown, the electronic device 100 may include one or more processors 102, a system control logic unit 101 connected to at least one of the processors 102, a system memory 105 connected to the system control logic unit 101, a memory 103 connected to the system control logic unit 101, and a network interface 107 connected to the system control logic unit 101.

[0188] It is understood that the structures illustrated in the embodiments of this application do not constitute a limitation on the only possible implementation of the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0189] Processor 102 may include one or more single-core or multi-core processors. In some embodiments, processor 102 may include any combination of general-purpose processors and special-purpose processors (e.g., application processors, baseband processors, etc.). It is understood that in this embodiment, processor 102 may be configured to execute executable instructions 104 stored in memory 103 to implement the data processing method of this embodiment. When at least one instruction is executed in processor 102, electronic device 100 implements the data processing method of this embodiment.

[0190] System control logic unit 101 may include any suitable interface controller to provide any suitable interface to at least one of the processors 102 and / or any suitable device or component communicating with system control logic unit 101. System control logic unit 101 may include one or more memory controllers to provide an interface to system memory 105. System memory 105 may be used to load and store data and / or instructions. In some embodiments, system memory 105 of electronic device 100 may include any suitable volatile memory, such as suitable dynamic random access memory.

[0191] Memory 103 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, memory 103 may include any suitable volatile memory and / or any suitable non-volatile storage device, such as memory 103 may include random access memory (RAM) and / or cache memory cells, and may further include read-only memory (ROM).

[0192] The memory 103 may include a portion of the storage resources on the device on which the electronic device 100 is installed, or it may be accessible by the device, but is not necessarily part of the device. For example, the memory 103 may be accessed over a network via the network interface 107.

[0193] Specifically, system memory 105 and memory 103 may each include a temporary copy and a permanent copy of instruction 104. Instruction 104 may include, when executed by at least one of processors 102, causing electronic device 100 to implement the data processing method of the embodiments of this application. In some embodiments, instruction 104, hardware, firmware and / or its software components may additionally / alternatively be located in system control logic unit 101, network interface 107 and / or processor 102.

[0194] Network interface 107 may include a transceiver for providing a radio interface to electronic device 100, thereby enabling communication with any other suitable device (such as a front-end module, antenna, etc.) via one or more networks. In some embodiments, network interface 107 may be integrated into other components of electronic device 100. For example, network interface 107 may be integrated into at least one of processor 102, system memory 105, memory 103, and firmware device (not shown) with instructions.

[0195] Network interface 107 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, network interface 107 may be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.

[0196] The electronic device 100 may further include an input / output (I / O) device 106. The input / output device 106 enables a user to interact with the electronic device 100.

[0197] In some embodiments, the electronic device 100 may further include, but is not limited to, a display (e.g., a liquid crystal display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., a light-emitting diode flash) and a keyboard.

[0198] In some embodiments, the electronic device 100 further includes a sensor for determining at least one of environmental conditions or location information associated with the electronic device 100.

[0199] In some embodiments, the sensor may include, but is not limited to, a gyroscope sensor, an accelerometer, a proximity sensor, an ambient light sensor, and a positioning unit. The positioning unit may also be part of or interact with the network interface 107 to communicate with components of the positioning network, such as global positioning system (GPS) satellites.

[0200] The embodiments disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0201] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digit gate processor, a microcontroller, an application-specific integrated circuit, or a microprocessor.

[0202] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0203] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, optical discs, magneto-optical disks, ROM, RAM, magnetic cards or optical cards, or tangible machine-readable memories for transmitting information using electrical, optical, acoustic, or other forms of propagation signals via the Internet (e.g., carrier waves, infrared signal digit gating, etc.). Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0204] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, including structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0205] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problem proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problem proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0206] It should be noted that in the examples and description of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0207] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. A data processing method, characterized in that, This is applied to electronic devices, which include a multimodal large language model used for processing visual data; and... The data processing method includes: The visual data to be inferred is obtained, and the visual data includes one or more first images; Each of the one or more first images is split into multiple first sub-images, and the multiple first sub-images are image encoded by a first visual encoder to obtain multiple first visual feature tokens; wherein, the size of each of the multiple first sub-images matches the processing resolution of the first visual encoder, and at least two adjacent first sub-images have overlapping areas in the multiple first sub-images split from each first image; Multiple first visual tokens are compressed based on a preset compression ratio to obtain multiple compressed second visual tokens. The number of compressed second visual tokens is less than the number of first visual tokens before compression. Based on the multiple second visual tokens, the reasoning results corresponding to the visual data are obtained through the multimodal large language model.

2. The method according to claim 1, characterized in that, The step of splitting each of the one or more first images into multiple first sub-images includes: The dynamic patch size is determined based on the base block patch size of the first visual encoder and the preset compression rate; Based on the dynamic patch size, the size of at least one of the one or more first images is adjusted to obtain a corresponding second image, wherein the size of the second image is greater than or equal to the size of the corresponding first image, and the size of the second image is an integer multiple of the dynamic patch size; The overlap size is determined based on the size of the second image and the processing resolution of the first visual encoder. Then, based on the overlap size and the processing resolution of the first visual encoder, the second image is segmented into the plurality of first sub-images; wherein, The overlap dimension includes overlap height and / or overlap width; and... In the plurality of first sub-images segmented from each first image, at least two adjacent first sub-images have overlapping regions, including: In the height direction of the second image, there is an overlapping region with a height equal to the overlap height between the bottom of the i-th first sub-image and the top of the adjacent (i+1)-th first sub-image, where i is a positive integer; and / or, In the width direction of the second image, there is an overlapping region with a width equal to the overlapping width between the right side of the j-th first sub-image and the left side of the adjacent (j+1)-th first sub-image, where j is a positive integer.

3. The method according to claim 2, characterized in that, The dimensions of the second image include its height and width; The height of the second image satisfies the following formula: ； Where H2 represents the height of the second image, H1 represents the height of the first image, and N token This indicates the size of the dynamic patch. This indicates rounding to the nearest integer. The width of the second image satisfies the following formula: ； Where W2 represents the width of the second image, W1 represents the width of the first image, and N token This indicates the size of the dynamic patch. This indicates rounding to the nearest integer.

4. The method according to claim 2, characterized in that, The overlap height satisfies the following formula: ； Wherein, H3 represents the overlap height, H0 represents the height in the processing resolution, and H2 represents the height of the second image. This indicates rounding to the nearest integer. The overlap width satisfies the following formula: ； Wherein, W3 represents the overlap width, W0 represents the width in the processing resolution, and W2 represents the width of the second image. This indicates rounding to the nearest integer.

5. The method according to claim 2, characterized in that, The first visual encoder performs image encoding on the multiple first sub-images to obtain multiple first visual tokens, including: The first visual encoder encodes the multiple first sub-images to obtain the visual tokens corresponding to each first sub-image. The visual tokens of each first sub-image include visual tokens corresponding to overlapping regions and visual tokens corresponding to non-overlapping regions. In the case where n first sub-images include the same first overlapping region, save the visual token corresponding to the first overlapping region in one of the n first sub-images, and delete the visual token corresponding to the first overlapping region in the other n-1 first sub-images, where n is a positive integer greater than 1. The plurality of first visual tokens are obtained based on the visual tokens corresponding to each overlapping region and the visual tokens corresponding to the non-overlapping regions in each first sub-image.

6. The method according to claim 5, characterized in that, The step of encoding the plurality of first sub-images using the first visual encoder to obtain the visual token corresponding to each first sub-image includes: During the process of image encoding of the multiple first sub-images by the first visual encoder, each first sub-image is divided into multiple patches based on the dynamic patch size, and the multiple patches are encoded to obtain the visual token corresponding to each first sub-image.

7. The method according to claim 1, characterized in that, The visual data is a video comprising multiple first images, and the preset compression rate is a first compression rate; or... The visual data is an image set including multiple first images, and the preset compression rate is a second compression rate; or... The visual data is a single first image, and the preset compression rate is a third compression rate; wherein... The first compression ratio is greater than the second compression ratio, and the second compression ratio is greater than the third compression ratio.

8. The method according to any one of claims 1 to 7, characterized in that, The compression of the plurality of first visual tokens based on a preset compression ratio to obtain a plurality of compressed second visual tokens includes: The plurality of first visual tokens are divided into one or more groups of first visual tokens, wherein... Each of the one or more sets of first visual tokens includes k×k adjacent first visual tokens, and the k×k first visual tokens include data of k×k×r dimensions, where k is the value of the preset compression rate, r is the dimension of the first visual token, k is a positive integer greater than 1, and r is a positive integer. The k×k×r dimension data corresponding to each group of first visual tokens is compressed into 1×r dimension data to obtain the multiple second visual tokens.

9. The method according to claim 1, characterized in that, The electronic device includes a plurality of visual encoders, wherein the first visual encoder among the plurality of visual encoders satisfies one or more of the following first conditions: The size of the first visual encoder is smaller than the size of the other visual encoders among the plurality of visual encoders; The base patch size of the first visual encoder is smaller than the base patch size of the other visual encoders; The processing resolution of the first visual encoder is greater than that of the other visual encoders.

10. The method according to claim 9, characterized in that, Before encoding the plurality of first sub-images using a first visual encoder, the method further includes: Obtain one or more of the processing resolution, scale, or base patch size of each of the plurality of visual encoders; Arrange the visual encoders in descending order of their processing resolution, and assign a first number to each of the descendingly arranged visual encoders in ascending order; and / or, Arrange the visual encoders in ascending order according to their size, and assign a second number to each of the sorted visual encoders in ascending order; and / or, Arrange the visual encoders in ascending order according to their base patch size, and assign a third number to each of the sorted visual encoders in ascending order; and / or, Based on one or more of the first number, the second number, or the third number corresponding to each visual encoder, the total number corresponding to each visual encoder is obtained; Among the total numbers corresponding to each visual encoder, the visual encoder with the smallest total number is selected as the first visual encoder.

11. The method according to claim 10, characterized in that, The step of selecting the visual encoder with the smallest total number from among the total numbers corresponding to each visual encoder as the first visual encoder includes: If at least two of the visual encoders have a total number that is equal to the minimum total number, the visual encoder that satisfies the first condition the most times will be designated as the first visual encoder.

12. The method according to claim 1, characterized in that, The method further includes: Obtain the text data to be inferred; The process of obtaining the inference result corresponding to the visual data based on the multiple second visual tokens and the multimodal large language model includes: Map the second visual token in the visual space to the third visual token in the text space; Based on the third visual token and the first text token corresponding to the text data, the inference results corresponding to the visual data and the text data are obtained through the multimodal large language model.

13. The method according to claim 12, characterized in that, The process of obtaining the inference results corresponding to the visual data and the text data through the multimodal large language model based on the third visual token and the first text token corresponding to the text data includes: The first text token corresponding to the text data is obtained through a text segmenter; The third visual token and the first text token are input into the large language neural network submodule of the multimodal large language model to obtain the inference result corresponding to the visual data and the text data.

14. An electronic device, characterized in that, include: At least one memory and at least one processor, the memory being coupled to the processor; the memory being used to store computer program code / instructions; when the computer program code / instructions are executed by the processor, causing the electronic device to perform the data processing method as described in any one of claims 1 to 13.

15. A readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the data processing method as described in any one of claims 1 to 13.

16. A computer program product, characterized in that, include: Computer instructions, when executed on an electronic device, cause the electronic device to perform the data processing method as described in any one of claims 1 to 13.

Images