Multimodal data compression method, information processing method based on multimodal data, electronic device, and storage medium

Abstract

The embodiment of the application relates to the technical field of data processing, and particularly discloses a multi-modal data compression method, an information processing method based on multi-modal data, an electronic device and a storage medium. The application precisely controls the number of compressed feature units by dynamically adjusting the compression ratio and optimizing the compression parameters, which is conducive to reducing information loss and improving the reliability of data compression, thereby being conducive to improving the understanding accuracy and processing effect of a multi-modal model.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a multimodal data compression method, an information processing method based on multimodal data, an electronic device, and a storage medium. Background Technology

[0002] With the development of artificial intelligence technology, multimodal models, such as large language models, have been widely used in information processing scenarios such as text-to-text question answering, video understanding, and scene description. Multimodal models can simultaneously process multiple types of data, including text, images, audio, and video, achieving deep understanding and reasoning of complex scenes by fusing information from different modalities. However, the massive volume of multimodal data leads to problems such as high storage costs and low transmission efficiency. Therefore, adopting effective data compression strategies before or during the input of multimodal data into the model becomes a key step in improving overall system performance and resource utilization efficiency.

[0003] Currently, compression methods for multimodal data typically employ compression strategies with fixed ratios or parameters. This approach makes it difficult to control the number of feature units after compression, easily exceeding the upper limit of the model context length or resulting in over-compression, leading to information loss and reducing the model's accuracy in understanding and processing multimodal data. Summary of the Invention

[0004] In view of this, embodiments of this application provide a multimodal data compression method, an information processing method based on multimodal data, an electronic device, and a storage medium, which can improve the reliability of data compression.

[0005] In a first aspect, embodiments of this application provide a multimodal data compression method, comprising: acquiring multimodal data and determining original feature units; determining the predicted number of target feature units after compression based on a compression ratio and the number of original feature units; adjusting the compression ratio based on the predicted number and a preset number to obtain an updated compression ratio; determining an updated predicted number based on the updated compression ratio, and determining compression parameters based on the updated predicted number; and compressing the original feature units according to the compression parameters to obtain target feature units.

[0006] In some embodiments, the step of adjusting the compression ratio according to the predicted number and the preset number to obtain the updated compression ratio further includes: when the predicted number is greater than the preset number, adjusting the compression ratio and returning to the step of determining the predicted number of compressed target feature units according to the compression ratio and the number of original feature units to obtain the updated compression ratio.

[0007] In some embodiments, the step of adjusting the compression ratio according to the predicted quantity and the preset quantity to obtain the updated compression ratio further includes: when the predicted quantity is less than or equal to the preset quantity, stopping the adjustment of the compression ratio and determining the current predicted quantity as the updated predicted quantity.

[0008] In some embodiments, the types of multimodal data include image data, video data, text data, and audio data; the step of determining the predicted number of target feature units based on the compression ratio and the number of original feature units includes: determining the compression ratio corresponding to the type of multimodal data based on the type of multimodal data; for each type of multimodal data, determining the type predicted number of the corresponding target feature unit based on the number of original feature units corresponding to the type of multimodal data and the compression ratio; and determining the predicted number based on the type predicted number.

[0009] In some embodiments, the original feature unit includes an image original feature unit or a video original feature unit; the step of determining the original feature unit includes: dividing the image data or the video data into a grid according to a first grid parameter to obtain image blocks, wherein the first grid parameter includes the number of image blocks along the length direction and the number of image blocks along the width direction; performing feature transformation processing on the first image blocks to obtain image original feature units or video original feature units.

[0010] In some embodiments, the compression parameters include the pooling kernel size and the pooling step size; the step of determining the compression parameters based on the updated prediction quantity further includes: for the image data or the video data, determining the second grid parameter corresponding to the target feature unit based on the prediction quantity of the target feature unit type corresponding to the image data or the video data; and determining the pooling kernel size and the pooling step size based on the first grid parameter and the second grid parameter.

[0011] In some embodiments, the aspect ratio of the first grid parameter is the same as that of the second grid parameter.

[0012] In some embodiments, the step of compressing the original feature units according to the compression parameters to obtain the target feature units further includes: performing two-dimensional mean pooling on the image data or the video data according to the pooling kernel size and the pooling step size to obtain the target feature units.

[0013] In some embodiments, the step of adjusting the compression ratio includes: adjusting the compression ratio of each type of multimodal data according to the same preset adjustment coefficient; or, adjusting the compression ratio of each type of multimodal data according to the preset adjustment coefficient corresponding to each type of multimodal data.

[0014] In some embodiments, the method further includes: splicing and fusing target feature units corresponding to each type in a preset order to obtain a multimodal target feature sequence.

[0015] Secondly, embodiments of this application provide an information processing method based on multimodal data, including: acquiring target feature units or multimodal target feature sequences based on multimodal data; inputting the target feature units or the multimodal target feature sequences into a multimodal model to generate results and obtain result information.

[0016] Thirdly, embodiments of this application provide an electronic device including at least one processor and at least one memory, wherein the at least one memory is communicatively connected to the at least one processor, and the at least one memory is used to store at least one program, wherein when the at least one program is executed by the at least one processor, the at least one processor implements the above-described method.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions, which, when executed, implement the above-described method.

[0018] The beneficial effects of the embodiments of this application are as follows: Unlike existing technologies, the embodiments of this application provide a multimodal data compression method, an information processing method based on multimodal data, an electronic device, and a storage medium. This application, by dynamically adjusting the compression ratio and optimizing compression parameters, precisely controls the number of feature units after compression, which helps reduce information loss and improve the reliability of data compression, thereby improving the understanding accuracy and processing effect of multimodal models. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application; Figure 2This is a flowchart illustrating a multimodal data compression method provided in an embodiment of this application; Figure 3 This is a flowchart illustrating a process for segmenting multimodal data to obtain original feature units, as provided in an embodiment of this application. Figure 4 yes Figure 2 A schematic diagram of a sub-process of step S20 in the multimodal data compression method shown; Figure 5 yes Figure 2 A schematic diagram of a sub-process of step S30 in the multimodal data compression method shown; Figure 6 This is a flowchart illustrating an information processing method based on multimodal data provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a multimodal data compression device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To facilitate understanding of this application, a more detailed description is provided below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. It should be noted that, unless otherwise specified, the various features in the embodiments of this application can be combined with each other, all within the scope of protection of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish between identical or similar items with essentially the same function and effect.

[0022] Please see Figure 1 , Figure 1This is a schematic diagram illustrating an application scenario provided in an embodiment of this application. Terminal device 1 establishes a communication connection with server 2 via a network. Terminal device 1 can be a touchscreen phone, smartphone, tablet computer, portable terminal device, or other terminal electronic device with a display screen. Server 2 can be a computer with data processing capabilities, a server cluster, or a cloud computing platform. The network includes, but is not limited to, one or more of the following: Local Area Network (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), 4G / 5G network, Wi-Fi, Bluetooth, and peer-to-peer (P2P) communication networks.

[0023] In this embodiment, both terminal device 1 and server 2 may include one or more processors. These processors may be central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement this embodiment; no limitation is made here. The one or more processors included in terminal device 1 may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs; no limitation is made here.

[0024] A multimodal interactive application is installed on terminal device 1. Users can input text, upload images, videos, or audio through this application. The multimodal model on terminal device 1 or server 2 processes the input data and returns the results. With the development of artificial intelligence technology, multimodal models can simultaneously process information from multiple modalities such as images, videos, audio, and text, demonstrating powerful capabilities in tasks such as text-based question answering, video understanding, and scene description.

[0025] In multimodal model processing, images or videos are converted into a sequence of visual tokens (terms / feature units) by a visual encoder, then concatenated with text tokens before being input into a large language model for inference. Tokens are the basic units of information processed by the model, and the total number of tokens accepted by the model (i.e., the context length) usually has a fixed upper limit. In video understanding scenarios, multiple frames of images are input simultaneously. Each frame generates a large number of visual tokens after processing by the visual encoder, resulting in an extremely large total number when superimposed; a single high-resolution image also generates a large number of visual tokens. An excessive number of tokens can lead to exceeding the model's context length limit, making inference impossible, and drastically increasing computational and memory consumption, significantly reducing inference speed.

[0026] To address the aforementioned issues, this application embodiment constructs an adaptive token compression mechanism that automatically calculates the number of target feature units and derives compression parameters in reverse, achieving adaptive compression without manual intervention. This mechanism accurately satisfies context length constraints, thereby improving the inference efficiency and applicability of multimodal models.

[0027] Firstly, embodiments of this application provide a multimodal data compression method; please refer to [link to relevant documentation]. Figure 2 The multimodal data compression method includes the following steps: S10: Acquire multimodal data and determine the original feature units.

[0028] Specifically, the system receives input multimodal data, including but not limited to text, image, video, and audio data. The multimodal data is then segmented to obtain raw feature units.

[0029] In some embodiments, the input multimodal data includes at least one of text data, image data, and video data, such as plain text, text + image, or text + video. All inputs are constrained to a fixed upper limit on the feature unit length, i.e., a preset number. In some examples, the preset number is [missing information]. .

[0030] In some embodiments, the multimodal data compression method further includes: classifying the acquired multimodal data to determine the type of multimodal data. Specifically, classification and identification are performed based on the modal characteristics of the input data, such as file format and data stream type.

[0031] Please refer to the following: Figure 3 , Figure 3 A flowchart illustrating the process of segmenting multimodal data to obtain original feature units, as provided in this application embodiment, includes: S11: Determine the segmentation and processing strategy based on the type of multimodal data.

[0032] Based on multimodal data, the original feature units include, but are not limited to, image original feature units, video original feature units, text original feature units, and audio original feature units.

[0033] For image data, the segmentation strategy is determined to be an image segmentation strategy. The image segmentation strategy includes: dividing the image data into grids according to a first grid parameter to obtain image blocks, where the first grid parameter includes the number of image blocks H along the length direction of the image data and the number of image blocks W along the width direction of the image data; and performing feature transformation processing on the image blocks to obtain the original image feature units.

[0034] For video data, the segmentation processing strategy is determined to be a video segmentation strategy. The video segmentation strategy includes: extracting frames from the video data according to a preset frame extraction rate to obtain multiple video frame data; for each video frame data, performing an image segmentation strategy to obtain the original feature units of the video.

[0035] For text data, the segmentation strategy is determined to be a text segmentation strategy. The text segmentation strategy includes: semantically dividing the text data to obtain the original text feature units.

[0036] In some embodiments, for audio data, the segmentation processing strategy is determined to be an audio segmentation strategy. The audio segmentation strategy includes: segmenting the audio data according to a preset time window to obtain multiple audio segments; and performing feature transformation processing on the audio segments to obtain original audio feature units.

[0037] S12: According to the segmentation processing strategy, the multimodal data is segmented to obtain the original feature units.

[0038] For image data, a visual encoder (ViT, Vision Transformer) performs block-based processing, dividing the image into several fixed-size blocks according to a first grid parameter. Each block undergoes feature transformation processing, such as linear projection, to generate a corresponding original image feature unit. In some examples, the first grid parameter corresponding to the spatial resolution of the image includes the number of image blocks H along the length direction and the number of image blocks W along the width direction. Therefore, the number of original image feature units is: .

[0039] For video data, frames are extracted from the video at a preset frame rate to obtain F frames. The image segmentation strategy described above is then applied to each frame, generating F frames per frame. The number of image feature units is 1, and the sequence of image feature units from each frame is concatenated along the time dimension. The original number of feature units in the video is: .

[0040] For text data, semantic segmentation is performed to obtain original text feature units. The number of original video feature units is: .

[0041] For audio data, the number of audio segments A is determined based on the total duration of the audio data and the duration of a preset time window. Acoustic features are extracted from each audio segment, for example, by converting the audio segment into spectral features through short-time Fourier transform, or by extracting high-dimensional audio feature vectors through a pre-trained audio encoder. Each audio segment corresponds to a raw audio feature unit.

[0042] After the segmentation process is completed, the number of original feature units corresponding to each type of multimodal data is counted and recorded.

[0043] S20: Determine the predicted number of target feature units after compression based on the compression ratio and the number of original feature units.

[0044] Specifically, please refer to the following: Figure 4 , Figure 4 for Figure 2 The diagram illustrates a sub-process of step S20 in the multimodal data compression method shown. Based on the compression ratio and the number of original feature units, the predicted number of target feature units is determined, including: Step S21: Determine the compression ratio corresponding to the type of multimodal data based on the type of multimodal data.

[0045] Based on the type of multimodal data, the corresponding compression ratio is read from the preset configuration parameters. In some examples, the preset configuration parameters are pre-stored in the configuration of terminal device 1 or server 3, and are set separately for different modal data types. The compression ratio refers to the ratio of the number of target feature units after compression to the number of original feature units.

[0046] Single images have dense detail information and relatively low redundancy; excessive compression can easily lead to the loss of key semantics and detailed features. Videos, on the other hand, contain a large amount of inter-frame similarity information and temporal redundancy, allowing them to withstand higher compression levels. Therefore, in some embodiments, the compression ratio of video data is less than or equal to that of image data. By configuring differentiated compression ratios for videos and images, it is possible to effectively reduce the number of video features, lowering the risk of the total feature units exceeding the model context length limit after multimodal fusion, while also improving the preservation of image detail features. The formula is expressed as: ,in, , , The compression ratio of the image data. This represents the compression ratio of the video data. In some examples, , .

[0047] In some embodiments, the compression ratio of the text data is less than or equal to 1.

[0048] Step S22: For each type of multimodal data, determine the type prediction number of the corresponding target feature unit based on the number of original feature units and the compression ratio corresponding to the type of multimodal data.

[0049] In some embodiments, the number of target feature units predicted for image data is: The number of target feature units predicted for video data is [number]. ： .

[0050] In some embodiments, the compression ratio of the text data is 1, and the number of predicted types of the target feature units of the text data is: .

[0051] Step S23: Determine the predicted quantity based on the type. .

[0052] In some embodiments, That is, the number of target feature units predicted for multimodal data is equal to the sum of the number of target feature type predictions for each type of data.

[0053] Step S30: Adjust the compression ratio according to the predicted quantity and the preset quantity to obtain the updated compression ratio.

[0054] Specifically, the updated compression ratio is obtained by adjusting the compression ratio based on the predicted quantity and the preset quantity, and further includes: Step S31: If the predicted quantity is greater than the preset quantity, adjust the compression ratio and return to step S20 to obtain the updated compression ratio.

[0055] Specifically, an execution context length constraint check is performed, comparing the predicted number with a preset number. For example, the difference between the predicted number and the preset number is calculated, and it is determined whether the difference is greater than zero. When the difference between the predicted number and the preset number is greater than zero, i.e., the predicted number is greater than the preset number, the current compression ratio is determined to be too large. The compression ratios for each modality are then adjusted downwards overall or differentiated to improve the compression degree of multimodal features. After the compression ratio adjustment is completed, the process returns to step S20, iteratively calculating the predicted number of target feature units for each type of data until the predicted number of target feature units meets the context length constraint condition, thus obtaining the final updated compression ratio.

[0056] In some embodiments, the step of adjusting the compression ratio includes: adjusting the compression ratio of each type of multimodal data according to the same preset adjustment coefficient. A globally unified preset adjustment coefficient is used to synchronously and proportionally scale the compression ratios of image data, video data, etc., uniformly reducing the feature retention ratio of each type of data and synchronously increasing the compression intensity, so that the compression amplitude of different types of multimodal data remains consistent. In some examples, this global preset adjustment coefficient is a fixed coefficient, and each adjustment multiplies the current compression ratio by the preset adjustment coefficient.

[0057] In other embodiments, the step of adjusting the compression ratio includes: adjusting the compression ratio of each type of multimodal data according to preset adjustment coefficients corresponding to each type of multimodal data. For different modalities such as image data and video data, independent adjustment coefficients are configured, and the compression ratios are adjusted differently based on the redundancy and information importance of each type of data. For example, given the high spatiotemporal redundancy of video data, a larger compression adjustment is applied, while a relatively gentle adjustment is used for detail-sensitive image data, achieving precise compression control across different modalities.

[0058] In some embodiments, the compression ratio of video data is determined by the extraction frame rate and the single-frame compression ratio. Adjusting the compression ratio of video data can adjust at least one of the extraction frame rate and the single-frame compression ratio, and then return to the steps of "determining the original feature units" and S20, or return to step S20 to redetermine the predicted number of target feature units after compression. In some examples, the extraction frame rate is kept constant, and only the single-frame compression ratio is adjusted, then return to step S20. In other examples, the single-frame compression ratio is kept constant, and only the extraction frame rate is adjusted, then return to the steps of "determining the original feature units" and S20. In still other embodiments, both the single-frame compression ratio and the extraction frame rate are adjusted, then return to the steps of "determining the original feature units" and S20. Through different adjustment strategies, video token compression is made independently controllable in the temporal and spatial dimensions, and the compression path can be flexibly selected according to the temporal redundancy and spatial detail importance of the video content, which is beneficial for adapting to different application scenarios.

[0059] Specifically, when the difference between the predicted quantity and the preset quantity is less than or equal to zero, that is, when the predicted quantity is less than or equal to the preset quantity, it indicates that the predicted quantity meets the context length constraint condition. In this case, the compression ratio is stopped, and the current predicted quantity is determined as the updated predicted quantity.

[0060] In some embodiments, when the number of predictions corresponding to the compression ratio read from the preset configuration parameters is less than or equal to the preset number, it indicates that the initial compression configuration has met the context length constraint, and there is no need to trigger the iterative adjustment mechanism. At this time, the compression ratio in the preset configuration parameters is directly determined as the updated compression ratio, and the number of predictions for each type of target feature unit is determined as the updated number of predictions. Then, the process directly proceeds to step S40 to perform compression parameter derivation and subsequent compression processing.

[0061] In other embodiments, when the number of predictions corresponding to the compression ratio read from the preset configuration parameters is greater than the preset number, step S31 is entered to iteratively adjust the compression ratio until the stopping condition of step S32 is met.

[0062] Step S40: Determine the updated prediction quantity based on the updated compression ratio, and determine the compression parameters based on the updated prediction quantity.

[0063] Specifically, compression parameters are determined based on the updated prediction quantity obtained in step S32. The compression parameters include the pooling kernel size and the pooling step size.

[0064] In some embodiments, please refer to Figure 5 , Figure 5 yes Figure 2 The diagram illustrates a sub-process of step S40 in the multimodal data compression method shown. Determining compression parameters based on the updated prediction quantity includes: Step S41: For image data or video data, determine the second grid parameters corresponding to the target feature units based on the updated type prediction number of the target feature units corresponding to the image data or video data.

[0065] Specifically, based on the updated type prediction number of the target feature units corresponding to the image data or video data, the second grid parameter is determined by integer decomposition or year-on-year decomposition. The second grid parameter includes the number of target image blocks H′ along the length direction and the number of target image blocks W′ along the width direction.

[0066] In some embodiments, integer decomposition refers to determining H′ and W′ such that H′×W′ is equal to or not greater than the updated type prediction number, and H′×W′ is the integer grid size closest to the updated type prediction number.

[0067] In some embodiments, aspect ratio decomposition means that the aspect ratios of the first grid parameter and the second grid parameter are the same. This is done to avoid introducing spatial structural deformation during compression and to maintain the aspect ratio. Under the premise of [condition], the second grid parameters corresponding to the target feature unit are determined according to the following formula.

[0068] , , in, The number of target feature units along the length direction of image data or video frame data. The number of target feature units along the length direction of the image data or video frame data. The number of updated type predictions for target feature units corresponding to image or video data, i.e. or .

[0069] Step S42: Determine the pooling kernel size and pooling step size based on the first grid parameters and the second grid parameters.

[0070] Calculate the pooling parameters from the length and width directions respectively.

[0071] Length direction: .

[0072] Width direction:

[0073] in, , These represent the pooling step size and pooling kernel size along the length direction, respectively. , These represent the pooling step size and pooling kernel size in the width direction, respectively.

[0074] In some embodiments, when H=W, meaning the number of blocks in the length and width directions of the grid is the same, the pooling parameters in the length and width directions are the same. This is symmetric pooling, which helps improve the uniformity of dimensionality reduction of image data in the length and width dimensions, making the extracted features more spatially symmetrical and facilitating subsequent understanding and processing of feature information. In some examples, the input image or video frame is a square image during symmetric pooling.

[0075] Step S50: Compress the original feature units according to the compression parameters to obtain the target feature units.

[0076] Specifically, based on the compression parameters corresponding to each type of multimodal data, differential compression processing is performed on the original feature units of different types of multimodal data.

[0077] In some embodiments, two-dimensional average pooling is performed on the image data or video frame data according to the pooling kernel size and pooling stride to obtain the target feature unit. Based on the pooling parameters obtained in step 42, a two-dimensional average pooling operation is performed on the image data or video frame data: .

[0078] Mean pooling aggregates and averages adjacent feature units in the spatial dimension, which helps to smooth local feature noise, weaken subtle texture interference, balance and integrate spatial information in local areas, and achieve spatial downsampling of visual features, outputting compressed image target feature units or video target feature units.

[0079] In some embodiments, for text data, compression processing is performed according to the corresponding compression ratio and semantic simplification rules to obtain text target feature units.

[0080] In some embodiments, the multimodal data compression method further includes: Step S60: Concatenate and fuse the target feature units corresponding to each type in a preset order to obtain a multimodal target feature sequence.

[0081] Specifically, according to the preset modal priority and splicing rules, text target feature units, image target feature units and video target feature units are sequentially spliced ​​and dimensionally fused to form a multimodal target feature sequence that meets the requirements of the model input format.

[0082] In this embodiment, an adaptive compression mechanism is constructed to dynamically adjust the compression ratio and optimize parameters for multimodal data. This ensures that the number of target feature units generated in the final model meets the model context length constraint, which is beneficial for improving the processing efficiency and reliability of the multimodal model. Simultaneously, the compression parameters are automatically derived, eliminating the need for manual modification of the model structure and hard-coded configuration, thus reducing engineering deployment and maintenance costs.

[0083] Secondly, embodiments of this application provide an information processing method based on multimodal data. Please refer to [link to relevant documentation]. Figure 6 The information processing method includes the following steps: Step S71: Obtain target feature units or multimodal target feature sequences based on multimodal data.

[0084] Specifically, the multimodal data compression method described in the above embodiments is invoked to obtain target feature units or multimodal target feature sequences that meet the model context length requirements.

[0085] Step S72: Input the target feature unit or multimodal target feature sequence into the multimodal model to generate the result information.

[0086] Specifically, the target feature units or multimodal target feature sequences are input into a pre-trained multimodal large language model. The model performs joint encoding and understanding reasoning on text semantics, image spatial features, video temporal features, etc., and performs processing such as image-text question answering, video understanding, scene description, and instruction execution. Finally, it outputs reasoning result information in text or image-text form.

[0087] In this embodiment, the total number of target feature units or multimodal target feature sequences input to the multimodal model is constrained to a preset number through an adaptive compression mechanism, enabling the multimodal model to stably complete joint encoding and inference under limited computing resources. This reduces the problem of forced input truncation or inference failure caused by exceeding the token limit, thereby improving the response integrity and output reliability of complex multimodal tasks such as text-to-image question answering and video understanding.

[0088] Thirdly, embodiments of this application provide a multimodal data compression device; please refer to [link to relevant documentation]. Figure 7The multimodal data compression device 100 includes an acquisition module 101, a prediction module 102, an adjustment module 103, a determination module 104, and a compression module 105. The acquisition module 101 acquires multimodal data and determines the original feature units. The prediction module 102 determines the predicted number of target feature units after compression based on the compression ratio and the number of original feature units. The adjustment module 103 adjusts the compression ratio based on the predicted number and a preset number to obtain an updated compression ratio. The determination module 104 determines the updated predicted number based on the updated compression ratio and determines compression parameters based on the updated predicted number. The compression module 105 compresses the original feature units according to the compression parameters to obtain the target feature units.

[0089] In some embodiments, the multimodal data compression device 100 further includes a fusion module, which is used to splice and fuse target feature units corresponding to each type in a preset order to obtain a multimodal target feature sequence.

[0090] In this embodiment, the multimodal data compression device 100 can be a software module. The software module includes several instructions, which are stored in a memory. The processor can access the memory and call the instructions to execute them, so as to complete the multimodal data compression methods of the above embodiments.

[0091] In the embodiments of this application, the multimodal data compression device 100 can also be constructed from hardware devices. For example, the multimodal data compression device 100 can be constructed from one or more chips, and the chips can work together to complete the multimodal data compression method described in the above embodiments. As another example, the multimodal data compression device 100 can also be constructed from various logic devices, such as general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, ARM (Acorn RISC Machine) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination of these components.

[0092] The multimodal data compression device 100 in this application embodiment can be a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0093] The multimodal data compression device 100 in this embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment does not specifically limit its use.

[0094] The multimodal data compression device 100 provided in this application embodiment can implement all the processes implemented in the method embodiment of this application, and will not be described again here to avoid repetition.

[0095] It should be noted that the above-described apparatus can execute the multimodal data compression method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in the apparatus embodiments can be found in the multimodal data compression method provided in the embodiments of this application.

[0096] In this embodiment, the various modules of the multimodal data compression device 100 cooperate with each other to adaptively adjust the compression parameters and precisely control the number of target feature units, which is beneficial to improving the inference efficiency and applicability of the multimodal model.

[0097] Fourthly, embodiments of this application provide an electronic device; please refer to [link to relevant documentation]. Figure 8 The electronic device 200 includes at least one processor 201 and at least one memory 202, the at least one memory 202 being used to store at least one program. When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the multimodal data compression method or the information processing method based on multimodal data in any embodiment of the method of this application.

[0098] The processor 201 and the memory 202 can be connected via a bus or other means. The processor 201 is used to execute the multimodal data compression method or the information processing method based on multimodal data in any embodiment of this application.

[0099] The memory 202, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the multimodal data compression method or the information processing method based on multimodal data in the embodiments of this invention. The processor 201 executes various functional applications and data processing of the electronic device 200 by running the non-volatile software programs, instructions, and modules stored in the memory 202, thereby implementing the multimodal data compression method or the information processing method based on multimodal data in the embodiments of this application.

[0100] Memory 202 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 202 may optionally include memory remotely located relative to processor 201. Examples of the above-described networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0101] In some embodiments, the electronic device 200 can be such as Figure 1 The terminal device 1 or server 2 shown.

[0102] The multimodal data compression method or the information processing method based on multimodal data in any embodiment of this application can be divided into one or more functional modules and stored in the memory 202. When executed by one or more processors 12, the multimodal data compression method or the information processing method based on multimodal data in any embodiment of this application is executed.

[0103] Fifthly, embodiments of this application also provide a computer-readable storage medium storing a computer program or instructions. When the computer program or instructions are executed, they implement the multimodal data compression method or the information processing method based on multimodal data in any of the above method embodiments. For example, one or more processors may execute the multimodal data compression method or the information processing method based on multimodal data in any of the above method embodiments, or execute the various steps described above.

[0104] The apparatus or device embodiments described above are merely illustrative. The unit modules described as separate components may or may not be physically separate, and the components shown as module units may or may not be physical units; that is, they may be located in one place or distributed across multiple network module units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general-purpose hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions for a computer device (which may be a personal computer, server, or network device, etc.) to execute the various embodiments or some parts of the embodiments.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; under the concept of the present invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the present invention as described above, which are not provided in detail for the sake of brevity; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal data compression method, characterized in that, Includes the following steps: Acquire multimodal data to determine the original feature units; The number of predicted target feature units after compression is determined based on the compression ratio and the number of original feature units. The compression ratio is adjusted based on the predicted quantity and the preset quantity to obtain the updated compression ratio; The updated prediction quantity is determined based on the updated compression ratio, and the compression parameters are determined based on the updated prediction quantity. The original feature unit is compressed according to the compression parameters to obtain the target feature unit.

2. The method according to claim 1, characterized in that, The step of adjusting the compression ratio based on the predicted quantity and the preset quantity to obtain the updated compression ratio further includes: When the predicted number is greater than the preset number, the compression ratio is adjusted, and the step of determining the predicted number of target feature units after compression based on the compression ratio and the number of original feature units is returned to obtain the updated compression ratio.

3. The method according to claim 2, characterized in that, The step of adjusting the compression ratio based on the predicted quantity and the preset quantity to obtain the updated compression ratio further includes: When the predicted number is less than or equal to the preset number, the compression ratio adjustment is stopped, and the current predicted number is determined as the updated predicted number.

4. The method according to claim 1, characterized in that, The types of multimodal data include image data, video data, text data, and audio data; The step of determining the predicted number of target feature units based on the compression ratio and the number of original feature units includes: Based on the type of the multimodal data, determine the compression ratio corresponding to the type of multimodal data; For each type of multimodal data, the number of predicted types of the corresponding target feature units is determined based on the number of original feature units corresponding to the type of multimodal data and the compression ratio. The predicted quantity is determined based on the predicted quantity of the type.

5. The method according to claim 4, characterized in that... , The original feature unit includes an image original feature unit or a video original feature unit; The step of determining the original feature unit includes: The image data or video data is divided into grids according to a first grid parameter to obtain image blocks. The first grid parameter includes the number of image blocks along the length direction and the number of image blocks along the width direction. The first image block is subjected to feature transformation processing to obtain the original image feature unit or the original video feature unit.

6. The method according to claim 5, characterized in that, The compression parameters include the pooling kernel size and the pooling step size; The step of determining the compression parameters based on the updated predicted quantity further includes: For the image data or the video data, the second grid parameter corresponding to the target feature unit is determined based on the updated type prediction number of the target feature unit corresponding to the image data or the video data. The pooling kernel size and pooling step size are determined based on the first grid parameters and the second grid parameters.

7. The method according to claim 6, characterized in that, The aspect ratio of the first grid parameter is the same as that of the second grid parameter.

8. The method according to claim 6, characterized in that, The step of compressing the original feature unit according to the compression parameters to obtain the target feature unit further includes: Based on the pooling kernel size and the pooling step size, the image data or the video data is subjected to two-dimensional mean pooling to obtain the target feature unit.

9. The method according to any one of claims 4-8, characterized in that, The step of adjusting the compression ratio includes: Adjust the compression ratio of each type of multimodal data according to the same preset adjustment coefficient; or... The compression ratio of the multimodal data of each type is adjusted according to the preset adjustment coefficients corresponding to each type of multimodal data.

10. The method according to any one of claims 4-8, characterized in that, The method further includes: The target feature units corresponding to each type are spliced ​​and fused in a preset order to obtain a multimodal target feature sequence.

11. An information processing method based on multimodal data, characterized in that, include: Obtain target feature units or multimodal target feature sequences based on multimodal data; The target feature unit or the multimodal target feature sequence is input into the multimodal model to generate the result information.

12. An electronic device, characterized in that, include: At least one processor; At least one memory, communicatively connected to the at least one processor, the memory being used to store at least one program, which, when executed by at least one of the at least one processor, causes the at least one processor to implement the method of any one of claims 1-11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or instructions that, when executed, implement the method as described in any one of claims 1-11.

Images