Method, apparatus, device, medium and program product for processing data

Abstract

The embodiment of the invention relates to a data processing method and device, equipment, a medium and a program product. The method includes obtaining first media content at a client, the first media content corresponding to a first modality of a plurality of modalities for a multi-modality model. The method further includes generating a first content feature for the first media content based on a first portion of the multi-modal model deployed at the client. The method further includes sending the first content feature to the server to process the first content feature and a second content feature of second media content corresponding to a second modality of the plurality of modalities using a second portion of the multi-modality model deployed on the server. The method further includes receiving, from the server, processing results for the first media content and the second media content output by the multi-modal model. Through the method, the multi-modal model is deployed on the client and the server in a mixed manner, so that the time delay is reduced, the bandwidth requirement of the service is reduced, and the calculation cost of the server is reduced.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to the field of machine learning models, and more specifically to methods, apparatuses, devices, media, and program products for processing data. Background Technology

[0002] Currently, the machine learning industry is developing at an increasingly rapid pace. More and more different types of machine learning models are emerging and being widely applied across various industries and fields. For example, vision-related models can be applied to visual inspection and autonomous driving, while language-related models can be applied to text processing and question answering. Furthermore, multimodal machine learning models, which integrate models with different focuses, can be used for more complex tasks that connect multiple different domains.

[0003] With the rapid development of the machine learning industry, new technologies related to machine learning are emerging in an endless stream, and the application scenarios of machine learning models are becoming increasingly diverse. As these application scenarios become more complex, the barrier to entry for using machine learning models is also rising. For example, complex scenarios may require the use of many different types of machine learning models, or the same machine learning model may need to be used on different nodes. In such cases, the need to meet work requirements as much as possible while conserving resources places high demands on the use of machine learning models. Therefore, there are still many aspects to be researched regarding how to apply machine learning models more efficiently. Summary of the Invention

[0004] Embodiments of this disclosure provide a method, apparatus, device, medium, and program product for processing data.

[0005] According to a first aspect of this disclosure, a method for processing data is provided. The method includes acquiring first media content at a client, the first media content corresponding to a first modality among multiple modalities of a multimodal model. The method further includes generating a first content feature for the first media content based on a first portion of the multimodal model deployed on the client. The method further includes sending the first content feature to a server to process the first content feature and a second content feature of second media content corresponding to a second modality among the multiple modalities using a second portion of the multimodal model deployed on the server, the second portion having a greater computational complexity than the first portion and a greater number of parameters than the first portion. The method further includes receiving processing results from the server for the first and second media content output by the multimodal model.

[0006] According to a second aspect of this disclosure, a method for processing data is provided. The method includes receiving, at a server, a first content feature from a client for first media content, the first media content corresponding to a first modality among multiple modalities of a multimodal model, and the first content feature being generated by processing the first media content through a first part of the multimodal model. The method further includes determining a second content feature for second media content corresponding to a second modality among the multiple modalities. The method further includes generating a processing result for the first media content and the second media content by applying the first content feature and the second content feature to a second part of the multimodal model, the second part having a greater computational cost than the first part and a greater number of parameters than the first part. The method further includes sending the processing result for the first media content and the second media content to the client.

[0007] In a third aspect of this disclosure, an apparatus for processing data is provided. The apparatus includes a first media content acquisition module configured to acquire first media content at a client, the first media content corresponding to a first modality among multiple modalities of a multimodal model; a first content feature generation module configured to generate first content features for the first media content based on a first part of a multimodal model deployed on the client; a first content feature and a second content feature processing module configured to send the first content features to a server to process the first content features and a second content feature of the second media content corresponding to a second modality among multiple modalities using a second part of a multimodal model deployed on the server, the second part having a greater computational load than the first part and the second part having a greater number of parameters than the first part; and a processing result receiving module configured to receive processing results from the server for the first media content and the second media content output by the multimodal model.

[0008] In a fourth aspect of this disclosure, an apparatus for processing data is provided. The apparatus includes a first content feature receiving module configured to receive, at a server, a first content feature from a client for first media content, the first media content corresponding to a first modality among multiple modalities of a multimodal model, and the first content feature being generated by processing the first media content through a first part of the multimodal model; a second content feature determining module configured to determine a second content feature for second media content corresponding to a second modality among the multiple modalities; a processing result generating module configured to generate a processing result for the first media content and the second media content by applying the first content feature and the second content feature to a second part of the multimodal model, the second part having a greater computational complexity than the first part and a greater number of parameters than the first part; and a processing result sending module configured to send the processing result for the first media content and the second media content to the client.

[0009] In a fifth aspect of this disclosure, an electronic device is provided, including at least one processor; and a storage device for storing at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the methods according to the first and second aspects of this disclosure.

[0010] In a sixth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods according to the first and second aspects of this disclosure.

[0011] In a seventh aspect of this disclosure, a computer program product is provided. This computer program product includes a computer program that, when executed by a processor, implements the methods according to the first and second aspects of this disclosure.

[0012] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0014] Figure 1 The illustration shows a schematic diagram of an example environment in which some embodiments of the present disclosure may be implemented;

[0015] Figure 2Examples of deploying multimodal models according to some embodiments of the present disclosure are illustrated;

[0016] Figure 3 The illustration shows a schematic diagram of an example method for processing data according to some embodiments of the present disclosure;

[0017] Figure 4 The illustration shows a schematic diagram of an example method for processing data according to some embodiments of the present disclosure;

[0018] Figure 5 The illustration shows a schematic diagram of an example of a stable diffusion model for a multimodal model according to some embodiments of the present disclosure;

[0019] Figure 6 The illustration shows a schematic diagram of an example of a multimodal model that is a bootstrap-image pre-trained model according to some embodiments of the present disclosure;

[0020] Figure 7 The illustration shows a schematic block diagram of an apparatus for processing data according to some embodiments of the present disclosure.

[0021] Figure 8 The illustration shows a schematic block diagram of an apparatus for processing data according to some embodiments of the present disclosure;

[0022] Figure 9 A schematic block diagram of an example device suitable for implementing various embodiments of the present disclosure is illustrated.

[0023] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0024] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0025] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0026] For example, upon receiving a user's proactive request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0027] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0028] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0029] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0030] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0031] With the development of pre-training and self-supervised techniques and the introduction of natural language modalities into vision tasks, many tasks have achieved breakthroughs. Among them, large-scale multimodal models based on vision and language have become a hot topic in academic research and productization, leading to the emergence of numerous generative and understanding models. Currently, many machine learning models, such as large-scale models, rely on server-side computation, with the client acting only as a capability output and service request initiator, sending data to the server. A large number of requests can cause excessive server load, and both uplink and downlink bandwidth become new challenges.

[0032] In the aforementioned traditional solutions, pure cloud services rely heavily on network conditions and server resources, resulting in high costs. Pure edge deployment, on the other hand, places excessive demands on edge hardware and presents deployment difficulties. Therefore, neither of these two individual deployment solutions effectively addresses the issue of insufficient utilization of machine learning models on the server and edge. Furthermore, network fluctuations on servers significantly reduce computational efficiency and impose a heavy computational load on servers, posing a challenge to server computing costs. Additionally, placing all computational tasks on the edge places excessively high demands on edge hardware such as graphics processing units (GPUs). Given the current lack of computing power on client devices, edge hardware costs will also increase significantly. Both of these traditional solutions place extremely high demands on hardware performance and maintenance costs, and are susceptible to network fluctuations leading to unstable computational tasks, significantly reducing the user experience when using machine learning models for computational tasks.

[0033] To address at least the aforementioned and other potential problems, embodiments of this disclosure propose a method for processing data. In this method, first media content is first obtained at the client. This first media content corresponds to a first modality among multiple modalities of a multimodal model. Next, a first content feature for the first media content is generated using a first part of the multimodal model deployed on the client. Then, the client sends the first content feature to a server to process the first content feature and a second content feature for the second media content corresponding to a second modality among multiple modalities using a second part of the multimodal model deployed on the server. The second part has a higher computational load than the first part and a higher number of parameters than the first part, allowing a large amount of model-related computation to be performed on the server side, while the processing of user-related content in different modalities can be implemented on the client side. The client can then receive the processing results for the first and second media content output by the multimodal model from the server. This method, by co-deploying the multimodal model on both the client and server, reduces latency, decreases service bandwidth requirements, lowers server computational costs, and improves processing efficiency and user experience.

[0034] The embodiments of this disclosure will now be described in further detail with reference to the accompanying drawings. Figure 1The illustration shows an example environment in which the devices and / or methods of embodiments of this disclosure can be implemented. In environment 100, client 104 can be used to process computationally intensive tasks in a multimodal model that consume relatively few computational resources, such as encoding input media content using an encoder or performing transformations with few dimensions. Server 116 can be used to process computationally intensive tasks in a multimodal model that consume massive amounts of computational resources, such as the joint processing of a large number of text features and image features. The multimodal model can process data of different modalities; for example, the multimodal model can be a multimodal model that processes data of image modality, video modality, text modality, and / or audio modality, and the multiple modalities can include at least two different modalities. The first media content 102 and the second media content 114 can be content corresponding to different modalities.

[0035] First, first media content 102 is obtained at client 104, such as an image or video input by the user to be processed. The first media content 102 corresponds to a first modality among the multiple modalities that the multimodal model 106 can process. In some embodiments, the multiple modalities may include a visual modality, a language modality, and an auditory modality. In some embodiments, the multiple modalities may include an image modality, a video modality, a text modality, and an audio modality. In this case, the first media content 102 may correspond to one of the aforementioned modalities.

[0036] For example, when the first media content 102 is an image, it corresponds to an image modality or a visual modality. If the first media content 102 is text content, it corresponds to a language modality or a text modality. When the media content 102 is audio, it corresponds to an auditory modality or an audio modality.

[0037] Subsequently, the first media content 102 is processed using the first part 106 of the multimodal model deployed on the client 104 to generate a first content feature 108 for the first media content 102. Next, the client 104 sends the first content feature 108 to the server 112 to process the first content feature 108 and the second content feature 114 of the second media content 110 corresponding to the second modality in the multimodal model, using the second part 116 of the multimodal model deployed on the server 112. The second part 116 of the multimodal model has a higher computational cost than the first part 106, and the second part 116 has a higher number of parameters than the first part 106.

[0038] In some embodiments, the second content feature 114, similar to the first content feature 108, is also obtained by processing the second media content 110 via the first part 106 of the multimodal model. In some embodiments, the second content feature 114 is obtained by processing the second media content 110 via the second part 116 of the multimodal model. The above examples are merely for describing this disclosure and are not intended to specifically limit this disclosure.

[0039] In some embodiments, the first part 106 of the multimodal model in the client 104 is used to process content with low computational load and low computational resource consumption, such as obtaining features of the first media content 102. The second part 116 of the multimodal model in the server 116 is used to process content with high computational load and high computational resource consumption, such as processing a combination of the first content features and the second content features.

[0040] In some embodiments, the computational workload and number of parameters of the computational task handled by the second part 116 of the multimodal model are much greater than the computational workload and number of parameters of the computational task handled by the first part 106 of the multimodal model. The number of parameters of the computational task handled by the second part 116 of the multimodal model and the first part 106 of the multimodal model may differ by several orders of magnitude.

[0041] Additionally, the first part 106 of the multimodal model is used only to process user-related data, such as user-provided image data, while the second part 116 of the multimodal model is used to process feature data transformed from user-provided data. In some embodiments, the client 104 also includes hardware devices that can accelerate computational tasks, such as graphics processors, accelerators, etc.

[0042] After performing computational tasks on the first content feature 108 and the second content feature 114 using the second part 116 of the multimodal model, the resulting processing result 118 is sent from the server 116 to the client 104. It is understood that communication between the client 104 and the server 116 is achieved over a network. Figure 1 The description of a multimodal model for handling first and second media content is merely an example and not a specific limitation of this disclosure. This multimodal model can handle any suitable number of modalities of media content.

[0043] In some embodiments, to reduce the amount of data transmitted, the first content feature 108 can be quantized. For example, the data in the first content feature 108 can be converted from a 32-bit floating-point number to an 8-bit integer. Then, after the server 112 receives the quantized first content feature 108, it performs dequantization to obtain the recovered first content feature. Additionally, after quantizing the first content feature 108, the quantized first content feature can be encoded, for example, using a Portable Network Graphics (PNG) method to further compress the data. The server 112 can decode the compressed and encoded first content feature to obtain the second content feature 114, and can convert the dimension to the same dimension as the second content feature 114 so that the second part 116 of the multimodal model can be processed uniformly.

[0044] This method processes user-provided content on the client side, maximizing the use of both client and server computing resources, reducing server computing load, decreasing overall service bandwidth, and accelerating computing tasks using client-side hardware. This significantly improves the efficiency of computing task execution, rationally allocates computing resource consumption, and enhances the user experience when performing computing tasks.

[0045] The above combination Figure 1 The following is a schematic diagram illustrating an example environment in which some embodiments of this disclosure may be implemented, in conjunction with... Figure 2 Examples of deploying multimodal models according to some embodiments of this disclosure are described.

[0046] like Figure 2 As shown in Example 200, an example of deploying a multimodal model during data processing is illustrated. In Box 202, large-scale models are suitable for cloud computing and are thus deployed in the cloud. Currently, some multimodal models deployed in the cloud typically possess extremely high data processing capabilities, capable of handling very large throughputs of data simultaneously, and due to their extremely high computing power requirements, they usually require a large number of hardware devices.

[0047] Cloud servers typically accept data processing requests from different users over the network and can handle a large number of different requests simultaneously. Therefore, they need to have very high uplink and downlink bandwidth to support data transmission. Simultaneously, they also need storage capacity to store massive amounts of data, including user-related data and other relevant information.

[0048] At box 204, the visual modality matching layer can be partially edge-deployed, allowing it to be deployed on the client side. Since the visual modality matching layer typically interacts directly with user-provided data, and this data is usually relatively small, it can be processed directly on the client side, for example, using the user's device's built-in graphics processor or other types of hardware accelerators. Therefore, moving the visual modality matching layer to the client side for direct processing reduces the server's computational load and decreases the bandwidth increment and latency of data transmission between the client and server, especially noticeable when dealing with a large number of users.

[0049] At box 206, the language modality adaptation layer can be partially edge-enabled, allowing it to be deployed on the client side. Similar to the visual modality matching layer, the language modality matching layer also generally directly interacts with user-provided data, and can also process visual modality-related content directly on the client side. Additionally, the audio modality matching layer can also be partially edge-enabled, processing user-related audio data directly on the client side.

[0050] By deploying this multimodal model with partial client-side processing, user-provided content can be processed on the client side, maximizing the utilization of client and server computing resources, reducing server computing load, decreasing overall service bandwidth, and accelerating computing tasks using edge hardware, greatly improving the execution efficiency of computing tasks, rationally allocating computing resource consumption, and enhancing the user experience when performing computing tasks.

[0051] The above combination Figure 2 Examples of deploying multimodal models according to some embodiments of this disclosure are described below, in conjunction with Figure 3 A schematic diagram illustrating an example method 300 for processing data according to some embodiments of the present disclosure. Figure 3 The method shown can be derived from Figure 1 It can be executed on client 104 or any suitable computing device.

[0052] like Figure 3 As shown, in example method 300, at box 302, first media content 102 is obtained at client 104. First media content 102 corresponds to a first modality among the multiple modalities of the multimodal model. Typically, the data processed by the multimodal model is provided by the user through a client device, such as images or videos that the user wants to process. Additionally, the user can also input information from the client for other modalities of the multimodal model, such as text information.

[0053] In some embodiments, the first media content may consist of only a single modality, such as only image content, video content, text content, or audio content. In other embodiments, the first media content may include multimodal content, which may include multiple or all of image content, video content, text content, or audio content.

[0054] At box 304, a first content feature for the first media content is generated based on the first part of the multimodal model deployed on the client. To make efficient use of the client's hardware resources, a portion of the multimodal model's functionality can be configured on the client to process the media content acquired by client 104. For example, if the first media content 102 includes image content, the first part of the multimodal model's image processing service can be deployed on client 104 to process the image content in the first media content.

[0055] In some embodiments, when the image resolution in the first media content 102 is 1920x1080 and is a JPEG image, the first part of the multimodal model can be used to perform dimensionality processing on the image, such as reducing the image's dimensionality. For example, the 1920x1080 resolution can be processed by an encoder to obtain an image with a resolution of 320x180, and then the feature vector of the reduced-resolution image can be generated as the first content feature for the first media content.

[0056] Additionally, the encoded or quantized image can be encoded in an image format. For example, a JPEG image can be encoded into a PNG image. Compared to traditional image formats, PNG images can be stored repeatedly, are less prone to distortion, and can achieve complex image effects.

[0057] Additionally, when the first media content includes an image, the multimodal model used in the process of generating the first content feature is a stable diffusion model, and the encoder in the first part used for compressing and encoding the image is a variational autoencoder in the stable diffusion model.

[0058] In some embodiments, when the first media content includes video content, the first part of the multimodal model can also be used to process the video content portion of the first media content. When processing the video content, since videos are typically generated through compression encoding, a decoding operation is first required to determine the decoded video. Subsequently, frame extraction is performed on the decoded video, for example, using a pre-set frame extraction algorithm or a previously manually extracted frame extraction script. In one example, 10 frames are extracted from an original 24-frame video, and then a set of video frames is determined using these 10 extracted images. This set of video frames is then preprocessed. Common preprocessing methods include adjusting the size of the video frames, performing color space transformation on the images of the video frames, and / or normalizing the video frames.

[0059] In some embodiments, a set of video frames can be uniformly reduced or enlarged. In some embodiments, adjusting the size of video frames includes unifying image sizes, merging frames of different sizes into images of the same size. One method is to use a cropping algorithm to extract a portion (224x224) of the original image (e.g., an original image of size 256x256) while preserving the original image's true aspect ratio. This can be done by cropping from the center of the original image or from each of the four corners. The cropping algorithm can extract the main information from the image while ignoring other less important information. Another method is to use a padding method to fill the top, bottom, or sides of the original image with a specified value, such as filling the top, bottom, or sides of the image with a black border of the same size. This method does not change the original data format and does not lose the original information of the image.

[0060] Preprocessing also includes spatial conversion of the image colors of a set of video frames. For example, spatial conversion identifiers can be used to convert between different colors and spaces, such as red (R), green (G), blue (B) and BGR, or lightness (Y), chroma (U), and saturation (V). Single-pixel mapping can also be used to convert image colors from RGB to hue (H), saturation (S), and lightness (V) spaces, and bitwise operations can be used to convert image colors from RGB to YUV spaces.

[0061] Furthermore, the converted images can be normalized to convert the pixel values ​​of a set of video image data to a specified range. Additionally, when the first media content includes video, the multimodal model used in generating the first content features is a bootstrap language-image pre-trained model using a frozen image encoder and a large language model, the first part of which includes a portion of a visual transformer.

[0062] At box 306, the first content feature is sent to the server to process the first content feature and the second content feature of the second media content corresponding to the second modality in the multimodal model deployed on the server. The computational cost of the second part is greater than that of the first part, and the number of parameters of the second part is greater than that of the first part.

[0063] In some embodiments, after obtaining the first content feature, the client 104 will also perform quantization processing on the first content feature 108, for example, quantizing the original 32-bit floating-point precision or 16-bit floating-point precision first content feature data into integer precision, so as to reduce the bandwidth and time required for subsequent data transmission.

[0064] Subsequently, the quantized first content feature is transmitted to the server for further processing using the second part of the multimodal model deployed on the server. Additionally, to ensure consistent accuracy of the content features, the transmitted first content feature needs to be dequantized to restore its original data precision and guarantee the accuracy of subsequent calculations.

[0065] After the first content feature is quantized, it can be further encoded, for example, by performing PNG encoding, to determine the quantized first content feature as the encoded first content feature. Subsequently, the client 104 transmits the encoded first content feature to the server 112, which uses the multimodal model deployed on the server for further computation and processing.

[0066] At box 308, the server receives the processing results for the first and second media content output by the multimodal model. After processing the first content feature 102 and the second content feature 114, the server 112 returns the corresponding processing results to the client 104. Additionally, the second part 116 of the multimodal model can also process other content features.

[0067] In this system, client 104 and server 112 communicate via a network. The bandwidth increment and time required for communication are determined by the data processing speed of client 104 and server 112, as well as the size of the data contained in the processing results. It should also be noted that the number of parameters in the data processed by the server is far greater than the number of parameters in the data processed by the client; therefore, the computational load on the server is also far greater than that on the client.

[0068] Additionally, to rationally allocate the portions of the multimodal model required for deployment on the client and server, the multimodal model needs to be partitioned, for example, into a first candidate part and a second candidate part. Subsequently, the first and second candidate parts can be used to process the first media content data. Then, calculations are performed on data transmission-related time degradation and bandwidth increments to determine the likelihood that the first and second candidate parts can be used as the first and second parts, respectively.

[0069] For example, when processing the first media content, if the time degradation between the first candidate part and the second candidate part does not exceed a first threshold, such as 10%, and the bandwidth increment does not exceed the corresponding second threshold, such as 1.1 times, then the first candidate part and the second candidate part can be used as the first part and the second part.

[0070] If the time degradation of the first media content between the first candidate part and the second candidate part exceeds the first threshold or the bandwidth increment exceeds the corresponding second threshold, then the first candidate part and the second candidate part will not be used as the first part and the second part.

[0071] In some embodiments, the first media content and the second media content can be different media content, and the specific content information they contain may be different. Additionally, the first media content and the second media content can be different parts of the same original media content, together forming the complete original media content.

[0072] This method allows user-provided content to be processed on the client side instead of the server, maximizing the utilization of computing resources on both the client and server sides, reducing the server's computing load, decreasing the overall bandwidth of the service, and accelerating computing tasks using client-side hardware. This significantly improves the execution efficiency of computing tasks, rationally allocates the consumption of computing resources, and enhances the user experience when performing computing tasks.

[0073] The above combination Figure 3 A schematic diagram of an example method 300 for processing data according to some disclosed embodiments is described below. Figure 4 A schematic diagram illustrating an example method 400 for processing data according to some embodiments of the present disclosure. Figure 4 The method shown can be derived from Figure 1 It can be executed on server 112 or any suitable device.

[0074] like Figure 4As shown, in example method 400, at block 402, server 112 receives a first content feature 108 from client 104 for first media content 102, the first media content 102 corresponding to a first modality among multiple modalities of a multimodal model, and the first content feature is generated by processing the first media content through a first part of the multimodal model.

[0075] In some embodiments, after the first media content 102 has been processed by the first part 106 of the multimodal model deployed on the client 104, the server 112 receives a first content feature 108 for the first media content 102 transmitted from the client 104.

[0076] Additionally, the first content feature is a quantized first content feature. Upon receiving the quantized first content feature, to ensure data accuracy during further processing by the second part, the quantized first feature content will first undergo dequantization. After dequantization is complete, the first content feature will be provided to the second part of the server-side multimodal model for processing.

[0077] At box 404, a second content feature of the second media content corresponding to the second modality in the multimodal model is determined. This multimodal model can process data from multiple modalities. Additionally, besides processing the first media content 102 and the second media content 110, this multimodal model can also process content features of other content.

[0078] In some embodiments, similar to the first content feature 108, the second content feature 114 can also be obtained by the client 104 processing the second media content 110. In some embodiments, the second content feature 114 can be calculated by applying the second media content to a second part of a multimodal model of the server.

[0079] At box 406, processing results for the first and second media content are generated by applying the first and second content features to the second part of the multimodal model. The computational complexity and number of parameters in the second part are greater than those in the first part. The second part includes a large-scale model computation component of the multimodal model, which can simultaneously process the first content features, the second content features, and their combined features. Additionally, if the multimodal model can process other content, other content features can also be processed in the second part.

[0080] In some embodiments, when the second media content includes image content, the multimodal model is a stable diffusion model, and the second part is a contrastive language-image pre-trained model and a U-shaped network model of the stable diffusion model.

[0081] In some embodiments, when the second media content includes video content, the multimodal model is a bootstrap language-image pre-trained model of a frozen image encoder and a large language model, and the second part includes another part of a visual transformer, a query transformer, and a large language model.

[0082] At box 408, the processing results for the first and second media content are sent to client 104. After server 112 processes the data features, it can return the processing results to client 104. For example, if the multimodal model is a text-to-image model, the processing result is the generated image. Then, client 104 presents the processing results.

[0083] This method allows user information-related content to be processed on the client side, maximizing the use of client and server computing resources, reducing server computing load, decreasing overall service bandwidth, and accelerating computing tasks using edge hardware, thus greatly improving the execution efficiency of computing tasks, rationally allocating computing resource consumption, and enhancing the user experience when performing computing tasks.

[0084] The above combination Figure 4 A schematic diagram of an example method 400 for processing data according to some disclosed embodiments is described below. Figure 5 A schematic diagram illustrating an example of a stable diffusion model for a multimodal model according to some embodiments of the present disclosure.

[0085] like Figure 5 Taking Example 500 as an example, the multimodal model in this example is a stable diffusion model. The first part is the variational autoencoder of the stable diffusion model, and the second part is the contrastive language-image pre-trained model and the U-shaped network model of the stable diffusion model. The parts with solid borders represent the computational tasks performed on the client side, and the parts with dashed borders represent the computational tasks performed on the server side.

[0086] First, on the client side, at the 502 error, the user selects an image (resolution 720x1280) and a prompt. The image resolution is not limited to 720x1280; the user can also select images of other resolutions, such as 540x960. The image format can be common formats such as JPEG, JPG, and PNG. This application does not impose any restrictions on this.

[0087] After the user selects an image, the image is input into a Variational Auto-Encoder (VAE) 504. The VAE encoder then processes the input image; for example, it calculates that the original image resolution is reduced to one-eighth, meaning the original 720x1280 image is transformed into eight 90x160 resolution images. During image processing, the data type of the image is also determined, for example, specifying it as a 32-bit floating-point image (fp32).

[0088] Subsequently, at position 506, compressed and encoded image features are provided. Then, at position 508, online quantization and encoding are performed, quantizing the previously encoded image features and performing operations such as PNG encoding, finally determining the quantized and encoded image features.

[0089] Subsequently, at position 510, client 104 initiates a network request. After receiving the network request, server 112 parses it. After obtaining the quantized image features, server 112 first performs dequantization at position 512 to ensure that the dimension of the image features is the same as the dimension before quantization.

[0090] Subsequently, at point 514, the second part of the server performs operations such as contrastive language-image pretraining (Clip) and multi-step sampling of the image features of the user-selected image and the prompt text using a U-shaped network (U-Net). After the computation task of the second part is completed, at point 516, the processed result with a data precision of fp32 is further sent down to the client.

[0091] After receiving the compressed and encoded processing result, the client 104 uses the VAE decoder 518 to perform the decoding operation, and finally displays the decoded processing result on the client screen 520.

[0092] surface combination Figure 5 A schematic diagram illustrating an example of a stable diffusion model based on some disclosed embodiments of the multimodal model is described below. Figure 6 A schematic diagram illustrating an example of a bootstrap-image pre-trained model of a multimodal model according to some embodiments of the present disclosure.

[0093] like Figure 6Taking Example 600 as an example, the multimodal model in this example is a bootstrap language-image pre-trained model using a frozen image encoder and a large language model. The first part includes a portion of the Vision Transformer (ViT), and the second part includes another portion of the Vision Transformer, a Querying Transformer (Qformer), and a large language model (LLM). The solid-lined borders represent the computational tasks performed on the client side, and the dashed-lined borders represent the computational tasks performed on the server side.

[0094] First, at position 602, the user selects a video. After the client receives the selected video, at position 604, it performs video decoding and frame extraction (16 frames). At this point, the compressed video selected by the user is decoded, and a set of 16 video frames is extracted from the video using a frame extraction algorithm or script. Then, similar to Example 500, the resolution of each video frame in the set is adjusted to 90x160, and the data type for the image is determined to be 32-bit floating-point numbers.

[0095] Subsequently, at point 608, preprocessing and a portion of the model processing for the edge-side ViT are performed, which is part of the visual transformer. At this point, a set of video frames can be preprocessed, including resizing the images corresponding to the video frames, spatial conversion or normalization of image colors, etc., and then processed using a portion of ViT.

[0096] Subsequently, at position 610, online feature quantization is performed. At position 612, the features are quantized by 4 bits and organized into 8-bit integer (u)8 data, which is then uploaded to the network server. The u8 data is calculated as 16 (frames) x 257 token size x 1408 feature dimensions.

[0097] Subsequently, at position 614, the client initiates a network request. Upon receiving the request, the server, similarly at position 616, dequantizes the quantized set of video frames and at position 618 provides a set of video frames with a resolution of 90x160 to the second part of the model. This means that the content features are sent as 32-bit floating-point data to the second part of the multimodal model, namely the cloud-based ViT part2 model, Qformer, and LLM at position 620. Then, at position 622, the processed result is sent to the client as a string.

[0098] like Figure 7As shown, the device 700 includes a first media content acquisition module 702, configured to acquire first media content at a client, the first media content corresponding to a first modality among multiple modalities of a multimodal model; a first content feature generation module 704, configured to generate a first content feature for the first media content based on a first part of a multimodal model deployed on the client; a first content feature and a second content feature processing module 706, configured to send the first content feature to a server to process the first content feature and the second content feature of the second media content corresponding to a second modality among multiple modalities using a second part of a multimodal model deployed on the server, the second part having a greater computational load than the first part and the second part having a greater number of parameters than the first part; and a processing result receiving module 708, configured to receive the processing result for the first media content and the second media content output by the multimodal model from the server.

[0099] In some embodiments, the first content feature and the second content feature processing module 706 includes: a first content feature quantization module configured to quantize the first content feature to generate a quantized first content feature; and a quantized first content feature sending module configured to send the quantized first content feature to a server.

[0100] In some embodiments, the quantized first content feature sending module includes: a first content feature encoding module configured to generate an encoded first content feature by encoding the quantized first content feature; and an encoded first content feature sending module configured to send the encoded first content feature to a server.

[0101] In some embodiments, the first media content includes an image, and the first content feature generation module 704 includes an image dimensionality reduction module configured to generate a first content feature for the image by reducing the dimension of the image using a first part of a multimodal model.

[0102] In some embodiments, the multimodal model is a stable diffusion model, the first part is a variational autoencoder of the stable diffusion model, and the second part is a contrastive language-image pre-trained model and a U-shaped network model of the stable diffusion model.

[0103] In some embodiments, the apparatus 700 further includes a processing result decoding module configured to perform a decoding operation on the processing result.

[0104] In some embodiments, the first media content includes video, and the first content feature generation module 704 includes: a video decoding module configured to perform a decoding operation on the video to determine a decoded video; a set of video frames determination module configured to determine a set of video frames for the video by extracting frames from the decoded video; a set of video frames preprocessing module configured to preprocess the set of video frames to generate a preprocessed set of video frames; and a first content feature generation module configured to generate a first content feature for the preprocessed set of video frames based on a first part of a multimodal model deployed on the client.

[0105] In some embodiments, preprocessing a set of video frames includes at least one of the following: a video frame size adjustment module configured to adjust the size of each video frame in the set of video frames; an image color space conversion module configured to perform color space conversion on the image of the video frames; or a video frame normalization module configured to perform normalization processing on the video frames.

[0106] In some embodiments, the multimodal model is a bootstrap language-image pre-trained model using a frozen image encoder and a large language model, the first part including a portion of a visual transformer, and the second part including another portion of the visual transformer, a query transformer, and a large language model.

[0107] In some embodiments, the apparatus 700 further includes: a multimodal model partitioning module configured to partition a multimodal model into a first candidate portion that can be deployed on a client and a second candidate portion that can be deployed on a server; a time-consuming degradation and bandwidth increment determination module configured to determine, based on first media content, the time-consuming degradation of data transmission between the first candidate portion and the second candidate portion and the bandwidth increment between the client and the server; and a probability determination module configured to determine, based on the time-consuming degradation and bandwidth increment, the probability that the first candidate portion and the second candidate portion are used as the first portion and the second portion.

[0108] In some embodiments, the first media content and the second media content include at least one of the following: video content, image content, text content, and audio content.

[0109] like Figure 8As shown, the apparatus 800 includes a first content feature receiving module 802, configured to receive a first content feature for a first media content from a client at a server, the first media content corresponding to a first modality among multiple modalities of a multimodal model, and the first content feature being generated by processing the first media content through a first part of the multimodal model; a second content feature determining module 804, configured to determine a second content feature for a second media content corresponding to a second modality among multiple modalities; a processing result generating module 806, configured to generate a processing result for the first media content and the second media content by applying the first content feature and the second content feature to a second part of the multimodal model, the computational load and number of parameters of the second part being greater than the computational load and number of parameters of the first part; and a processing result sending module 808, configured to send the processing result for the first media content and the second media content to the client.

[0110] In some embodiments, the first content feature receiving module 802 includes: a quantized first content feature receiving module configured to receive quantized first content features for first media content from a client at a server.

[0111] In some embodiments, the processing result generation module 806 includes: a quantized first content feature dequantization module, configured to obtain a first content feature by dequantizing the quantized first content feature; and a processing result generation module, configured to generate a processing result for the first media content and the second media content by applying the first content feature and the second content feature to a second part of a multimodal model.

[0112] In some embodiments, the multimodal model is a stable diffusion model, the first part is a variational autoencoder of the stable diffusion model, and the second part is a contrastive language-image pre-trained model and a U-shaped network model of the stable diffusion model.

[0113] In some embodiments, the multimodal model is a bootstrap language-image pre-trained model using a frozen image encoder and a large language model, the first part including a portion of a visual transformer, and the second part including another portion of the visual transformer, a query transformer, and a large language model.

[0114] Figure 9 A schematic block diagram of an example device 900 that can be used to implement embodiments of the present disclosure is shown. Figure 1The client 104 and server 112 can be implemented using device 900. As shown, device 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 902 or loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 can also store various programs and data required for the operation of device 900. CPU 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0115] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage page 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0116] The various processes and handling described above, such as methods 300 and 400, can be executed by processing unit 901. For example, in some embodiments, methods 300 and 400 can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by CPU 901, one or more actions of the example methods 300 and 400 described above can be performed.

[0117] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0118] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0119] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0120] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0121] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0122] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0123] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0125] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for processing data, comprising: The first media content is obtained at the client, and the first media content corresponds to the first modality among the multiple modalities for the multimodal model; Based on the first part of the multimodal model deployed on the client, a first content feature is generated for the first media content; The first content feature is sent to the server to process the first content feature and the second content feature of the second media content corresponding to the second modality in the multiple modalities using the second part of the multimodal model deployed on the server, wherein the computational cost of the second part is greater than that of the first part and the number of parameters of the second part is greater than that of the first part. as well as Receive from the server the processing results of the first media content and the second media content output by the multimodal model.

2. The method of claim 1, wherein sending the first content feature to a server to process the first content feature and a second content feature of second media content corresponding to a second modality among the multiple modalities using a second portion of the multimodal model deployed on the server comprises: The first content feature is quantized to generate a quantized first content feature; as well as The quantified first content feature is sent to the server.

3. The method of claim 2, wherein sending the quantified first content feature to the server comprises: The first content feature is generated by encoding the quantized first content feature; as well as The encoded first content feature is sent to the server.

4. The method of claim 1, wherein the first media content includes an image, and generating a first content feature for the first media content includes: The first content feature for the image is generated by reducing the dimensionality of the image using the first part of the multimodal model.

5. The method according to claim 4, wherein the multimodal model is a stable diffusion model, the first part is a variational autoencoder of the stable diffusion model, and the second part is a contrastive language-image pre-trained model and a U-shaped network model of the stable diffusion model.

6. The method according to claim 5, further comprising: The processing result is then decoded.

7. The method of claim 1, wherein the first media content includes video, and generating a first content feature for the first media content includes: The video is decoded to determine the decoded video; By extracting frames from the decoded video, a set of video frames for the video is determined; The set of video frames is preprocessed to generate the preprocessed set of video frames; as well as Based on the first part of the multimodal model deployed on the client, a first content feature is generated for the preprocessed set of video frames.

8. The method of claim 7, wherein preprocessing the set of video frames comprises at least one of the following: Adjust the size of each video frame in the group of video frames; Perform spatial conversion on the image colors of the video frame; or The video frames are normalized.

9. The method of claim 8, wherein the multimodal model is a bootstrap language-image pre-trained model using a frozen image encoder and a large language model, the first portion comprising a portion of a visual transformer, and the second portion comprising another portion of a visual transformer, a query transformer, and a large language model.

10. The method according to claim 1, further comprising: The multimodal model is divided into a first candidate part that can be deployed on the client and a second candidate part that can be deployed on the server; Based on the first media content, determine the time degradation of data transmission between the first candidate part and the second candidate part and the bandwidth increment between the client and the server; as well as Based on the time consumption degradation and the bandwidth increment, the probability that the first candidate portion and the second candidate portion are used as the first portion and the second portion is determined.

11. The method of claim 1, wherein the first media content and the second media content comprise at least one of the following: Video content, image content, text content, and audio content.

12. A method for processing data, comprising: The server receives a first content feature from the client for a first media content, the first media content corresponding to a first modality among multiple modalities of a multimodal model, and the first content feature is generated by processing the first media content through a first part of the multimodal model; Determine a second content feature of the second media content corresponding to the second modality among the multiple modalities; The first content feature and the second content feature are applied to the second part of the multimodal model to generate the processing result for the first media content and the second media content. The computational cost of the second part is greater than that of the first part, and the number of parameters in the second part is greater than that in the first part. as well as The processing results for the first media content and the second media content are sent to the client.

13. The method of claim 12, wherein receiving a first content feature for the first media content from a client at the server comprises: The server receives quantified first content features for the first media content from the client.

14. The method of claim 13, wherein generating a processing result for the first media content and the second media content by applying the first content feature and the second content feature to a second part of the multimodal model comprises: The first content feature is obtained by inverse quantization of the quantized first content feature; as well as The processing results for the first media content and the second media content are generated by applying the first content feature and the second content feature to the second part of the multimodal model.

15. The method of claim 12, wherein the multimodal model is a stable diffusion model, the first part is a variational autoencoder of the stable diffusion model, and the second part is a contrastive language-image pre-trained model and a U-shaped network model of the stable diffusion model.

16. The method of claim 12, wherein the multimodal model is a bootstrap language-image pre-trained model using a frozen image encoder and a large language model, the first portion comprising a portion of a visual transformer, and the second portion comprising another portion of a visual transformer, a query transformer, and a large language model.

17. The method of claim 12, further comprising: The multimodal model is divided into a first candidate part that can be deployed on the client and a second candidate part that can be deployed on the server; Based on the first media content, determine the time degradation of data transmission between the first candidate part and the second candidate part and the bandwidth increment between the client and the server; as well as Based on the time consumption degradation and the bandwidth increment, the probability that the first candidate portion and the second candidate portion are used as the first portion and the second portion is determined.

18. An apparatus for processing data, comprising: The first media content acquisition module is configured to acquire first media content at the client, the first media content corresponding to the first modality among multiple modalities for the multimodal model; The first content feature generation module is configured to generate a first content feature for the first media content based on a first part of the multimodal model deployed on the client. A first content feature and a second content feature processing module are configured to send the first content feature to a server to process the first content feature and the second content feature of the second media content corresponding to the second modality in the multiple modalities using a second part of the multimodal model deployed on the server. The computational amount of the second part is greater than that of the first part, and the number of parameters of the second part is greater than that of the first part. as well as The processing result receiving module is configured to receive from the server the processing results for the first media content and the second media content output by the multimodal model.

19. An apparatus for processing data, comprising: A first content feature receiving module is configured to receive a first content feature for a first media content from a client at a server, the first media content corresponding to a first modality among multiple modalities of a multimodal model, and the first content feature is generated by processing the first media content through a first part of the multimodal model. The second content feature determination module is configured to determine the second content feature of the second media content corresponding to the second modality among the multiple modalities; The processing result generation module is configured to generate processing results for the first media content and the second media content by applying the first content feature and the second content feature to a second part of the multimodal model, wherein the computational amount of the second part is greater than that of the first part and the number of parameters of the second part is greater than that of the first part. as well as The processing result sending module is configured to send the processing results for the first media content and the second media content to the client.

20. An electronic device, comprising: At least one processor; as well as A storage device for storing at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the method according to any one of claims 1-17.

21. A computer-readable storage medium having a computer program stored thereon, the computer program implementing the method according to any one of claims 1-17 when executed by a processor.

22. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-17.

Images