A method for processing multimedia data, storage medium and electronic device

By splitting access requests into remote and local processing tasks, using large models to process multimedia data, the resource consumption and security issues during the processing of large models are solved, and efficient and secure multimedia data processing and rendering are achieved.

CN118535750BActive Publication Date: 2024-11-15ZHEJIANG NIAOCHAO SUPPLY CHAIN MANAGEMENT CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411001198.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-15
Estimated Expiration
2044-07-24

AI Technical Summary

Technical Problem

In the prior art, there are problems with high resource consumption and data security in the processing of large model data, especially in data transmission and processing, which can easily lead to waste of computing resources and data leakage.

Method used

By splitting the access request into two pending tasks, they are processed by the remote large model and the local large model respectively. The remote large model is responsible for obtaining multimodal access data and synthesis method data, and the local large model performs feature recognition and synthesis processing. , generate multimedia data and render output locally.

Benefits of technology

It realizes the improvement of data security while ensuring processing efficiency, makes full use of cloud and local resources, supports super-large resources, personalized and real-time features, and can realize offline + real-time application service platform operation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118535750B_ABST
    Figure CN118535750B_ABST
Patent Text Reader

Abstract

The present application discloses a method for processing multimedia data, a storage medium and an electronic device, wherein the method comprises: according to the analysis of an access request, splitting the to-be-processed task corresponding to the access request into a first to-be-processed task and a second to-be-processed task, and sending the first to-be-processed task to a first large model or a data server; obtaining multimodal access data corresponding to the first to-be-processed task and synthesis method data of the multimodal access data determined by the first large model or the data server according to the first to-be-processed task; taking the synthesis method data and the multimodal page data as the second to-be-processed task, processing the second to-be-processed task through a locally deployed second large model, and generating multimedia data corresponding to the access request; thereby, while avoiding the consumption of computing resources and improving data security, the advantages of both the cloud large model (or the data server) and the local end-side environment can be fully utilized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more particularly to a method and device for processing multimedia data. The present application also relates to a multimedia data processing system, a multimedia data interaction method and device, a computer storage medium, and an electronic device. Background Art

[0002] Large models refer to machine learning models with large-scale parameters and complex computational structures. These models are usually built with deep neural networks and have billions or even hundreds of billions of parameters. Large models are designed to improve the expressiveness and predictive performance of the model and to handle more complex tasks and data. Since the number of parameters in large models can reach billions or even hundreds of billions, they can handle massive amounts of data and complex tasks. Therefore, they are widely used in natural language processing, computer vision, finance, healthcare, smart manufacturing, smart transportation and other fields. For example, large models can be used for tasks such as machine translation, speech recognition, text summarization, sentiment analysis, and image recognition. Summary of the invention

[0003] The present application provides a method for processing multimedia data to solve the problems of security and data transmission limitation involved in the data processing process in the prior art.

[0004] The present application provides a method for processing multimedia data, comprising:

[0005] According to the analysis of the access request, split the pending task corresponding to the access request into at least two pending tasks including a first pending task and a second pending task, and send the first pending task to the first large model or the data service end; wherein the first large model is a remote large model;

[0006] Acquire multimodal access data corresponding to the first task to be processed, determined by the first large model or the data server according to the first task to be processed, and data of a synthesis method of the multimodal access data;

[0007] The synthesis method data and the multimodal page data are taken as the second task to be processed, and the second task to be processed is processed by a locally deployed second large model to generate multimedia data corresponding to the access request; wherein the second large model is a local large model.

[0008] In some embodiments, the step of splitting the to-be-processed task corresponding to the access request into at least two to-be-processed tasks including a first to-be-processed task and a second to-be-processed task according to the parsing of the access request, and sending the first to-be-processed task to the first large model or the data server includes:

[0009] Determining an access page corresponding to the access request based on the analysis of the access request;

[0010] The to-be-processed task generated according to the access page is split to determine the first to-be-processed task and the second to-be-processed task.

[0011] In some embodiments, splitting the to-be-processed task generated according to the access page to determine the first to-be-processed task and the second to-be-processed task includes:

[0012] Determine an acquisition task for acquiring the multimodal page data in the accessed page as the first task to be processed;

[0013] A task for synthesizing the multimodal page data in the accessed page is determined as the second task to be processed.

[0014] In some embodiments, the obtaining of the multimodal page data corresponding to the first task to be processed and determined by the first large model or the data server according to the first task to be processed, and the synthesis method data of the multimodal page data, includes:

[0015] Acquire the multimodal subject object data and multimodal image material data corresponding to the access request, which are searched by the first large model or the data server according to the first task to be processed;

[0016] The synthesis method data is obtained by the first large model or the data server according to the multimodal main data and the multimodal material data.

[0017] In some embodiments, the taking the synthesis method data and the multimodal page data as the second task to be processed, processing the second task to be processed by a locally deployed second large model, and generating multimedia data corresponding to the access request includes:

[0018] The first end-side large model in the second large model deployed locally determines the page reference data corresponding to the access request according to the requester characteristic data of the access request;

[0019] The second end-side large model in the second large model performs synthesis processing on the multimodal page data and the page reference data according to the synthesis method data to generate the multimedia data.

[0020] In some embodiments, the first end-side large model in the second locally deployed large model determines the page reference data corresponding to the access request according to the requester feature data of the access request, including:

[0021] The first end-side big model identifies the access intention of the requester according to the access request, and determines the target multimodal page data matching the access request from the multimodal page data;

[0022] The second end-side large model in the second large model performs synthesis processing on the multimodal page data and the page reference data according to the synthesis method data to generate the multimedia data, including:

[0023] The second end-side large model performs synthesis processing on the target multimodal page data and the page reference data according to the synthesis method data to generate the multimedia data.

[0024] The present application also provides a multimedia data processing system, comprising: a client, a second large model deployed in the client, and a first large model or a data server; wherein the first large model is a remote large model, and the second large model is a local large model of the client;

[0025] The client is used to generate an access request according to a trigger of the application service platform by a user, and according to the analysis of the access request, split the pending task corresponding to the access request into at least two pending tasks including a first pending task and a second pending task, and send the first pending task to the first large model or data service end; obtain the multimodal access data corresponding to the first pending task and the synthesis method data of the multimodal access data determined by the first large model or data service end according to the first pending task; take the synthesis method data and the multimodal page data as the second pending task, process the second pending task through the second large model, and generate multimedia data corresponding to the access request;

[0026] The first large model or data server is used to search for multimodal access data corresponding to the access request and synthesis method data of the multimodal access data according to the first task to be processed, and send the multimodal access data and the synthesis method data to the second large model locally deployed on the client;

[0027] The second large model is used to determine the page reference data corresponding to the access request based on the requester characteristic data of the access request in the client, and to synthesize the multimodal page data and the page reference data according to the received synthesis method data to generate the multimedia data.

[0028] In some embodiments, the second large model includes: a first end-side large model and a second end-side large model;

[0029] The first end-side large model is used to determine page reference data corresponding to the access request according to the requester characteristic data of the access request;

[0030] The second end-side large model is used to perform synthesis processing on the multimodal page data and the page reference data according to the synthesis method data to generate the multimedia data.

[0031] The present application also provides a multimedia data interaction method, including:

[0032] In response to an access request to an application service page, obtain multimedia data corresponding to the access request as determined by the multimedia data processing method described above; wherein the multimedia data includes: resource object subject information in the access page corresponding to the access request, and reference information synthesized with the resource object subject information and used to describe the resource object subject;

[0033] In response to the acquired multimedia data, the multimedia data is rendered, and the resource object main body information and the reference information are output in a multimodal form in the access page.

[0034] The present application also provides a computer storage medium for storing data generated by a network platform, and a program for processing the data generated by the network platform;

[0035] When the program is read and executed by the processor, the above-mentioned multimedia data processing method is executed.

[0036] The present application also provides an electronic device, including:

[0037] processor;

[0038] The memory is used to store a program for processing data generated by a network platform. When the program is read and executed by the processor, the method for processing multimedia data as described above is executed.

[0039] Compared with the prior art, this application has the following advantages:

[0040] The present application provides a method for processing multimedia data, which is divided into two processing tasks through access requests, and is processed respectively through the first large model (remote large model) or one of the data service ends, and the second large model (local large model), that is: the first large model or the data service end can determine the multimodal page data and synthesis method data required for the relevant page corresponding to the access request according to the first task to be processed, and the second large model can perform feature recognition, extraction and other related processing on the multimodal page data, and use the synthesis method data for synthesis processing based on the processed data, and render and output the synthesized multimedia data through the second large model. In addition, while avoiding computing resource consumption and improving data security, it can give full play to the advantages of both cloud and local end-side environments, support ultra-large resources, personalization, and real-time features, and can also ensure that the application service platform can achieve offline + real-time operation.

[0041] The application provides a multimedia data interaction method, which obtains multimedia data corresponding to an access request based on the above-mentioned multimedia data processing method, and outputs the multimedia data at a display output end according to the rendering of the multimedia data, wherein the multimedia data is output in a multimodal form and includes resource object subject information and reference information of the resource object subject, and may also include entity resource object information corresponding to the access page. Thus, it is possible to realize a display mode of expressing the resource object subject in a multimodal manner at the display output end, improve the diversification and visual impact of the display, and avoid the cost increase problem caused by the consumption of computing resources during the rendering output process. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flowchart of a method for processing multimedia data provided by the present application.

[0043] Figure 2 It is a structural schematic diagram of a multimedia data processing device provided by the present application.

[0044] Figure 3 It is a structural diagram of a multimedia data processing system provided by the present application.

[0045] Figure 4 It is a timing diagram of an embodiment of a multimedia data processing method provided by the present application.

[0046] Figure 5 This is a flowchart of a multimedia data interaction method provided by the present application.

[0047] Figure 6 It is a structural schematic diagram of a multimedia data interaction device provided by the present application.

[0048] Figure 7It is a structural schematic diagram of an electronic device provided by this application. DETAILED DESCRIPTION

[0049] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0050] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The descriptions used in this application and the appended claims, such as "a", "first", and "second", are not limitations on quantity or sequence, but are used to distinguish the same type of information from each other.

[0051] Based on the above background, it can be seen that big model technology is widely used in all walks of life, thereby improving processing capabilities and processing efficiency. In the prior art, whether it is the processing of text information or image information, it is necessary to send the relevant processing data to the big model or server, and obtain the corresponding feedback results through the processing of the big model or server. However, there are at least two problems in this processing process. First, because of the restrictions of the big model or server on data transmission and processing, it will lead to resource consumption problems. Second, the big model or server will cause data leakage, which will lead to data security problems.

[0052] In view of the above problems in the prior art, the present application provides a method for processing multimedia data, such as Figure 1 As shown, Figure 1 This is a flowchart of a method for processing multimedia data provided by the present application, the method comprising:

[0053] Step S101: according to the analysis of the access request, split the to-be-processed task corresponding to the access request into at least two to-be-processed tasks including a first to-be-processed task and a second to-be-processed task, and send the first to-be-processed task to a first large model or a data service end; wherein the first large model is a remote large model;

[0054] Step S102: obtaining multimodal access data corresponding to the first task to be processed, determined by the first large model or the data server according to the first task to be processed, and data on a synthesis method of the multimodal access data;

[0055] Step S103: taking the synthesis method data and the multimodal page data as the second task to be processed, processing the second task to be processed by a locally deployed second large model to generate multimedia data corresponding to the access request; wherein the second large model is a local large model.

[0056] In summary, the present application processes the corresponding tasks to be processed respectively by the first large model and the second large model, or the data service end and the second large model process the corresponding tasks to be processed respectively, which can improve data security while ensuring processing efficiency. The above steps S101 to S103 are described in detail below.

[0057] Regarding step S101: based on the analysis of the access request, split the pending task corresponding to the access request into at least two pending tasks including a first pending task and a second pending task, and send the first pending task to a first large model or a data server; wherein the first large model is a remote large model.

[0058] In this embodiment, the access request may be an access request to the application service platform, for example, an access request to enter the homepage of the application service platform, an access request to a resource object under the current page of the application service platform, an access request to a specific page in the application service platform, and of course, if the application service platform is a life service application, the access request may be an access request to a shopping cart. In other words, the access request may be an access request based on any component of the application service platform that can obtain relevant information.

[0059] The pending tasks can be understood as when an access request occurs, the application service platform needs to output and display the corresponding information, and then needs to obtain the corresponding data according to the access request. This process is the processing process of the pending tasks. Different access requests can be divided into multiple pending tasks. Therefore, the specific implementation process of step S101 may include:

[0060] Step S101-11: determining an access page corresponding to the access request according to the analysis of the access request;

[0061] Step S101 - 12: splitting the to-be-processed task generated according to the access page to determine the first to-be-processed task and the second to-be-processed task.

[0062] The specific implementation process of step S101-12 may include:

[0063] Step S101-121: determining the acquisition task for acquiring the multimodal page data in the accessed page as the first task to be processed;

[0064] Step S101 - 122 : determining the synthesis task for the multimodal page data in the accessed page as the second task to be processed.

[0065] In the specific implementation process, the access request can obtain the relevant information of the page to be accessed through the buried data set by the user on the application service platform. By semantically processing the relevant information of the page to be accessed, the first task to be processed and the second task to be processed are determined in the order of execution. Of course, the access request can also be a request in one or more forms of expression such as voice, image, text, etc. input by the user based on the operation interface provided by the application service platform. For example: based on the application service platform, the voice information input is "help me order a cup of regular milk tea", and the voice information can be sent to the first large model for voice recognition and generate a corresponding prompt, such as obtaining the multimodal data of the page related to milk tea as the first task to be processed, and performing corresponding synthetic rendering on the obtained multimodal data of the page as the second task to be processed. Of course, when the second task to be processed is executed, the user's characteristic data can be obtained in combination with the "regular drink" in the user's voice information, and then the type of milk tea suitable for the user can be located. The specific implementation will be described in step S103 later. The above is only an example of various implementation scheme embodiments. This application is still mainly described by taking the access request as an example of access to the homepage, retail venue, venue, shopping cart, order and other pages of the application service platform. In this embodiment, homepage product recommendation refers to recommending specific goods or services on the homepage of a website or application. Retail venue refers to a page corresponding to a market or field in the retail industry. Venue refers to a specific page for goods or services established by the application platform based on a specific region or a specific time.

[0066] Therefore, in this embodiment, the generation of access requests is not limited to the user's click operation on the application service platform, but may also include a variety of other implementable solutions, such as: voice, reservation (for example, setting a breakfast reservation, and automatically popping up the corresponding breakfast page information at the reservation time), etc.

[0067] Because, in this embodiment, the task to be processed is split into two tasks and the first task to be processed is sent to the first large model or the data server. The first large model is a remote large model and can also be understood as a cloud large model. Therefore, diversified page material data and a synthesis method that is more suitable for diversified page material data can be obtained. In the synthesis process, there is no cost increase problem caused by data transmission requirements. The second task to be processed is sent to the second large model. The second large model is a local large model, that is, a client local large model. When performing synthesis processing, it can not only reduce the excessive consumption of computing resources, but also achieve processing efficiency through local computing resources, avoiding the limitation of data processing capabilities (such as bandwidth, etc.) caused by the remote large model.

[0068] In this embodiment, whether the access request is sent to the first large model or the data server can be determined in combination with the specific access content of the access request. For example, when the page information involved in the access request is a product information arrangement, the corresponding multimodal page data can be obtained through the data server. In other words, before the first task to be processed is processed, it can be determined whether to send the first task to be processed to the first large model or to the data server in combination with the processing requirements of the first task to be processed.

[0069] Regarding the step S102: obtaining the multimodal access data corresponding to the first task to be processed, which is determined by the first large model or the data server according to the first task to be processed, and the synthesis method data of the multimodal access data.

[0070] The multimodal access data may be the relevant data required by the access page corresponding to the access request, that is, the material data and synthesis method data of the product or service, for example: one or more of different types of data such as images, videos, texts, and voices, and may also include the dynamic material template data of the product or service, which may be pre-designed and stored in a cloud server. The first large model may perform a corresponding search based on the cloud server, and the data server may perform a search based on the relevant data stored on the server, or may perform a search based on network data. Therefore, the specific implementation process of step S102 may include:

[0071] Step S102-11: obtaining the multimodal subject object data and multimodal image material data corresponding to the access request, which are searched by the first large model or the data server according to the first task to be processed;

[0072] Step S102-12: Obtain the synthesis method data determined by the first large model or the data server according to the multimodal main data and the multimodal material data.

[0073] The multimodal subject object data in step S102-11 may be multimodal data of goods or services searched according to the access request, for example: the goods data to be output in the access page corresponding to the access request, the goods data may determine the goods data to be recalled according to the location information in the access request, and the recall of the goods data is realized through the first large model or the data service end. In this embodiment, the multimodal subject object data may be one or more combination data of static images, dynamic images, text information, voice information, etc. The multimodal image material data may be one or more of the dynamic material data, static material data, etc. of the goods, and the multimodal image material data may be pre-stored dynamic and / or static template material data, for example: searched by storing in the cloud material service center, or dynamic and / or static material data searched through the network. For example: if the recalled goods are coffee, the dynamic material may be a dynamic effect template material matching the attributes of the goods (such as: hot or cool dynamic effects), or a dynamic effect material for the specified information provided by the goods (such as: the dynamic effect of the preferential activities, of course, it can also be a static material), and the synthesis method data that can be used for the goods and material data.

[0074] Regarding the step S103: taking the synthesis method data and the multimodal page data as the second task to be processed, processing the second task to be processed through the second large model deployed locally, and generating multimedia data corresponding to the access request.

[0075] The purpose of step S103 is to synthesize the multimodal page data sent by the first large model or the data server according to the synthesis method data through the locally deployed second large model to generate multimedia data for output on the client.

[0076] The multimodal page data can be determined based on the access page corresponding to the access request. For example, when the access page corresponding to the access request is the home page of the application service platform, the first large model or data service end can recall the data of the relevant goods or services of the corresponding home page in combination with the access request, and can also search for multimodal image material data, that is, material data for supplementing and modifying the data of goods or services. If the access page corresponding to the access request is the product detail page, the first large model or data service end can search for the specific content data related to the product and the multimodal image material data of the product in combination with the access request. The multimodal image material data can be the activity material data, dynamic template data, dynamic data, etc. of the product. The second large model can also obtain page reference data, that is, real-time personalized data, which can also be understood as recommended data, based on the requester feature data of the access request.

[0077] In order to ensure user privacy and data security, step S103 is to complete the acquisition of real-time personalized data through the second largest model deployed locally. The specific implementation process may include:

[0078] Step S103-11: The first end-side big model in the second locally deployed big model determines the page reference data corresponding to the access request according to the requester feature data (user feature data) of the access request; wherein the first end-side big model may be an end-intelligent SDK (which may also be understood as a visual intelligent end), and integrates the input user feature data acquisition and the page reference data of the access request response, such as: related image processing such as identification and feature extraction, and matching relationship between goods and multimodal image material data. The page reference data may be real-time commodity data, application platform activity data, merchant data, etc. Because the determination of the page reference data for the requester feature data (user feature data) is completed in the first end-side big model (local client), therefore, obtaining the user feature data through the local end-side big model can ensure the security of the data and avoid the risk of user privacy leakage caused by uploading the user feature data to the cloud through the cloud big model. The user feature data may include user historical access data (historical behavior data), user historical acquisition data (historical order data), etc.

[0079] Step S103-12: The second end-side large model in the second large model performs synthesis processing on the multimodal page data and the page reference data according to the synthesis method data to generate the multimedia data. In this embodiment, the second end-side large model can be an end-intelligent PythonVM (Python virtual machine, which can be understood as a data processing end), and the data integrated by the first end-side large model is synthesized according to the synthesis method data to realize the generation of multimedia data in the access page. For example, when the access page is a product detail page, the multimedia data can be a dynamic effect synthesis page data for the product, which can include dynamic effects reflecting product attributes, dynamic effects reflecting the product participating in application service platform activities, and dynamic effects reflecting the product merchant participating in activities. The product details page can also include multimedia data of page reference data (recommended products), and can also include multimedia data synthesized from corresponding dynamic effects and recommended products. The multimedia data can be different according to the type of access page. For example, for the home page, it can include dynamic effect multimedia data for application service platform activities, and can also include dynamic effect multimedia data for merchant activities. By synthesizing and rendering static data and dynamic data through the second-end large model, local GPU computing resources can be used to achieve related processing such as downloading and rendering, which not only avoids waste of computing resources but also improves processing efficiency while reducing processing costs, and avoids resource consumption problems caused by limitations of cloud-based large models in data transmission and processing.

[0080] Therefore, the step S103 includes:

[0081] Step S103-21: the first end-side big model identifies the access intention of the requester according to the access request, and determines the target multimodal page data matching the access request from the multimodal page data;

[0082] Step S103 - 22 : the second end-side large model performs synthesis processing on the target multimodal page data and the page reference data according to the synthesis method data to generate the multimedia data.

[0083] After generating the multimedia data, the method may further include:

[0084] The first end-side large model renders and outputs the multimedia data, that is, outputs it to the end side of the access request.

[0085] In this embodiment, because the access request is split into two processing tasks and processed by the remote large model and the local large model respectively, it can fully utilize the advantages of both cloud and local end environments while avoiding computing resource consumption and improving data security, support ultra-large resources, personalization, and real-time features, and can also ensure that the application service platform can achieve offline + real-time operation.

[0086] The above is a detailed description of an embodiment of a method for processing multimedia data provided by the present application. Corresponding to the embodiment of a method for processing multimedia data provided above, the present application also discloses an embodiment of a device for processing multimedia data. Figure 2 Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0087] like Figure 2 As shown, Figure 2 : is a schematic diagram of the structure of a multimedia data processing device provided by the present application. The device embodiment may include:

[0088] The sending unit 201 is used to split the pending task corresponding to the access request into at least two pending tasks including a first pending task and a second pending task according to the analysis of the access request, and send the first pending task to the first large model or the data server; wherein the first large model is a remote large model;

[0089] An acquisition unit 202 is used to acquire multimodal access data corresponding to the first task to be processed, determined by the first large model or the data server according to the first task to be processed, and data of a synthesis method of the multimodal access data;

[0090] The generation unit 203 is used to use the synthesis method data and the multimodal page data as the second task to be processed, process the second task to be processed through the second large model deployed locally, and generate multimedia data corresponding to the access request; wherein the second large model is a local large model.

[0091] Wherein, the sending unit 201 includes: a first determining subunit and a second determining subunit;

[0092] The first determining subunit is used to determine the access page corresponding to the access request according to the analysis of the access request;

[0093] The second determining subunit is used to split the to-be-processed task generated according to the access page to determine the first to-be-processed task and the second to-be-processed task.

[0094] In this embodiment, the second determination subunit includes: determining the acquisition task for acquiring the multimodal page data in the access page as the first task to be processed; and determining the synthesis task for the multimodal page data in the access page as the second task to be processed.

[0095] The acquisition unit 202 includes: a first acquisition subunit and a second acquisition subunit;

[0096] The first acquisition subunit is used to acquire the multimodal subject object data and multimodal image material data corresponding to the access request, which are searched by the first large model or the data server according to the first task to be processed;

[0097] The second acquisition subunit is used to obtain the synthesis method data determined by the first large model or the data server according to the multimodal main data and the multimodal material data.

[0098] The generating unit 203 includes: a determining subunit and a generating subunit;

[0099] The determination subunit is used for the first end-side large model in the second large model deployed locally to determine the page reference data corresponding to the access request according to the requester feature data of the access request; specifically, the first end-side large model is used to identify the access intention of the requester according to the access request, and determine the target multimodal page data matching the access request from the multimodal page data;

[0100] The generating sub-unit is used for the second end-side large model in the second large model to synthesize the multimodal page data and the page reference data according to the synthesis method data to generate the multimedia data; specifically, the second end-side large model is used for synthesizing the target multimodal page data and the page reference data according to the synthesis method data to generate the multimedia data.

[0101] For the specific implementation process of the above-mentioned device embodiment, reference may be made to the contents of the above-mentioned method embodiment, which will not be described in detail here.

[0102] Based on the above content, the present application also provides a multimedia data processing system, such as Figure 3 As shown, the system may include: a client 301, a first large model or a data server 302, and a second large model 303 deployed by the client.

[0103] The client 301 is used to generate an access request according to a trigger of the application service platform by a user, and according to the analysis of the access request, split the pending task corresponding to the access request into at least two pending tasks including a first pending task and a second pending task, and send the first pending task to the first large model or data service end; obtain the multimodal access data corresponding to the first pending task determined by the first large model or data service end according to the first pending task, and the synthesis method data of the multimodal access data; use the synthesis method data and the multimodal page data as the second pending task, process the second pending task through the second large model, and generate multimedia data corresponding to the access request.

[0104] The first large model or data server 302 is used to search for multimodal access data corresponding to the access request and synthesis method data of the multimodal access data according to the first task to be processed, and send the multimodal access data and the synthesis method data to the second large model locally deployed on the client.

[0105] The second large model 303 is used to determine the page reference data corresponding to the access request according to the requester characteristic data of the access request in the client, and synthesize the multimodal page data and the page reference data according to the received synthesis method data to generate the multimedia data.

[0106] like Figure 4 As shown, Figure 4This is a timing diagram of an embodiment of a multimedia data processing method provided by the present application. The client generates an access request and splits it into a first task to be processed and a second task to be processed, and sends the first task to be processed to the first large model or data server. The first large model or data server searches for multimodal access data (multimodal subject object data and multimodal image material data) corresponding to the access request through the material center and / or the network based on the intention recognition of the first task to be processed (the intention recognition of the access request), wherein the multimodal subject object data is the relevant data of the main elements in the access page corresponding to the access request, and the multimodal image material data can be the relevant data used to supplement the description, modification, packaging, etc. of the subject object data. And the synthesis method data based on the relevant multimodal subject object data and multimodal image material data found. The first end-side large model 3031 in the second large model performs corresponding image and / or data processing according to the received multimodal subject object data, multimodal image material data, synthesis method data, etc., such as: recognition, extraction (cutout), etc. of the subject object, recognition, extraction, etc. of text features, matching processing of material data, etc., and determination of real-time personalized recommendation data, etc. The processed image and / or data and synthesis method data are sent to the second end-side large model 3032 in the second large model for synthesis processing to generate multimedia data, and then the synthesized multimedia data is sent to the first end-side large model for rendering, and output and display through the client. Finally, a multimodal product service page is displayed on the client, that is, a multimedia display page of dynamic product effects.

[0107] The above is a description of the execution timing process in combination with an embodiment of a multimedia data processing method and a multimedia data processing system provided in the present application. For specific related content, please refer to the description of the above processing method embodiment, which will not be described in detail here.

[0108] As mentioned above, the present application also provides a multimedia data interaction method, such as Figure 5 As shown, Figure 5 This is a flowchart of a multimedia data interaction method provided by the present application. The method includes:

[0109] Step S501: in response to an access request to an application service page, obtaining multimedia data corresponding to the access request determined by the multimedia data processing method; wherein the multimedia data includes: resource object subject information in the access page corresponding to the access request, and reference information synthesized with the resource object subject information and used to describe the resource object subject;

[0110] Step S502: In response to the acquired multimedia data, the multimedia data is rendered, and the resource object main body information and the reference information are output in a multimodal form in the access page.

[0111] In this embodiment, the description is mainly based on the display end of the application service page, that is, the client on the electronic device generates the access request, and based on the second largest model of the client, the multimedia data generated after the multimodal page data and the page reference data are synthesized according to the synthesis method data is rendered, and the multimedia data is output on the client. The multimedia data can output different contents according to the different data contents involved in the access page. In this embodiment, it mainly includes resource object subject information and reference information describing the resource object subject, such as: product A and discount information of product A. The client has been used as an example for description here. In actual scenarios, it is not limited to the client, and any method that can output and display the multimedia on the electronic device is acceptable.

[0112] In some embodiments, it may also include:

[0113] In response to triggering the reference information or the resource object body in the output multimedia information, outputting a reference page corresponding to the reference information;

[0114] In response to obtaining the reference data in the reference page, the reference information is bound to the access requester.

[0115] In this embodiment, when the reference information is preferential information, a preferential information page is output, and the user can obtain preferential data related to the resource object subject through the acquisition operation of the preferential information page, and bind the preferential data to the user, so as to facilitate the user to perform real-time calculation when obtaining the resource object subject. This embodiment can realize the interaction between the user and the page through the output multimedia data (that is, the interaction with the user is realized through the dynamic effect page output by the output end), facilitate the direct acquisition of data, and improve the user experience.

[0116] Accordingly, the present application also provides a multimedia data interaction device, such as Figure 6 As shown, the device comprises:

[0117] The acquisition unit 601 is used to respond to an access request to an application service page and acquire multimedia data corresponding to the access request determined by the processing method of the multimedia data; wherein the multimedia data includes: resource object body information in the access page corresponding to the access request, and reference information synthesized with the resource object body information and used to describe the resource object body.

[0118] The output unit 602 is used to render the multimedia data in response to the acquired multimedia data, and output the resource object main body information and the reference information in a multimodal form in the access page.

[0119] The specific implementation process of the device can be combined with the description of the above-mentioned multimedia data processing method and multimedia data interaction method, and will not be described in detail here.

[0120] Based on the above content, the present application also provides a computer storage medium for storing data generated by a network platform, and a program for processing the data generated by the network platform;

[0121] When the program is read and executed by the processor, the program executes the relevant contents of step S101 to step S103 in the above-mentioned multimedia data processing method.

[0122] Based on the above content, the present application also provides an electronic device, such as Figure 7 As shown, Figure 7 : is a schematic diagram of the structure of an electronic device provided by the present application, and the embodiment includes:

[0123] Processor 701;

[0124] The memory 702 is used to store a program for processing data generated by a network platform. When the program is read and executed by the processor, the program executes the relevant contents of steps S101 to S103 in the above-mentioned multimedia data processing method.

[0125] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0126] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0127] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0128] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0129] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0130] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A method for processing multimedia data, characterized in that: include: According to the analysis of the access request, split the pending task corresponding to the access request into at least two pending tasks including a first pending task and a second pending task, and send the first pending task to the first large model or the data service end; wherein the first large model is a remote large model; Acquiring the multimodal access data corresponding to the first task to be processed and the synthesis method data of the multimodal access data determined by the first large model or the data service end according to the first task to be processed; including: acquiring the multimodal subject object data and the multimodal image material data corresponding to the access request and searched by the first large model or the data service end according to the first task to be processed; acquiring the synthesis method data determined by the first large model or the data service end according to the multimodal subject data and the multimodal material data; The synthesis method data and the multimodal page data are taken as the second task to be processed, and the second task to be processed is processed by a locally deployed second large model to generate multimedia data corresponding to the access request; wherein the second large model is a local large model.

2. The method for processing multimedia data according to claim 1, characterized in that: The step of splitting the to-be-processed task corresponding to the access request into at least two to-be-processed tasks including a first to-be-processed task and a second to-be-processed task according to the parsing of the access request, and sending the first to-be-processed task to the first large model or the data server includes: Determining an access page corresponding to the access request based on the analysis of the access request; The to-be-processed task generated according to the access page is split to determine the first to-be-processed task and the second to-be-processed task.

3. The method for processing multimedia data according to claim 2, characterized in that: The step of splitting the to-be-processed task generated according to the access page to determine the first to-be-processed task and the second to-be-processed task includes: Determine an acquisition task for acquiring the multimodal page data in the accessed page as the first task to be processed; A task for synthesizing the multimodal page data in the accessed page is determined as the second task to be processed.

4. The method for processing multimedia data according to claim 1, characterized in that: The step of taking the synthesis method data and the multimodal page data as the second task to be processed, processing the second task to be processed by a second large model deployed locally, and generating multimedia data corresponding to the access request includes: The first end-side large model in the second large model deployed locally determines the page reference data corresponding to the access request according to the requester characteristic data of the access request; The second end-side large model in the second large model performs synthesis processing on the multimodal page data and the page reference data according to the synthesis method data to generate the multimedia data.

5. The method for processing multimedia data according to claim 4, characterized in that: The first end-side large model in the second large model deployed locally determines page reference data corresponding to the access request according to the requester feature data of the access request, including: The first end-side big model identifies the access intention of the requester according to the access request, and determines the target multimodal page data matching the access request from the multimodal page data; The second end-side large model in the second large model performs synthesis processing on the multimodal page data and the page reference data according to the synthesis method data to generate the multimedia data, including: The second end-side large model performs synthesis processing on the target multimodal page data and the page reference data according to the synthesis method data to generate the multimedia data.

6. A multimedia data processing system, characterized in that: include: A client, a second large model deployed in the client, and a first large model or a data server; wherein the first large model is a remote large model, and the second large model is a local large model of the client; The client is used to generate an access request according to a trigger of the application service platform by a user, and according to the analysis of the access request, split the pending task corresponding to the access request into at least two pending tasks including a first pending task and a second pending task, and send the first pending task to the first large model or data service end; obtain the multimodal access data corresponding to the first pending task and the synthesis method data of the multimodal access data determined by the first large model or data service end according to the first pending task; take the synthesis method data and the multimodal page data as the second pending task, process the second pending task through the second large model, and generate multimedia data corresponding to the access request; The first large model or data server is used to search for multimodal access data corresponding to the access request and synthesis method data of the multimodal access data according to the first task to be processed, and send the multimodal access data and the synthesis method data to the second large model locally deployed on the client; The second large model is used to determine the page reference data corresponding to the access request based on the requester characteristic data of the access request in the client, and to synthesize the multimodal page data and the page reference data according to the received synthesis method data to generate the multimedia data.

7. A multimedia data interaction method, characterized in that: include: In response to an access request to an application service page, obtaining multimedia data corresponding to the access request determined according to the multimedia data processing method according to any one of claims 1 to 5; wherein the multimedia data includes: resource object subject information in the access page corresponding to the access request, and reference information synthesized with the resource object subject information and used to describe the resource object subject; In response to the acquired multimedia data, the multimedia data is rendered, and the resource object main body information and the reference information are output in a multimodal form in the access page.

8. A computer storage medium for storing data generated by a network platform and a program for processing the data generated by the network platform; When the program is read and executed by a processor, the program executes the multimedia data processing method as described in any one of claims 1 to 5 above, or executes the multimedia data interaction method as described in claim 7 above.

9. An electronic device, comprising: processor; A memory for storing a program for processing data generated by a network platform, wherein when the program is read and executed by the processor, the program executes a method for processing multimedia data as described in any one of claims 1 to 5 above, or executes a method for interacting with multimedia data as described in claim 7 above.