Behavior scheme determination method and device based on large model, electronic equipment and medium

Abstract

The invention provides a behavior scheme determination method based on a large model, and relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like. According to the specific implementation scheme, a hybrid expert model is called based on features of multi-modal data, the hybrid expert model comprises a plurality of first large models for processing different modal data, and the features represent search words for performing in-library search on the first large models; inputting the multi-modal data into a hybrid expert model, and performing information extraction on each modal data to obtain text data, the text data including behavior information; inputting the text data into a second large model to obtain behavior content, and selecting and calling the second large model based on the category of the behavior information; and determining a behavior scheme by using the behavior content. The invention further provides a behavior scheme determination device based on the large model, electronic equipment and a storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the fields of computer vision, deep learning, large models, etc., and can be applied to multimodal data processing scenarios. More specifically, the present disclosure provides a method, device, electronic device, and storage medium for determining a behavior plan based on a large model. Background Art

[0002] With the development of artificial intelligence technology, information processing by machines is increasingly being used in all walks of life.

[0003] However, in some specific scenarios, such as advertising placement and marketing plan determination, plan planning and advertising placement are still mainly completed manually, which is time-consuming and has high labor costs. Summary of the invention

[0004] The present disclosure provides a method, apparatus, device and storage medium for determining a behavior plan based on a large model.

[0005] According to one aspect of the present disclosure, a method for determining a behavior plan based on a large model is provided, comprising: calling a hybrid expert model based on features of multimodal data, the hybrid expert model comprising a plurality of first large models for processing data of different modalities, the features representing search terms for in-library search of the first large models; inputting the multimodal data into the hybrid expert model, extracting information from each modal data to obtain text data, the text data comprising behavior information; inputting the text data into a second large model to obtain behavior content, the second large model being called based on category selection of the behavior information; and determining the behavior plan using the behavior content.

[0006] According to another aspect of the present disclosure, a device for determining a behavior plan based on a large model is provided, and the device includes: a calling module, which is used to call a hybrid expert model based on the features of multimodal data, the hybrid expert model includes multiple first large models that process data of different modalities, and the features represent search terms for performing in-library retrieval on the first large models; an extraction module, which is used to input the multimodal data into the hybrid expert model, extract information from each modal data respectively, and obtain text data, and the text data includes behavior information; a generation module, which inputs the text data into the second large model to obtain behavior content, the second large model is called based on category selection of the behavior information, and the behavior content is used to determine the behavior plan; and a determination module, which is used to determine the behavior plan using the behavior content.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0009] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the method provided according to the present disclosure is implemented.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 is a schematic diagram of an exemplary system architecture to which a method and apparatus for determining a behavior scenario based on a large model can be applied according to an embodiment of the present disclosure;

[0013] Figure 2 is a flow chart of a method for determining a behavior plan based on a large model according to an embodiment of the present disclosure;

[0014] Figure 3 is an architectural diagram of calling a hybrid expert model according to an embodiment of the present disclosure;

[0015] Figure 4 is an architectural diagram of a hybrid expert model according to an embodiment of the present disclosure;

[0016] Figure 5 is an architectural diagram of a hybrid expert model according to another embodiment of the present disclosure;

[0017] Figure 6 is an architectural diagram of a second large model according to an embodiment of the present disclosure;

[0018] Figure 7 is a flow chart for automatically labeling samples according to an embodiment of the present disclosure;

[0019] Figure 8 is a block diagram of a device for determining a behavior plan based on a large model according to an embodiment of the present disclosure;

[0020] Fig. 9 is a structural block diagram of an intelligent agent according to an embodiment of the present disclosure; and

[0021] Fig.10The invention is a block diagram of an electronic device to which a behavior scheme determination method based on a large model can be applied according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] A multimodal large language model (hereinafter referred to as a multimodal model) is a model that is trained by combining multiple modal information such as text, images, videos, and audio, and is capable of converting and generating multimodal data.

[0024] Multimodal models can include image-to-text models (such as image understanding, image-text question-and-answer, image creation, etc.), text-to-image models (such as image generation models, video generation models, etc.), text-to-speech (such as speech synthesis models), speech-to-text (such as speech recognition models), etc.

[0025] Visual multimodal models can perform image understanding, image-text question and answer, image-text creation, etc. based on the images and text information passed in by the user. Image understanding refers to generating descriptive text for images. In addition to asking what the image contains, image-text question and answer can also specifically ask what the name of the person in the image is, what the name of the building in the image is, what the name of the animal is, how many objects are in the image, etc. Image-text creation includes image creation (for example, generating a picture based on a text description, etc.) and text creation (for example, generating articles and poems based on images, etc.).

[0026] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0027] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0028] Figure 1 is a schematic diagram of an exemplary system architecture to which a method and device for determining a behavior plan based on a large model can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0029] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0030] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptops, desktop computers, etc.

[0031] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0032] It should be noted that the behavior solution determination method based on the big model provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the behavior solution determination device based on the big model provided in the embodiment of the present disclosure can generally be set in the server 105. The behavior solution determination method based on the big model provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the behavior solution determination device based on the big model provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0033] Figure 2 The present invention is a flowchart of a method for determining a behavior plan based on a large model according to an embodiment of the present invention.

[0034] like Figure 2 As shown, the large model-based behavior plan determination method 200 may include operations S210 to S240.

[0035] In operation S210, a hybrid expert model is called based on the features of the multimodal data, where the hybrid expert model includes a plurality of first large models for processing data of different modalities, and the features represent search terms for performing in-library search on the first large models.

[0036] The determination of the behavior plan disclosed herein may be, for example, the determination of a marketing plan. Multimodal data may include, for example, the collection of multiple modal data from social media, e-commerce platforms, and offline activities (such as exhibitions, roadshows, etc.), including text (user comments, product descriptions, etc.), images (product pictures, photos of the event site, etc.), videos (product advertisements, user unboxing videos, etc.), and audio (user feedback recordings, product introduction voice, etc.).

[0037] Hybrid expert models can be, for example, large text processing models: using large models such as BERT to specifically process text data and extract key information such as user sentiment and product features. Large image processing models: using convolutional neural networks such as ResNet or EfficientNet to analyze product appearance, user interaction scenarios, etc. in images. Large video processing models: combining 3D CNN and LSTM to capture dynamic information in videos, such as user behavior patterns and product usage effects. Large audio processing models: using WaveNet or automatic speech recognition (ASR) models to convert audio into text and extract features such as speech sentiment and speech speed.

[0038] Before processing the multimodal data, the present disclosure generates a unified feature vector according to the features of each modal data, and uses it as a search term to efficiently search in the library of the hybrid expert model to determine the large model corresponding to each modal data. The features of each modal data can be, for example, the type of multimodal data, such as text, picture, audio, etc. It can also be a task to be processed, such as a text processing task, a picture processing task, an audio processing task, etc.

[0039] In some embodiments, more detailed features can be used to retrieve more accurate large models. For example, "fashion elements" and "young group preferences" in pictures, "good reviews" and "negative feedback points" in texts, "user interaction time" and "interest point dwell time" in videos, etc. These are used as search terms to retrieve models with corresponding processing functions in the established model library.

[0040] In operation S220, the multimodal data is input into the hybrid expert model, and information is extracted from each modal data to obtain text data, where the text data includes behavior information.

[0041] The multimodal data is input into the hybrid expert model, and each model processes the data of the corresponding modality, extracts key information and integrates it into structured text data. Such text data includes, for example, user behavior preferences, product usage feedback, market trends and other marketing-related behavior information.

[0042] Before extracting key information from each modality, the collected data can be cleaned to remove noise and irrelevant information. Then, the corresponding large model is used to perform word segmentation, stop word removal, and stem extraction on the text; key frame extraction, object recognition, and scene classification on images and videos; and speech recognition and conversion of audio into text.

[0043] In operation S230, the text data is input into the second largest model to obtain the behavior content, and the second largest model is called based on the category selection of the behavior information.

[0044] According to the category of behavioral information in the text data (such as user interests, purchase intentions, product evaluations, etc.), different second-largest models (such as LSTM-based sentiment analysis models and graph neural network-based user portrait construction models) are intelligently selected or combined for processing.

[0045] Through the analysis of the second largest model, specific marketing behavior content is generated, such as target user group portraits, product selling point extraction, marketing activity suggestions (such as limited-time discounts, bundling sales, social media promotion strategies), etc.

[0046] In operation S240, a behavior scheme is determined using the behavior content.

[0047] Comprehensively analyze the behavioral content output by the second largest model, combine market trends, competitor strategies and corporate resources, and formulate a detailed marketing plan.

[0048] The plan may include: Target market positioning: clarify the target user group and its needs. Product strategy: optimize the product portfolio and highlight the differentiated advantages of products. Price strategy: set a reasonable price range and consider promotion and discount strategies. Channel strategy: select appropriate marketing channels, such as online e-commerce platforms, social media, offline stores, etc. Promotion strategy: formulate a specific promotion plan, including advertising, KOL cooperation, content marketing, etc.

[0049] Ultimately, by implementing this marketing plan, the goals of brand exposure, user growth and sales increase can be achieved.

[0050] According to the disclosed embodiment, through comprehensive analysis of multimodal data, the hybrid expert model is used to extract key features, and then the second largest model is used to intelligently select marketing strategies according to the behavior information category, and finally a targeted marketing plan is determined. This method not only improves the efficiency of generating marketing plans, but also greatly improves the scientificity and accuracy of decision-making, and enhances the targeting and effectiveness of marketing.

[0051] According to an embodiment of the present disclosure, calling a hybrid expert model based on the features of multimodal data includes: performing feature recognition on the multimodal data to obtain search terms corresponding to each modal data. According to the search terms, searching a vector library to determine multiple target data processing models, the vector library stores attribute information of multiple data processing models, and the attribute information includes model name, model function and model calling address. Calling multiple target data processing models and assembling them to obtain a hybrid expert model.

[0052] For example, in an implementation example of preparing to develop a marketing strategy for a newly opened coffee shop, multimodal data were collected, including pictures on social media (showing the store environment and coffee types), customer reviews (in text form, including taste evaluations and service feedback), and in-store surveillance videos (capturing customer behavior patterns).

[0053] Image recognition technology (such as CNN) can be used to extract features such as "modern minimalist style" and "hand-brewed coffee characteristics" from pictures as search terms for visual modality. Sentiment analysis and keyword extraction of text reviews can be performed to obtain "mellow taste" and "enthusiastic service" as search terms for text modality. Video data can be used through behavior recognition algorithms to extract features such as "long customer stay time" and "high interaction rate" as search terms for video modality.

[0054] The attribute information includes model name, model function and model calling address.

[0055] For example, there is a pre-built vector library that stores attribute information of various data processing models, including model name (such as "Sentiment Analysis Model V3"), model function (such as "Identify positive / negative sentiment in text"), and model call address (such as the URL of the network API).

[0056] Based on the search terms generated in the previous step, a search is performed in the vector library to match the "environmental style analysis model" suitable for processing visual modality, the "emotion and keyword analysis model" for processing text modality, and the "customer behavior pattern recognition model" for processing video modality.

[0057] After the target data processing models are determined, they can be integrated together through their respective call addresses (such as API interfaces) to form a hybrid expert model. This hybrid expert model can process multimodal data at the same time. First, each model is run in parallel to process the data of each modality, and then the processing results are aggregated into a central processing unit. In this unit, the processing results of each modality are comprehensively analyzed to form a comprehensive understanding of the overall market performance of the coffee shop.

[0058] Based on the output of the hybrid expert model, the main advantages of the coffee shop (such as unique decoration style, high-quality coffee taste) and potential improvement points (such as fewer customer interaction links) were identified. Subsequently, based on these analysis results, a set of targeted marketing strategies were formulated, including posting beautiful pictures showing the store environment on social media, using positive comments screened by the sentiment analysis model for word-of-mouth marketing, and designing activities to increase customer interaction (such as coffee tasting sessions and photo check-in areas) to improve customer engagement and loyalty.

[0059] According to the embodiments of the present disclosure, from feature recognition and search term generation of multimodal data, to vector library retrieval and selection of target data processing models, to model calling and assembly, a hybrid expert model that can process complex data and output valuable insights is finally formed. This method not only improves the efficiency and accuracy of data processing, but also makes data-based decision-making more scientific and comprehensive, providing strong support for formulating effective action plans.

[0060] According to an embodiment of the present disclosure, performing feature recognition on multimodal data to obtain search terms corresponding to each modal data includes at least one of the following operations: determining multiple data types in the multimodal data as search terms. Determining multiple content generation requirements obtained by classifying the multimodal data as search terms.

[0061] For example, when formulating a marketing strategy for electronic products on an e-commerce platform, multimodal data of the products was first collected, including product pictures (image data), product description copy (text data), and product instruction videos (video data, but this can also be regarded as a combination of voice and images).

[0062] In the feature recognition stage, the type of each data is first determined by rule judgment:

[0063] Product images are directly identified as "image data" and used as one of the search terms to retrieve image processing models, such as color analysis and style recognition.

[0064] For product description copy, it is identified as "text data" and used as a search term to retrieve text processing models, such as sentiment analysis and keyword extraction.

[0065] Although the product instruction video is complex, it can be split into voice track and image track through preprocessing, and identified as "voice data" and "image data" respectively, and used as search terms respectively.

[0066] You can also use a large model for model reasoning to extract deeper content generation requirements. For the above data:

[0067] For product images, the big model may infer the "brand information" (such as logo recognition) and "product categories" (such as mobile phones and laptops) that need to be refined. These also become search terms for more accurate retrieval of relevant processing models.

[0068] For product description copy, the big model may infer the "product selling points" (such as high-performance processors, ultra-long battery life) and "target groups" (such as game enthusiasts, business people) that need to be analyzed in detail, and use them as search terms to help select a more suitable text processing model.

[0069] For video data, the large model may not only identify key information in the voice (such as user comments and function introductions), but also identify the user's actual usage scenarios in combination with images, and further infer "usage scenario description" as search terms.

[0070] Figure 3 is an architecture diagram of calling a hybrid expert model according to an embodiment of the present disclosure.

[0071] like Figure 3 As shown, in the architecture 300 for calling the hybrid expert model, after feature recognition 312 is performed on the multimodal data 311, the search terms are obtained. According to the search terms generated above, the corresponding processing models are retrieved and called in the vector library 313, such as the image processing model for color matching analysis, the text processing model for selling point extraction and copywriting optimization, and the voice processing model for user feedback analysis. These models together constitute a hybrid expert model for comprehensive processing of multimodal data. The attribute information of the data processing model is stored in the vector library. The called data processing models 314 and 317 are activated and assembled into the hybrid expert model. The uncalled data processing models 315 and 316 remain unused.

[0072] In some embodiments, the retrieval of the vector library 313 may obtain information through RAG (Retrieval-Augmented Generation) retrieval enhancement technology to determine which appropriate data processing models to use for assembling the hybrid expert model.

[0073] The management of adding and deleting data processing models can also be achieved by adding attributes of new data processing models in the vector library 313 and deleting attribute information of existing data processing models.

[0074] Based on the output of the hybrid expert model, a targeted marketing strategy was developed, including optimizing product images to highlight brand characteristics and product highlights, adjusting copywriting to better attract the target audience, and adjusting product introductions and instructions based on user feedback videos to improve user experience and purchase conversion rates.

[0075] According to the embodiments of the present disclosure, rule determination can directly and quickly identify data types, while model reasoning can dig out deeper content generation needs. These two methods complement each other and provide accurate and comprehensive data support for building a hybrid expert model. This not only improves the efficiency of data processing and strategy formulation, but also ensures the effectiveness and pertinence of the strategy.

[0076] According to an embodiment of the present disclosure, searching a vector library according to a search term to determine multiple target data processing models includes: searching a vector library according to the search term to obtain multiple data processing models corresponding to each modal data, and the multiple data processing models have different scales. Determining the cost values ​​of the multiple data processing models based on the amount of resources required for each data processing model to process the corresponding modal data. Screening the multiple data processing models according to the cost values ​​to obtain multiple target data processing models.

[0077] For example, when developing a marketing plan for a newly launched smart home device, a large amount of multimodal data was collected, including product demonstration videos (video modality), user reviews (text modality), and product appearance pictures (image modality).

[0078] First, corresponding search terms were generated based on these data, such as “smart home device video”, “user review sentiment analysis”, and “product appearance style recognition”.

[0079] Next, we use these search terms to search the vector library. The vector library stores data processing models of various sizes, ranging from small and lightweight models to large and high-performance models. For example, for the video modality, we retrieved multiple video processing models, including a lightweight video summarization model, a medium-sized video content classification model, and a large video behavior analysis model.

[0080] After obtaining multiple data processing models corresponding to each modality data, it is necessary to evaluate the amount of resources required by these models to process the corresponding modality data to determine their cost values. The amount of resources may include computing resources (such as CPU and GPU usage time), memory usage, and possible additional costs (such as the cost of using third-party APIs).

[0081] For example, for the video modality, a lightweight video summary model may only require less computing resources and time, and the cost is low. However, a large video behavior analysis model, although powerful, may require more computing resources and time, and the cost is correspondingly higher. Similarly, for the text modality and image modality, the cost value of each model can also be evaluated separately.

[0082] After evaluating the cost values ​​of all candidate data processing models, screen them based on the project's budget and performance requirements. If the project budget is limited and real-time requirements are not high, a lightweight model with lower cost may be selected. If the project has high requirements for the accuracy and comprehensiveness of the results and the budget is sufficient, a large model with higher cost but better performance may be selected.

[0083] After screening, multiple target data processing models were identified that met the performance requirements of the project while meeting budget constraints. For example, for the video modality, a medium-sized video content classification model might be selected as the target model. For the text modality, a composite text processing model that can perform sentiment analysis and keyword extraction at the same time was selected. For the image modality, an image processing model that can recognize multiple product styles was selected.

[0084] According to the embodiments of the present disclosure, through the steps of retrieval, cost evaluation and screening, it is possible to find a combination of data processing models that meets project requirements and is economical and efficient. This not only improves the efficiency and accuracy of data processing, but also ensures that the formulation of action plans can achieve the best results with limited resources.

[0085] According to an embodiment of the present disclosure, the multimodal data includes, for example, at least one of original text data, voice data, and image data. The multimodal data is input into a hybrid expert model, and information is extracted from each modal data respectively, and the obtained text data includes at least one of the following: the original text data is input into a text processing model to obtain first text data. The voice data is input into a voice processing model to obtain second text data. The image data is input into an image processing model to obtain third text data.

[0086] For example, when developing a marketing strategy for a smart home brand, we collected multimodal data, including original text data of user reviews, voice data of product introduction videos, and image data of product display pictures. Our goal is to use this data to deeply understand user needs, product features, and market feedback, so as to develop an effective promotion strategy.

[0087] The original text data may be, for example, user reviews of smart home products collected from social media, e-commerce platforms and other channels. These reviews may include information such as user satisfaction with the product, functional requirements, and improvement suggestions.

[0088] The original text data is input into a text processing model (such as BERT, etc.). The model uses natural language processing technology to perform sentiment analysis, keyword extraction, topic identification and other operations on the text to obtain the first text data - that is, the processed text information, including user emotional tendencies, focus, etc.

[0089] For example, voice data may be extracted from product introduction videos, which are usually recorded by brands or professional reviewers to introduce product functions, advantages, and usage scenarios.

[0090] The voice data is input into a voice processing model (such as a voice recognition model + NLP model), voice recognition is first performed to convert the voice into text, and then natural language processing technology is applied again to analyze the converted text to obtain the second text data - that is, the text information corresponding to the voice content, including product function descriptions, usage suggestions, etc.

[0091] Image data may be, for example, collected product display pictures, including product appearance, usage scenarios, and matching with other household products.

[0092] The image data is input into image processing models (such as CNN, ResNet, etc.). These models can recognize objects, scenes, colors and other information in the image, and may convert the image content into text description through image description generation technology (such as image-to-text generation model) to obtain the third text data, that is, the textual representation of the image content, including product appearance features, usage scenario description, etc.

[0093] Integrate the text data obtained from the above three channels to form comprehensive market feedback and user portraits.

[0094] Figure 4 is an architectural diagram of a hybrid expert model according to an embodiment of the present disclosure.

[0095] like Figure 4 As shown, in the architecture 400 of the hybrid expert model of the present disclosure, the hybrid expert model is obtained by calling and assembling the vector library through LLMRoute (Large Language Model Routing) 4104. For example, the original text data 4101, speech data 4102 and image data 4103 are respectively input into the hybrid expert model, and processed by the corresponding text processing model 4105, speech processing model 4106 and image processing model 4107 to obtain the first text data 4108, the second text data 4109 and the third text data 4110.

[0096] Analyze user emotional tendencies, demand pain points, product advantages and disadvantages, combine market trends and competitive product analysis, and formulate targeted marketing strategies.

[0097] For example, we can optimize product features based on the functional requirements mentioned in user reviews. We can design more attractive product demonstration and promotion videos based on the usage scenarios in voice and image data. We can also adjust marketing language and promotion angles based on user emotional tendencies.

[0098] According to the embodiments of the present disclosure, by processing the original text data, voice data and image data separately and converting the processing results into a unified text data form, it is possible to understand market demand and user feedback more comprehensively. This method not only improves the efficiency and accuracy of data processing, but also provides strong support for formulating accurate and effective marketing strategies.

[0099] According to an embodiment of the present disclosure, the behavior plan determination method based on the large model also includes: before inputting the text data into the second large model, using the first large language model to fuse at least two of the first text data, the second text data and the third text data to obtain the fused text data.

[0100] For example, when developing a marketing strategy for a smart home appliance brand, we collected multimodal data from different channels, including user comments (text data), product demonstration videos (voice + image data), and pictures on social media (image data). Our goal is to use this data to deeply understand consumers' purchasing behavior, preferences, and market trends, so as to develop accurate and effective marketing strategies.

[0101] The user comments are input into the text processing model to obtain the first text data containing information such as user emotions and needs.

[0102] Extract speech from product demonstration videos and convert it into secondary text data containing information such as product features and advantages through speech recognition and NLP technology.

[0103] Perform image recognition and analysis on product images on social media, and convert image content into third-party text data containing information such as product appearance and usage scenarios through image description generation technology.

[0104] Figure 5 is an architectural diagram of a hybrid expert model according to another embodiment of the present disclosure.

[0105] like Figure 5 As shown, in another architecture 500 of a hybrid expert model of the present disclosure, for example, a first large language model 5108 may be used to fuse text data generated based on each modal data to obtain fused text data 5109 .

[0106] For example, the vector library is called and assembled through LLMRoute (Large Language Model Routing) 5104 to obtain a hybrid expert model. For example, the original text data 5101, speech data 5102 and image data 5103 are respectively input into the hybrid expert model, and processed through the corresponding text processing model 5105, speech processing model 5106 and image processing model 5107 to obtain the first text data, the second text data and the third text data.

[0107] Before the above-mentioned text data are input into the second largest model, the first largest language model 5108 (such as BERT, etc.) is used to fuse the first text data, the second text data, and the third text data. This fusion process is intended to integrate information from different channels, eliminate redundancy, and enhance the coherence and consistency of text data. For example, if the first text data mentions the user's energy-saving needs for smart home appliances, and the second text data details the energy-saving function of the product, the first largest language model 5108 may fuse these two parts of information to form a complete text that includes both user needs and product responses.

[0108] The fused text data is input into the second largest model, which extracts key behavioral content by analyzing the behavioral information in the text (such as purchase intention, usage habits, preference changes, etc.).

[0109] Based on these behaviors, specific marketing suggestions are further generated. For example, if the model identifies that a large number of users are interested in the remote control function of smart home appliances, the marketing plan may emphasize this feature of the product and launch corresponding promotional activities or experience activities. If the model finds that users have a high degree of attention to the appearance design of the product, it may recommend strengthening the publicity and promotion of the product appearance design.

[0110] According to the embodiments of the present disclosure, through the fusion processing of the first language model, product information from different channels and in different forms can be integrated into more complete and coherent text data. This not only improves the efficiency and accuracy of data processing, but also provides strong support for the second model to generate high-quality behavioral content.

[0111] According to an embodiment of the present disclosure, the second largest model includes, for example, a keyword extraction sub-model, a copy generation sub-model, and a picture generation sub-model. Inputting text data into the second largest model to obtain behavioral content includes: classifying text data according to the category of behavioral information to obtain multiple groups of classified texts, and the category of behavioral information includes generating keywords, generating copy, and generating pictures. Perform at least one of the following operations: input the classified text corresponding to the generated keywords into the keyword extraction sub-model to obtain keywords. Input the classified text corresponding to the generated copy into the copy generation sub-model to obtain the copy. Input the classified text corresponding to the generated picture into the picture generation sub-model to obtain the picture.

[0112] On the basis of further expanding the above embodiment, a more refined and personalized marketing plan generation method is proposed. The method refines the functions of the sub-models in the second model to better meet the needs of different marketing scenarios.

[0113] Figure 6is an architectural diagram of a second large model according to an embodiment of the present disclosure.

[0114] Similar to the above embodiment, multimodal data (such as user comments, social media posts, product images, etc.) are first collected, and feature extraction and information integration are performed through a hybrid expert model to obtain text data containing rich behavioral information.

[0115] Next, for example, natural language processing technology is used to classify the text data and divide it into multiple groups of classified texts according to the categories of behavioral information (generating keywords, generating copy, generating pictures). Figure 6 As shown, in the architecture 600 of the second largest model, the keyword extraction submodel 623, the copywriting generation submodel 624, and the image generation submodel 625 can also be called through LLMRoute (large language model routing) 622, and the classified texts corresponding to the generated keywords are respectively input into the keyword extraction submodel 623 to obtain keywords 626. This submodel extracts keywords that can represent product features, user interests or market trends through algorithms such as word frequency statistics, TF-IDF, Word2Vec, etc. These keywords will be used in scenarios such as search engine optimization and advertising.

[0116] The classified text corresponding to the generated copy is input into the copy generation sub-model 624 to obtain the copy 627. This sub-model automatically generates personalized marketing copy based on template filling, sequence-to-sequence generation or generative adversarial network (GAN) and other technologies, combined with information such as user portraits and product features. The copy may include product introductions, promotions, user reviews, etc., aiming to attract users' attention and stimulate their desire to buy.

[0117] And, the classified text corresponding to the generated image is input into the image generation sub-model 625 to obtain the image 628. This sub-model uses deep learning technology (such as convolutional neural network CNN, generative adversarial network GAN, etc.) to generate marketing images that match the brand image and product characteristics based on the text description. These images may include product display pictures, user scenario pictures, promotional posters, etc., to convey marketing information in a visual way.

[0118] Based on the behavioral content (keywords, copywriting, pictures) generated by the second largest model, combined with user portraits, market trends, competition analysis and other factors, formulate personalized marketing plans.

[0119] The marketing plan may include customized promotion strategies for different user groups, keyword-based search engine marketing, social media advertising, email marketing and other comprehensive channels.

[0120] According to the embodiments of the present disclosure, by further refining the functions of the second largest model, deep mining and personalized processing of multimodal data are achieved. The collaborative work of the keyword extraction sub-model, the copywriting generation sub-model, and the image generation sub-model not only improves the pertinence and attractiveness of marketing content, but also provides enterprises with a full range of personalized marketing solutions.

[0121] According to an embodiment of the present disclosure, determining a behavior solution using behavior content includes: calling a behavior solution creation service through a platform interface, and processing the behavior content using the behavior solution creation service to obtain a behavior solution.

[0122] In this embodiment, an automated marketing plan generation system that integrates large model processing capabilities and platform interface calls is pre-built. The system can automatically extract key information from multimodal data and call professional behavior plan creation services through the platform interface to quickly generate marketing plans that meet the needs of the enterprise.

[0123] Similar to the above embodiment, the system first collects multimodal data and processes it through a hybrid expert model and a second large model (eg, including a keyword extraction sub-model, a text generation sub-model, and an image generation sub-model).

[0124] The various models work together to extract and classify information from multimodal data and generate text data containing behavioral content such as keywords, copywriting, and images.

[0125] The system has built-in interfaces with marketing plan creation service platforms. These platforms usually provide a wealth of marketing templates, strategy libraries, and data analysis tools.

[0126] Through the platform interface, the system takes the generated behavioral content (keywords, text, pictures, etc.) as input and calls the behavioral plan creation service.

[0127] The behavior plan creation service automatically generates personalized marketing plans based on the input behavior content and combines the platform's strategy library, user portraits, market trends and other data.

[0128] After receiving the marketing plan returned from the platform, the system can also conduct preliminary inspection and evaluation. If it is found that there are contents in the plan that do not meet the requirements of the enterprise or market rules, the system will make necessary adjustments and optimizations.

[0129] The optimized marketing plan will be output to the user in an editable format (such as Word document, PPT presentation, etc.) to facilitate further review and modification.

[0130] At the same time, the system can also provide execution monitoring and effect evaluation functions of marketing plans, helping users to grasp the progress and effects of marketing activities in real time.

[0131] According to the embodiments of the present disclosure, an efficient and automated marketing plan generation system is constructed by combining large model processing capabilities and platform interface calling technology. The system can automatically extract key information from multimodal data and quickly generate personalized marketing plans through professional behavior plan creation services. This approach not only improves the efficiency and accuracy of marketing plan generation, but also reduces labor costs and time costs.

[0132] According to an embodiment of the present disclosure, the behavior plan determination method based on a large model also includes: using automatically labeled sample training to obtain at least one of the multiple data processing models.

[0133] In this embodiment, the content of the above embodiment is further expanded, and a method of automatically labeling sample training data processing models is introduced. This improvement makes the construction and update of the data processing model more efficient and accurate, thereby improving the intelligence level of the entire marketing plan determination process.

[0134] For example, first collect a large amount of raw data as candidate sets for training samples. These data may come from various channels, such as user behavior logs, market research reports, social media content, etc.

[0135] Next, the candidate set is preliminarily screened and preprocessed using existing data processing models or predefined rules to remove noise and irrelevant information.

[0136] Then, automatic labeling techniques (such as unsupervised learning, semi-supervised learning or weakly supervised learning algorithms) are used to label the processed samples. The labeling content may include key information such as data categories, attributes, and relationships.

[0137] Using the sample set obtained by automatic annotation, at least one of the multiple data processing models is trained. These models may include text classification models, image recognition models, sentiment analysis models, etc., which are used to process data of different modalities.

[0138] During the training process, the model parameters and structure are continuously optimized to improve its data processing capabilities and accuracy. Through iterative training and verification, the system gradually determines the optimal model configuration.

[0139] Similar to the above embodiment, according to the feature recognition results of the multimodal data and the vector library retrieval results, the trained data processing model is dynamically called and assembled into a hybrid expert model.

[0140] The hybrid expert model can comprehensively process data from multiple modalities and extract key behavioral information, providing strong support for the subsequent determination of marketing plans.

[0141] Based on the behavioral information generated by the hybrid expert model, behavioral content (such as keywords, text, pictures, etc.) is further generated through the second largest model and other related models.

[0142] Finally, the behavior plan creation service is called through the platform interface, and factors such as behavior content and market strategy are combined to generate a marketing plan that meets the needs of the enterprise.

[0143] According to the embodiments of the present disclosure, by introducing a method for automatically labeling samples to train a data processing model, the intelligence level of the marketing plan determination process is further improved. Automatic labeling technology can efficiently generate a large number of labeled samples, providing rich data resources for the training of data processing models. These trained models have higher processing power and accuracy, and can better adapt to the complex and changing market environment.

[0144] According to an embodiment of the present disclosure, the automatically annotated samples are determined in the following manner: performing at least one of regular expression determination and business code rule determination on the original training text to obtain structured text. Clustering the structured text according to semantics to obtain multiple data clusters. Extracting a preset number of target structured texts from each data cluster to obtain clustered samples. Using multiple second-largest language models to annotate the clustered samples respectively to obtain annotated text. Performing group decision scoring on the annotated text, and determining the automatically annotated samples based on the average score.

[0145] In this embodiment, the process of determining the automatically labeled samples is further refined, and more accurate and reliable labeled samples are generated through a series of complex text processing, cluster analysis and group decision scoring mechanisms. These samples are then used to train the data processing model, thereby improving the performance of the hybrid expert model and ultimately optimizing the marketing plan formulation process.

[0146] Figure 7 The present invention is a flowchart of automatically labeling samples according to an embodiment of the present invention.

[0147] For example, the original training text is first preprocessed, including steps such as noise removal, word segmentation, part-of-speech tagging, etc. The original training text may be, for example, a conversation record.

[0148] Then, if Figure 7 As shown, in the process 700 of automatically annotating samples, at least one of regular expression determination 702 and business code rule determination 703 is performed on the conversation record 701. Regular expression determination 702 uses regular expressions to match specific patterns in the text. Business code rule determination 703 screens and converts the text according to the preset business logic. These two operations act together on the original text to generate structured text.

[0149] The structured text is clustered according to semantics, and algorithms such as K-means 704 and DBSCAN 705 are used in combination with a large language model 706 to group similar texts into the same data cluster. For example, the number of piles to be classified is calculated by the DBSCAN 705 algorithm, and then the original training text is divided into piles using the K-means 704 algorithm. In the case of a large amount of data, the large language model 706 can also be used to divide the original training text into piles. For clustering structured text, the amount of data can be reduced by adjusting the compression ratio and entropy value 707 to obtain clustered data 708.

[0150] In order to ensure the diversity and representativeness of the labeled samples, a preset number of target structured texts are extracted from each data cluster to form the above-mentioned cluster data 708.

[0151] Multiple second largest language models are used to annotate cluster data 708 in parallel. Each model provides different annotation results for the sample based on its unique language understanding and generation capabilities. For example, the first annotation model 709, the second annotation model 710, the third annotation model 711, the fourth annotation model 712 and other large models can be used to annotate cluster data 708 to obtain annotated text 713.

[0152] In order to integrate the annotation results of multiple models, a group decision scoring mechanism is adopted. Specifically, the system scores the multiple annotation results of each sample and determines the final automatically annotated sample based on the average score. This process helps to eliminate the bias and noise of single model annotation and improve the accuracy and reliability of annotation.

[0153] Using the automatically annotated sample sets, multiple data processing models are trained. After optimization, these models can better process multimodal data and extract key information.

[0154] Then, according to the feature recognition results of multimodal data and the vector library retrieval results, the trained data processing model is dynamically called and assembled into a hybrid expert model. This model can comprehensively process data of multiple modalities and generate behavioral information for marketing plan formulation.

[0155] Based on the behavioral information and other relevant data (such as market trends, user portraits, etc.) generated by the hybrid expert model, the behavioral plan creation service is called through the platform interface to generate a marketing plan that meets the needs of the enterprise.

[0156] According to the embodiments of the present disclosure, by introducing a sophisticated automatic labeling sample determination process, the training quality of the data processing model and the performance of the hybrid expert model are further improved. The steps of structured text generation, text clustering and sample extraction, multi-model labeling and group decision scoring work together on the original training text to generate more accurate and reliable labeling samples. These samples provide strong support for the training of the data processing model, enabling the model to better adapt to the complex and changing market environment.

[0157] Figure 8 It is a block diagram of a device for determining a behavior plan based on a large model according to an embodiment of the present disclosure.

[0158] like Figure 8 As shown, the large model-based behavior plan determination device 800 may include a calling module 810 , an extracting module 820 , a generating module 830 and a determining module 840 .

[0159] The calling module 810 is used to call the hybrid expert model based on the features of the multimodal data. The hybrid expert model includes a plurality of first large models for processing data of different modalities. The features represent search terms for performing in-library search on the first large models.

[0160] The extraction module 820 is used to input the multimodal data into the hybrid expert model, extract information from each modal data, and obtain text data, which includes behavior information.

[0161] The generation module 830 is used to input text data into the second largest model to obtain behavior content. The second largest model is called based on the category selection of behavior information, and the behavior content is used to determine the behavior plan.

[0162] The determination module 840 is used to determine the behavior plan using the behavior content.

[0163] The calling module 810 includes an identification submodule, a retrieval submodule and an assembly submodule.

[0164] The recognition submodule is used to perform feature recognition on multimodal data to obtain search terms corresponding to each modal data.

[0165] The search submodule is used to search the vector library according to the search words to determine multiple target data processing models. The vector library stores the attribute information of multiple data processing models, and the attribute information includes the model name, model function and model calling address.

[0166] The assembly submodule is used to call multiple target data processing models and assemble them to obtain a hybrid expert model.

[0167] The identification submodule includes at least one of a determination unit and an inference unit.

[0168] The determination unit is used to perform rule determination on the multimodal data to determine multiple data types in the multimodal data.

[0169] The reasoning unit is used to perform model reasoning on the multimodal data to determine multiple content generation requirements obtained by classifying the multimodal data.

[0170] The retrieval submodule includes a retrieval unit, a determination unit and a screening unit.

[0171] The retrieval unit is used to search the vector library according to the search words to obtain multiple data processing models corresponding to each modal data, and the multiple data processing models have different scales;

[0172] The determining unit is used to determine the cost values ​​of the plurality of data processing models based on the amount of resources required for each data processing model to process the corresponding modal data;

[0173] The screening unit is used to screen multiple data processing models according to the cost value to obtain multiple target data processing models.

[0174] According to an embodiment of the present disclosure, the multimodal data includes at least one of original text data, voice data and image data, and the extraction module 820 includes at least one of a first generation submodule, a second generation submodule and a third generation submodule.

[0175] The first generating submodule is used to input the original text data into the text processing model to obtain the first text data.

[0176] The second generating submodule is used to input the speech data into the speech processing model to obtain second text data.

[0177] The third generating submodule is used to input the image data into the image processing model to obtain the third text data.

[0178] The extraction module 820 also includes a fusion submodule.

[0179] The fusion submodule is used to use the first large language model to fuse at least two of the first text data, the second text data and the third text data before inputting the text data into the second large model to obtain fused text data.

[0180] According to an embodiment of the present disclosure, the second largest model includes a keyword extraction sub-model, a text generation sub-model and a picture generation sub-model.

[0181] The generation module 830 includes a classification submodule, and at least one of a fourth generation submodule, a fifth generation submodule, and a sixth generation submodule.

[0182] The classification submodule is used to classify text data according to the categories of behavioral information to obtain multiple groups of classified texts. The categories of behavioral information include generating keywords, generating copy, and generating pictures.

[0183] The fourth generation submodule is used to input the classified text corresponding to the generated keywords into the keyword extraction submodel to obtain the keywords;

[0184] The fifth generation submodule is used to input the classified text corresponding to the generated copy into the copy generation submodel to obtain the copy;

[0185] The sixth generation submodule is used to input the classified text corresponding to the generated image into the image generation submodel to obtain the image.

[0186] The determination module 840 includes a calling submodule and a seventh generating submodule.

[0187] The calling submodule is used to call the behavior scheme creation service through the platform interface;

[0188] The seventh generation submodule is used to process the behavior content using the behavior plan creation service to obtain a behavior plan.

[0189] The large model-based behavior scenario determination device 800 further includes a training module 850 .

[0190] The training module 850 is used to obtain at least one of the multiple data processing models by training with automatically labeled samples.

[0191] The training module 850 includes a determination submodule, a clustering submodule, an extraction submodule, a labeling submodule and a decision submodule.

[0192] The determination submodule is used to perform at least one operation of regular determination and business code rule determination on the original training text to obtain a structured text.

[0193] The clustering submodule is used to cluster structured text according to semantics to obtain multiple data clusters.

[0194] The extraction submodule is used to extract a preset number of target structured texts from each data cluster to obtain cluster samples.

[0195] The annotation submodule is used to use multiple second-largest language models to respectively annotate cluster samples to obtain annotated texts.

[0196] The decision submodule is used to make group decision scores on the annotated text and determine the samples to be automatically annotated based on the average score.

[0197] According to an embodiment of the present disclosure, the present disclosure also provides an artificial intelligence agent configured to execute the large model-based behavior plan determination method according to any embodiment of the present disclosure.

[0198] Fig. 9 It is a structural block diagram of an intelligent agent according to an embodiment of the present disclosure.

[0199] In the embodiments of the present disclosure, inspired by the von Neumann structure in modern computer theory, such as Fig. 9 As shown, the AI ​​agent 900 may include five core modules: an input module 910 , a control module 920 , a storage module 930 , a calculation module 940 and an output module 950 .

[0200] The input module 910 is responsible for receiving or sensing information such as queries, requests, instructions, signals or data from the outside world (such as users or external environment), and converting them into a format that can be understood and processed by the AI ​​agent 900. The input module 910 is the primary link for the AI ​​agent 900 to interact with the outside world. It enables the AI ​​agent 900 to efficiently and accurately obtain the necessary "sensory" information from the outside world and respond to this information.

[0201] In an example, the input module 910 may input the multimodal data described above.

[0202] In the example, the control module 920 is the core support for the AI ​​agent 900 to handle complex tasks. The control module 920 can execute the behavior plan determination method based on the large model described above.

[0203] In the example, the control module 920 will continuously interact with the storage module 930, the computing module 940 and / or the output module 950 during operation. However, it should be noted that in the embodiment of the present disclosure, the control module 920 acts as a single initiator to initiate communication with the storage module 930, the computing module 940 and / or the output module 950, and there is no communication coupling between the storage module 930, the computing module 940 and the output module 950.

[0204] In the example, the performance of the control module 920 may be closely related to the large model on which the AI ​​agent 900 is based. In order to give full play to the capabilities of the large language model, the internal structure of the control module 920 may be designed to be highly configurable and scalable to cope with various types of tasks and requirements in real scenarios.

[0205] The storage module 930 may be responsible for memorizing information such as historical conversations, event flows, etc. The text data and behavior content as described above may be included in the storage module 930 .

[0206] In the example, after acquiring the multimodal data, the AI ​​agent 900 can retrieve the relevant prompt words and data resources from the storage module 930 and feed them back to the control module 920. Then, the control module 920 can use the feedback prompt words and data resources to obtain text data and pass the text data to the output module 950.

[0207] The operation module 940 can be regarded as a predefined tool library. The tools obtained by encapsulating the first large model can be included in the operation module 940.

[0208] In the example, when the AI ​​agent 900 needs to respond to multimodal data, the relevant tool encapsulating the corresponding first large model can be called from the operation module 940 and fed back to the control module 920. Then, the control module 920 can use the called tool to obtain text data and pass it to the output module 950. It can be understood that although the large language model has excellent language understanding and generation capabilities, it is like a human being, without the help of any tools, and the tasks that can be solved are very limited. When the AI ​​agent 900 is given the ability to call tools, more complex tasks can be achieved.

[0209] The AI ​​agent 900 according to the embodiment of the present disclosure can simply and effectively improve the level of intelligence, and improve flexibility and versatility.

[0210] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0211] Fig.10 It is a block diagram of an electronic device to which a large model-based behavior scenario determination method can be applied according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0212] like Fig.10As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 to a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0213] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0214] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a text processing method and / or a deployment method of a deep learning framework. For example, in some embodiments, the text processing method and / or the deployment method of a deep learning framework may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into RAM 1003 and executed by computing unit 1001, one or more steps of the text processing method and / or the method for deploying a deep learning framework described above may be performed. Alternatively, in other embodiments, computing unit 1001 may be configured to execute the text processing method and / or the method for deploying a deep learning framework in any other appropriate manner (e.g., by means of firmware).

[0215] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0216] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0217] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories, read-only memories, erasable programmable read-only memories (EPROM) or flash memories, optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0218] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) display or a liquid crystal display (LCD)) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0219] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a Local Area Network (LAN), a Wide Area Network (WAN), and the Internet.

[0220] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0221] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0222] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for determining a behavior plan based on a large model, comprising: Calling a hybrid expert model based on the features of the multimodal data, the hybrid expert model comprising a plurality of first large models for processing data of different modalities, the features representing search terms for performing in-library search on the first large models; Inputting the multimodal data into the hybrid expert model, extracting information from each modal data respectively, and obtaining text data, wherein the text data includes behavior information; Inputting the text data into a second large model to obtain behavior content, the second large model being invoked based on the category selection of the behavior information; as well as The behavior content is used to determine a behavior plan.

2. The method according to claim 1, wherein: The feature calling hybrid expert model based on multimodal data includes: Performing feature recognition on the multimodal data to obtain search terms corresponding to each modal data; According to the search term, searching a vector library to determine a plurality of target data processing models, the vector library storing attribute information of the plurality of data processing models, the attribute information including model name, model function and model calling address; and The multiple target data processing models are called and assembled to obtain the hybrid expert model.

3. The method according to claim 2, wherein: The performing feature recognition on the multimodal data to obtain a search term corresponding to each modal data comprises at least one of the following operations: determining a plurality of data types in the multimodal data as the search terms; A plurality of content generation requirements obtained by classifying the multimodal data are determined as the search terms.

4. The method according to claim 2, wherein: The step of searching the vector library according to the search term to determine a plurality of target data processing models comprises: According to the search term, a vector library is searched to obtain a plurality of data processing models corresponding to each modal data, wherein the plurality of data processing models have different scales; Determining cost values ​​of the plurality of data processing models based on the amount of resources required for each of the data processing models to process the corresponding modal data; The multiple data processing models are screened according to the cost values ​​to obtain the multiple target data processing models.

5. The method according to any one of claims 1 to 4, wherein: The multimodal data includes at least one of original text data, voice data and image data. The multimodal data is input into the hybrid expert model, and information is extracted from each modal data respectively to obtain text data including at least one of the following: Inputting the original text data into a text processing model to obtain first text data; Inputting the speech data into a speech processing model to obtain second text data; The image data is input into an image processing model to obtain third text data.

6. The method according to claim 5, further comprising: Before the text data is input into the second large model, the first large language model is used to perform fusion processing on at least two of the first text data, the second text data and the third text data to obtain fused text data.

7. The method according to claim 1, wherein: The second largest model includes a keyword extraction sub-model, a text generation sub-model and an image generation sub-model; The text data is input into the second largest model to obtain the behavior content including: Classifying the text data according to the categories of the behavior information to obtain multiple groups of classified texts, wherein the categories of the behavior information include generating keywords, generating text, and generating pictures; and Do at least one of the following: Inputting the classified text corresponding to the generated keywords into the keyword extraction sub-model to obtain keywords; Inputting the classified text corresponding to the generated copy into the copy generation sub-model to obtain the copy; The classified text corresponding to the generated image is input into the image generation sub-model to obtain the image.

8. The method according to claim 1, wherein: Determining the behavior plan by using the behavior content includes: Call the behavior plan creation service through the platform interface; The behavior content is processed using the behavior plan creation service to obtain the behavior plan.

9. The method according to claim 2, wherein: The method further comprises: At least one of the multiple data processing models is obtained by training with automatically labeled samples.

10. The method according to claim 9, wherein: The automatically labeled samples are determined in the following manner: Perform at least one of regular expression determination and business code rule determination on the original training text to obtain a structured text; Clustering the structured text according to semantics to obtain multiple data clusters; Extracting a preset number of target structured texts from each of the data clusters to obtain cluster samples; Using multiple second largest language models to respectively annotate the cluster samples to obtain annotated texts; as well as The annotated texts are scored by group decision, and the automatically annotated samples are determined according to the average score.

11. A device for determining a behavior plan based on a large model, comprising: A calling module, configured to call a hybrid expert model based on the features of the multimodal data, wherein the hybrid expert model includes a plurality of first large models for processing data of different modalities, and the features represent search terms for performing in-library search on the first large models; An extraction module, used for inputting the multimodal data into the hybrid expert model, extracting information from each modal data respectively, and obtaining text data, wherein the text data includes behavior information; A generating module, inputting the text data into a second large model to obtain behavior content, wherein the second large model is selected and called based on the category of the behavior information, and the behavior content is used to determine the behavior plan; as well as A determination module is used to determine a behavior plan using the behavior content.

12. The device according to claim 11, wherein The calling module includes: An identification submodule, used to perform feature recognition on the multimodal data to obtain search terms corresponding to each modal data; a search submodule, configured to search a vector library according to the search term to determine a plurality of target data processing models, wherein the vector library stores attribute information of the plurality of data processing models, wherein the attribute information includes a model name, a model function, and a model calling address; and The assembling submodule is used to call the multiple target data processing models and assemble them to obtain the hybrid expert model.

13. The device according to claim 12, wherein: The identification submodule includes at least one of the following units: a determination unit, configured to perform rule determination on the multimodal data to determine a plurality of data types in the multimodal data; An inference unit is used to perform model inference on the multimodal data to determine a plurality of content generation requirements obtained by classifying the multimodal data.

14. The device according to claim 12, wherein: The retrieval submodule includes: A retrieval unit, configured to search the vector library according to the retrieval term to obtain a plurality of data processing models corresponding to each modal data, wherein the plurality of data processing models have different scales; A determining unit, configured to determine cost values ​​of the plurality of data processing models based on the amount of resources required for each of the data processing models to process the corresponding modal data; A screening unit is used to screen the multiple data processing models according to the cost value to obtain the multiple target data processing models.

15. The device according to any one of claims 11 to 14, wherein: The multimodal data includes at least one of original text data, voice data and image data, and the extraction module includes at least one of the following: A first generating submodule, used for inputting the original text data into a text processing model to obtain first text data; A second generating submodule, used for inputting the speech data into a speech processing model to obtain second text data; The third generating submodule is used to input the image data into the image processing model to obtain third text data.

16. The device according to claim 15, wherein: The extraction module also includes: The fusion submodule is used to use the first large language model to fuse at least two of the first text data, the second text data and the third text data before inputting the text data into the second large model to obtain fused text data.

17. The device according to claim 11, wherein: The second largest model includes a keyword extraction sub-model, a text generation sub-model and an image generation sub-model; The generation module comprises: a classification submodule, configured to classify the text data according to the categories of the behavior information to obtain a plurality of groups of classified texts, wherein the categories of the behavior information include generating keywords, generating text, and generating pictures; and At least one of the following submodules: a fourth generation submodule, configured to input the classified text corresponding to the generated keyword into the keyword extraction submodel to obtain the keyword; A fifth generation submodule, configured to input the classified text corresponding to the generated copy into the copy generation submodel to obtain the copy; The sixth generation submodule is used to input the classified text corresponding to the generated image into the image generation submodel to obtain the image.

18. The device according to claim 11, wherein The determination module comprises: The calling submodule is used to call the behavior scheme creation service through the platform interface; The seventh generation submodule is used to process the behavior content using the behavior plan creation service to obtain the behavior plan.

19. The device according to claim 12, wherein: The device also includes: A training module is used to obtain at least one of the multiple data processing models by training with automatically labeled samples.

20. The device according to claim 19, wherein The training module includes: A determination submodule, used to perform at least one of regular expression determination and business code rule determination on the original training text to obtain a structured text; A clustering submodule, used for clustering the structured text according to semantics to obtain multiple data clusters; An extraction submodule, used to extract a preset number of target structured texts from each of the data clusters to obtain cluster samples; a labeling submodule, used to use a plurality of second largest language models to label the cluster samples respectively to obtain labeled texts; and The decision submodule is used to perform group decision scoring on the annotated text and determine the automatically annotated samples according to the average score.

21. An artificial intelligence agent configured to execute the method according to any one of claims 1 to 10.

22. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

23. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

24. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images