Scene prediction method and device based on large language model

Abstract

The invention discloses a scene prediction method and device based on a large language model, relates to the technical field of artificial intelligence and can also be used in the financial field, and the method comprises the following steps: inputting an obtained cue word sequence into a pre-established large language model, and outputting a hidden state matrix of each first Transform layer; performing full-link mapping on the second hidden state matrix by using the first multi-layer perceptron module to obtain a general predicted value; inputting the first hidden state matrix into a pre-established scene model to obtain a prediction feature matrix; inputting the prediction feature matrix and the second hidden state matrix into a second multi-layer perceptron module of the scene model for full-link mapping to obtain a scene prediction value; and obtaining a prediction result based on the general prediction value and the scene prediction value. Through a cross-layer semantic fusion structure and a scene exclusive feature modeling mechanism, unified modeling of cross-scene semantic representation and refined extraction of specific features of a target scene are realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and can also be used in the financial field, particularly to a scene prediction method and apparatus based on a large language model. Background Technology

[0002] In local lifestyle internet applications, multiple online marketing scenarios typically coexist, including merchant recommendations, coupon distribution, search suggestions, and news feeds. The types of candidate materials differ significantly across these scenarios, including merchant information, coupons, product details, short text search terms, and image / text or video content. User behaviors also vary across scenarios. For example, in a food merchant scenario, users might follow, comment, pay, or contact the merchant. In a news feed scenario, users might subscribe, like, view details, or click on featured content. Existing personalized ranking tasks generally use click-through rate prediction models as the primary modeling method. However, due to significant differences in data scale, user behavior sparsity, and material types across different scenarios, traditional modeling strategies based on a single feature structure or uniform data distribution struggle to adapt effectively to various scenarios, thus failing to meet the ranking requirements of diverse scenarios.

[0003] Existing technologies primarily employ two modeling approaches for multi-scenario recommendation or ranking tasks. One approach trains an independent click-through rate (CTR) prediction model for each scenario, relying solely on scenario-specific data for training. While adaptable to different scenarios, this approach fails to leverage behavioral correlations across scenarios, making it difficult to establish cross-scenario user interest relationships. It also leads to redundant data processing and model structure development, increasing model development and maintenance costs. The other approach involves simply fusing data from multiple scenarios, using methods like one-hot encoding to fuse discrete features, and then training the model using a unified CTR prediction model. However, one-hot encoding struggles to represent potential semantic relationships between materials, and in cases of uneven data distribution, scenarios with rich user behavior have a stronger impact on model parameter updates, thus weakening the learning ability in scenarios with sparse behavior.

[0004] In recent years, some technical solutions have attempted to introduce large language models into multi-scenario fusion tasks, but they usually only use the semantic information of the last layer of the large language model and fail to make full use of the multi-dimensional semantic features contained in different network layers of the large language model.

[0005] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0006] This invention provides a scene prediction method based on a large language model to address issues in existing multi-scene prediction technologies, such as difficulty in sharing cross-scene features, insufficient learning ability for weak scenes, and inadequate utilization of deep semantic information from large language models. By introducing a unified prompt word construction mechanism, a cross-layer semantic fusion structure, and a scene-specific feature modeling mechanism, unified modeling of cross-scene semantic representation and refined extraction of target scene-specific features are achieved, thereby improving the accuracy, stability, and generalization ability of multi-scene prediction tasks.

[0007] This scene prediction method based on a large language model includes:

[0008] The acquired prompt word sequence is input into a pre-established large language model, and the hidden state matrix of each first Transformer layer is output; wherein, the prompt word sequence is obtained by data preprocessing based on the acquired scene information, user information and material information; the hidden state matrix includes a first hidden state matrix and a second hidden state matrix output by the last first Transformer layer;

[0009] The second hidden state matrix is ​​fully mapped using the first multilayer perceptron module to obtain a general prediction value.

[0010] The first hidden state matrix is ​​input into the pre-established scene model to obtain the prediction feature matrix;

[0011] The predicted feature matrix and the second hidden state matrix are input into the second multilayer perceptron module of the scene model for fully linked mapping to obtain the scene prediction value;

[0012] The prediction result is obtained based on the general prediction value and the scenario prediction value.

[0013] In some embodiments, data preprocessing is performed based on the acquired scene information, user information, and material information to obtain a prompt word sequence, including:

[0014] Obtain the scene information, user information, and material information;

[0015] Fields are extracted from the scene information, user information, and material information to obtain general fields;

[0016] The general fields are processed according to a preset prompt word template to obtain a prompt word sequence.

[0017] In some embodiments, the step of inputting the acquired prompt word sequence into a pre-established large language model and outputting the hidden state matrix of each first Transformer layer includes:

[0018] The prompt word sequence is segmented to obtain multiple word groups and the length of the prompt word sequence;

[0019] Based on the vocabulary of the large language model, generate integer numbers corresponding to each word group;

[0020] The integer numbers are vectorized to obtain the embedding vectors corresponding to each word group;

[0021] Each of the embedding vectors is input into the first Transformer layer of the large language model for iterative semantic encoding processing to generate the hidden state matrix.

[0022] In some embodiments, the step of using a first multilayer perceptron module to perform a fully linked mapping on the second hidden state matrix to obtain a general prediction value includes:

[0023] An embedding vector at a preset position is selected from the hidden state matrix to obtain a sample-level hidden vector;

[0024] The sample-level hidden vectors are subjected to linear transformation and nonlinear activation processing to obtain feature scalars;

[0025] The feature scalar is normalized using an activation function to obtain a general prediction value.

[0026] In some embodiments, the method further includes: setting gating modules in the scene model according to preset frequency parameters; each gating module corresponds one-to-one with at least a portion of the first Transformer layer of the large language model.

[0027] In some embodiments, inputting the first hidden state matrix into a pre-established scene model to obtain a predicted feature matrix includes:

[0028] The first hidden state matrix output by the corresponding first Transformer layer is weighted by each of the gating modules to obtain the weight coefficients corresponding to the first hidden state matrix.

[0029] The first hidden state matrix and its corresponding weight coefficients are input into the second Transformer layer in the scene model for weighted processing to obtain the predicted feature matrix.

[0030] In some embodiments, the step of inputting the first hidden state matrix and its corresponding weight coefficients into the corresponding second Transformer layer in the scene model for weighted processing to obtain the predicted feature matrix includes:

[0031] The weight coefficients are modulated by the sigmoid activation function of the second Transformer layer to obtain the modulated weight coefficients.

[0032] The first hidden state matrix is ​​multiplied by its corresponding modulated weight coefficients to obtain the modulated first hidden state matrix.

[0033] The modulated first hidden state matrix is ​​concatenated with the hierarchical scene feature matrix output by the second Transformer layer to obtain the hierarchical prediction feature matrix.

[0034] The above steps are executed iteratively through each of the second Transformer layers to generate a prediction feature matrix.

[0035] This invention also provides a scene prediction device based on a large language model to address issues such as the difficulty in sharing cross-scene features, insufficient learning ability for weak scenes, and inadequate utilization of deep semantic information in existing multi-scene prediction technologies. By introducing a unified prompt word construction mechanism, a cross-layer semantic fusion structure, and a scene-specific feature modeling mechanism, unified modeling of cross-scene semantic representation and refined extraction of target scene-specific features are achieved, thereby improving the accuracy, stability, and generalization ability of multi-scene prediction tasks.

[0036] This scene prediction device based on a large language model includes:

[0037] The prompt word encoding module is used to input the acquired prompt word sequence into a pre-established large language model and output the hidden state matrix of each first Transformer layer; wherein, the prompt word sequence is obtained by data preprocessing based on the acquired scene information, user information and material information; the hidden state matrix includes a first hidden state matrix and a second hidden state matrix output by the last first Transformer layer;

[0038] A general prediction generation module is used to perform a fully linked mapping on the second hidden state matrix using the first multilayer perceptron module to obtain a general prediction value.

[0039] The scene feature construction module is used to input the first hidden state matrix into a pre-established scene model to obtain a prediction feature matrix;

[0040] The scene prediction generation module is used to input the prediction feature matrix and the second hidden state matrix into the second multilayer perceptron module of the scene model for fully linked mapping to obtain the scene prediction value;

[0041] The result fusion module is used to obtain prediction results based on the general prediction value and the scenario prediction value.

[0042] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described scene prediction method based on a large language model.

[0043] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described scene prediction method based on a large language model.

[0044] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described scene prediction method based on a large language model.

[0045] The scene prediction method and apparatus based on a large language model provided in this invention constructs a prompt word sequence from scene information, user information, and material information and inputs it into a large language model. This fully utilizes the different depths of semantic features contained in the multi-layer Transformer structure of the large language model, overcoming the technical deficiency of traditional click-through rate prediction models in constructing a unified semantic representation across scenes. By mapping the hidden state matrix of the final layer of the large language model through a first multi-layer perceptron module, a universal prediction value shared across scenes can be formed, providing a stable and consistent basic feature representation for different business scenarios. Simultaneously, by inputting the hidden state matrices output from multiple Transformer layers of the large language model into the scene model and generating a prediction feature matrix, intermediate layer semantic features related to different business scenarios can be dynamically extracted, improving the modeling and semantic expression capabilities for weak scenes. By jointly mapping the second hidden state matrix and the prediction feature matrix through a second multi-layer perceptron module, scene prediction values ​​corresponding to the target business scenario can be accurately generated. By fusing universal prediction values ​​and scene prediction values, this invention enables the model to simultaneously possess cross-scene generalization ability and scene-specific discrimination ability, thereby improving the accuracy, stability, and scalability of multi-scene prediction tasks. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0047] Figure 1 This is a flowchart illustrating a scene prediction method based on a large language model in one embodiment of the present invention.

[0048] Figure 2 This is a flowchart illustrating a scene prediction method based on a large language model in another embodiment of the present invention.

[0049] Figure 3 This is a flowchart illustrating a scene prediction method based on a large language model in another embodiment of the present invention.

[0050] Figure 4 This is a flowchart illustrating a scene prediction method based on a large language model in another embodiment of the present invention.

[0051] Figure 5 This is a flowchart illustrating a scene prediction method based on a large language model in another embodiment of the present invention.

[0052] Figure 6 This is a flowchart illustrating a scene prediction method based on a large language model in another embodiment of the present invention.

[0053] Figure 7 This is a flowchart illustrating a scene prediction method based on a large language model in another embodiment of the present invention.

[0054] Figure 8 This is a schematic diagram of the scene prediction device based on a large language model in an embodiment of the present invention;

[0055] Figure 9 This is a schematic diagram of the physical structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0057] The information collected in the technical solution of this application is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation portals are provided for users to choose to authorize or refuse.

[0058] The acquisition, transmission, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0059] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0060] To address the challenges of existing multi-scenario recommendation and click-through rate prediction models, such as the difficulty in effectively sharing features across scenarios, insufficient weak-scenario modeling capabilities due to uneven data distribution between scenarios, and inadequate semantic information extraction caused by large language models using only the final layer semantic representation, this invention proposes a scenario prediction method based on a large language model. The method involves preprocessing the acquired scenario, user, and material information to construct a structured prompt word sequence. This prompt word sequence is then input into a pre-trained large language model, where its multi-layer Transformer structure performs layer-by-layer semantic encoding to obtain a multi-layer hidden state matrix representing the contextual semantics. Based on this, a general multilayer perceptron module is used to feature map the hidden states of the final layer of the large language model, generating a universally shared prediction value across scenarios. Simultaneously, a gating module is incorporated into the scenario model to weight and modulate the hidden state matrices of different Transformer layers of the large language model. This, combined with the second Transformer layer and the multilayer perceptron module of the scenario model, extracts a prediction feature matrix relevant to the target business scenario and generates scenario prediction values. By integrating general prediction values ​​and scenario prediction values, this invention achieves effective fusion of semantic information across multiple scenarios, improving the prediction accuracy and generalization ability of large language models in different business scenarios, thereby overcoming the technical problems of traditional multi-scenario modeling being difficult to process uniformly and lacking scalability.

[0061] This invention provides a scene prediction method based on a large language model. For example... Figure 1 As shown, the scene prediction method based on the large language model includes steps 101 to 105.

[0062] Step 101: Input the acquired prompt word sequence into the pre-built Large Language Model and output the hidden state matrix of each first Transformer layer. The prompt word sequence is obtained through data preprocessing based on the acquired scene information, user information, and material information. The hidden state matrix includes: a first hidden state matrix and a second hidden state matrix output from the last first Transformer layer.

[0063] Step 102: Use the first multilayer perceptron module to perform a fully linked mapping on the second hidden state matrix to obtain the general prediction value.

[0064] Step 103: Input the first hidden state matrix into the pre-established scene model to obtain the prediction feature matrix.

[0065] Step 104: Input the predicted feature matrix and the second hidden state matrix into the second multilayer perceptron module of the scene model for fully linked mapping to obtain the scene prediction value.

[0066] Step 105: Obtain the prediction result based on the general prediction value and the scenario prediction value.

[0067] According to the above embodiments, this invention unifies scene information, user information, and material information into a prompt word sequence and inputs it into a large language model. This fully utilizes the different depth semantic features contained in the multi-layer Transformer structure of the large language model, thereby overcoming the technical deficiency of traditional click-through rate prediction models in constructing a unified semantic representation across scenes. By performing feature mapping on the hidden state matrix of the final layer of the large language model through the first multi-layer perceptron module, a universal prediction value shared across scenes can be formed, providing a stable and consistent basic feature representation for different business scenarios. At the same time, by inputting the hidden state matrices output from multiple Transformer layers of the large language model into the scene model and generating a prediction feature matrix, intermediate layer semantic features related to different business scenarios can be dynamically extracted according to the scenario, improving the modeling ability and semantic expression ability for weak scenarios. By jointly mapping the second hidden state matrix and the prediction feature matrix through the second multi-layer perceptron module, scene prediction values ​​corresponding to the target business scenario can be accurately generated. By fusing the universal prediction value and the scene prediction value, this invention enables the model to simultaneously possess cross-scene generalization ability and scene-specific discrimination ability, thereby improving the accuracy, stability, and scalability of multi-scene prediction tasks.

[0068] In this embodiment of the invention, the large language model used is a pre-trained general-purpose large language model with a model parameter scale of more than 10 billion, which is used to provide sufficient semantic encoding capabilities. This general-purpose large language model does not require special training or fine-tuning for scene prediction tasks, but is directly applied to the prediction method of the present invention by utilizing the general semantic representation formed in its pre-training stage.

[0069] like Figure 2As shown, the general-purpose large language model consists of multiple layers of Transformer modules, including Transformer module 1, Transformer module 2, ..., Transformer module N stacked sequentially. Each Transformer module is used to perform layer-by-layer semantic encoding on the input prompt word sequence to form hidden state matrices of different depths. Each Transformer module of the general-purpose large language model includes a multi-head self-attention layer and a feed-forward network layer (FFN).

[0070] like Figure 2 As shown, the scene model includes multiple Transformer modules stacked sequentially, specifically Transformer module 1, Transformer module 2, ..., Transformer module M, and a second multi-layer perceptron block (MLPBlock) placed after Transformer module M. The number of Transformer modules in the scene model can be equal to the number of Transformer modules in the large language model.

[0071] In some embodiments, such as Figure 3 As shown, based on the obtained scene information, user information and material information, data preprocessing is performed to obtain the prompt word sequence, including steps 301 to 303.

[0072] Step 301: Obtain Scenario Information, User Information, and Item Information.

[0073] Step 302: Extract fields from scene information, user information, and material information to obtain common fields.

[0074] Step 303: Process the general fields according to the preset prompt template to obtain the prompt sequence.

[0075] In this embodiment of the invention, the scenario information includes a scenario ID for identifying the business scenario, a scenario description text for describing the scenario content, and a scenario type for distinguishing business forms. The scenario type may include, but is not limited to, food merchant recommendation scenarios, search recommendation scenarios, coupon recommendation scenarios, and information feed recommendation scenarios.

[0076] User information is used to characterize user features and behavioral characteristics, including: basic user information (User Profile), user interests and preferences, and user historical behavior sequences. Basic user information may include attributes such as user age, gender, city, region, and type of device. User interests and preferences may include tagged interests such as food, entertainment, travel, education, and digital products. User behavior sequences may include historical behavioral events such as merchants or products clicked, text or video content clicked or viewed, travel guides viewed, search keywords, saved or liked content, and purchase or coupon records.

[0077] Material information describes candidate content or recommended items, including: material name, material type, material category, material tags, and material description text. The material name can include: merchant name and product name, etc. The material type can include: merchant category, coupon category, image and text category, video category, and product category, etc. The material category can include: content categories such as food, entertainment, travel, and education. Material tags can include attribute tags such as: established brands, suitable for gatherings, popular, content theme, and keywords, etc.

[0078] The acquired scenario information, user information, and material information undergo field extraction processing to extract general fields that can be used for semantic construction from the raw business data. The field extraction process includes identifying and retaining fields with stable semantic meaning that can be expressed in text form within the business scenario. Examples include scenario identifiers, scenario types, basic user attributes, user interests and preferences, user historical behavior sequences, material types, material categories, material tags, and material description text. By performing standardization, vectorization preprocessing, and field filtering operations on the above information, a general set of fields for prompt word generation can be constructed.

[0079] To ensure that the constructed prompt word sequence can be accurately parsed by a large language model, this invention employs a pre-defined prompt word template to standardize the organization of common fields within the prompt word sequence. This prompt word template defines the order of common fields in the prompt word sequence, a unified field identifier format, semantic separators between field contents, and structured tags used to distinguish hierarchical structures, thereby guaranteeing the semantic consistency and parsability of the prompt word sequence.

[0080] Under the constraints of the prompt template, the extracted general fields are formatted, including steps such as field encoding, semantic delimiter filling, structured label appending, and field content concatenation, to generate a prompt sequence compatible with the input format of the large language model.

[0081] According to the above embodiments, by standardizing the extraction of scene information, user information and material information, and using a preset prompt word template to structure the common fields into a prompt word sequence, the large language model can receive multi-source heterogeneous business data in a unified format, realizing consistent expression of cross-scene semantics, thereby improving the large language model's ability to understand and generalize multi-scene content.

[0082] In some embodiments, such as Figure 4 As shown, step 101 includes steps 401 to 404.

[0083] Step 401: Perform word segmentation on the prompt word sequence to obtain multiple word groups and the length of the prompt word sequence.

[0084] Step 402: Generate integer numbers corresponding to each word group based on the vocabulary of the large language model.

[0085] Step 403: Vectorize the integer numbers to obtain the embedding vectors corresponding to each word group.

[0086] Step 404: Input each embedding vector into the first Transformer layer of the large language model for iterative semantic encoding processing to generate the hidden state matrix. The first Transformer layer consists of the Transformer modules of the large language model.

[0087] In embodiments of the present invention, such as Figure 2 As shown, the input prompt word sequence is segmented into multiple word groups (tokens) by the tokenizer in the input processing layer of the large language model. Each word group is then mapped to a corresponding integer number (token_id) by the vocabulary lookup layer in the input processing layer of the large language model.

[0088] Each integer token_id is vectorized using the embedding layer to obtain an embedding vector corresponding to each token_id. The embedding layer then encodes the obtained embedding vectors according to the batch size, the sequence length, and the hidden dimension to generate an initial hidden state matrix.

[0089] Wherein, batch_size represents the number of prompt word sequences processed in parallel during a single forward computation, sequence_length represents the number of word groups after word segmentation, and hidden_size represents the embedding vector dimension corresponding to each word group.

[0090] The initial hidden state matrix is ​​input into Transformer module 1 of the large language model. The initial hidden state matrix is ​​then semantically encoded layer by layer through the attention layers and feedforward networks of each Transformer module (Transformer module 1 to Transformer module N), generating the corresponding hierarchical hidden state matrices. Transformer module N of the large language model outputs the final hidden state matrix. The hidden state matrix is ​​a three-dimensional tensor with the following dimensions:

[0091] .

[0092] According to the above embodiments, a unified vectorized representation of multi-source heterogeneous prompt word sequences is achieved through word segmentation, word list mapping, and embedding vector construction. The initial hidden state matrix is ​​semantically encoded layer by layer using the multi-layer Transformer module of the large language model, thereby fully mining the contextual associations and deep semantic information in the prompt word sequences. This provides a structured, stable, and highly expressive semantic feature representation for subsequent prediction tasks, improving the overall prediction performance and generalization ability of the large language model.

[0093] In some embodiments, such as Figure 5 As shown, step 102 includes steps 501 to 503.

[0094] Step 501: Select the embedding vector at the preset position from the hidden state matrix to obtain the sample-level hidden vector.

[0095] Step 502: Perform linear transformation and nonlinear activation on the sample-level hidden vectors to obtain feature scalars.

[0096] Step 503: Normalize the feature scalars using an activation function to obtain a general prediction value.

[0097] In embodiments of the present invention, such as Figure 2As shown, the first multilayer perceptron module is a general multilayer perceptron block, which includes a neural network composed of multiple fully-connected layers. This neural network performs linear transformations and non-linear activations on the final hidden state matrix output by the Transformer module N of the large language model. The general multilayer perceptron module is located at the output of the large language model and is connected to the Transformer module N of the large language model.

[0098] After the large language model performs multi-layer Transformer encoding on the prompt word sequence, the output dimension is as follows: The final hidden state matrix. Here, sequence_length is the number of word groups after segmentation, and hidden_size represents the dimension of the embedding vector corresponding to each word group.

[0099] For any target sample, its corresponding hidden state fragment can be represented as: The hidden state is a two-dimensional matrix. The general multilayer perceptron module performs vector aggregation on this hidden state two-dimensional matrix to generate sample-level hidden vectors for general prediction.

[0100] Specifically, the general-purpose multilayer perceptron module selects the hidden vector corresponding to a specific position (such as the CLS marker position for aggregation semantics or the start position of the sequence) from the hidden state two-dimensional matrix based on preset rules, or performs pooling operations (such as average pooling or max pooling) on ​​the sequence dimension to compress the hidden state two-dimensional matrix into a sample-level hidden vector with a length of hidden_size.

[0101] After obtaining the sample-level hidden vectors, feature mapping is performed through the multi-layer fully connected structure of a general multilayer perceptron module to obtain a one-dimensional feature scalar z. A sigmoid activation function is then applied to the feature scalar z to map it to... The interval is used to calculate the general predicted value. .

[0102] According to the above embodiments, by introducing a general multilayer perceptron module at the output end of the large language model, sequence aggregation and feature mapping are performed on the final hidden state matrix, which can compress high-dimensional sequence semantics into stable sample-level feature representations. Furthermore, through nonlinear normalization, a general prediction value that can be used for cross-scenario tasks is obtained, thereby improving the basic prediction capability and generalization performance of the large language model in different business scenarios and realizing the unified modeling and effective utilization of semantic information in multiple scenarios.

[0103] In some embodiments, the scene prediction method based on a large language model further includes setting gate blocks in the scene model according to preset frequency parameters. Each gate block corresponds one-to-one with at least a portion of the first Transformer layer of the large language model. The frequency parameters indicate the hierarchical interval for setting gate blocks in the scene model.

[0104] In embodiments of the present invention, such as Figure 2 As shown, the scene model also includes multiple gating modules, such as gating module 1 to gating module L. The number of gating modules is less than the number of Transformer modules in the scene model. Gating modules are set between the Transformer modules of the scene model according to a preset hierarchical interval, so that each gating module establishes an association with a specific Transformer module in the large language model and the corresponding Transformer module in the scene model.

[0105] For example, such as Figure 2 As shown, a gating module can be set up for every alternate layer of Transformer modules.

[0106] According to the above embodiments, by inserting gating modules into the multi-layer Transformer structure of the scene model at preset hierarchical intervals, the scene model can selectively modulate and introduce the semantic features output from different layers of the large language model layer by layer, thereby realizing hierarchical semantic adaptive extraction for different business scenarios, enhancing the scene model's ability to capture key information, and improving the expression effect and overall prediction accuracy of weak scenarios.

[0107] In some embodiments, such as Figure 6 As shown, step 103 includes steps 601 to 602.

[0108] Step 601: The first hidden state matrix output by the corresponding first Transformer layer is weighted by each gating module to obtain the weight coefficients corresponding to the first hidden state matrix.

[0109] Step 602: Input the first hidden state matrix and its corresponding weight coefficients into the corresponding second Transformer layer in the scene model for weighted processing to obtain the predicted feature matrix. Here, the first hidden state matrix is ​​the hidden state matrix output by each Transformer module of the large language model. The second Transformer layer consists of each Transformer module of the scene model.

[0110] In this embodiment of the invention, the gating module is a multilayer perceptron network composed of a multilayer fully connected structure, used to perform weight modulation on the hidden state matrix output by different levels of the large language model.

[0111] Specifically, the gating module receives the hidden state matrix output by its corresponding large language model Transformer module, and generates weight coefficients corresponding to the hidden state matrix based on its internal nonlinear mapping structure. The hidden state matrix and its corresponding weight coefficients are then input into the scene model Transformer module corresponding to the gating module.

[0112] For example, such as Figure 2 As shown, gating module 1 is connected to Transformer module 2 of the large language model and Transformer module 1 of the scene model. Gating module 1 receives the hidden state matrix output by Transformer module 2 of the large language model, and performs weight assignment on the hidden state matrix to obtain the weight coefficients corresponding to the hidden state matrix. Gating module 1 then inputs the hidden state matrix and its corresponding weight coefficients into Transformer module 1 of the scene model.

[0113] Similarly, gating module 2 is connected to Transformer module 4 of the large language model and Transformer module 2 of the scene model, respectively. Gating module L is connected to Transformer module N of the large language model and Transformer module M of the scene model, respectively.

[0114] According to the above embodiments, by introducing a gating module corresponding to the multi-layer semantics of the large language model, selective modulation of the hidden state matrix of each layer is realized, enabling the scene model to dynamically receive semantic features of different depths and perform hierarchical guidance, thereby enhancing the scene model's ability to capture key information, improving the modeling effect of weak scenes, and significantly improving the accuracy and adaptability of the overall multi-scene prediction task.

[0115] In some embodiments, such as Figure 7 As shown, step 602 includes steps 701 to 704.

[0116] Step 701: Modulate the weight coefficients using the sigmoid activation function of the second Transformer layer to obtain the modulated weight coefficients.

[0117] Step 702: Perform a multiplication operation on the first hidden state matrix and its corresponding modulated weight coefficients to obtain the modulated first hidden state matrix.

[0118] Step 703: Concatenate the modulated first hidden state matrix with the hierarchical scene feature matrix output by the second Transformer layer to obtain the hierarchical prediction feature matrix.

[0119] Step 704: Iterate through each of the second Transformer layers to generate the predicted feature matrix.

[0120] In this embodiment of the invention, each Transformer module of the scene model receives the hidden matrix of the large language model output by its corresponding gating module. and weighting coefficients By applying a sigmoid activation function to the weight coefficients, the hidden matrix is... Perform a modulation operation to obtain the modulated hidden matrix. The calculation formula is as follows:

[0121]

[0122] in, This is the hidden matrix output by the Transformer module, which corresponds to the gating module. It is a sigmoid activation function. The weight coefficient vector generated by the gating module corresponding to the scene model.

[0123] The Transformer module of the scene model will modulate the hidden matrix. With the scene feature matrix generated internally by this Transformer module Perform a tensor concat operation on the hidden vector dimension to obtain the hierarchical prediction feature matrix of this Transformer module.

[0124] The Transformer module outputs the generated hierarchical prediction feature matrix to the next layer of the Transformer module in the scene model.

[0125] Each Transformer module of the scene model sequentially receives the hidden matrix and weight coefficients output by its corresponding gating module, as well as the hierarchical prediction feature matrix output by the previous Transformer module. The above steps are executed iteratively until the last Transformer module of the scene model outputs the final prediction feature matrix.

[0126] For example, suppose the hidden matrix output by Transformer module 4 of the large language model is denoted as... Its dimensions are The scene feature matrix output by Transformer module 2 of the scene model is denoted as... Its dimensions are also Gating module 2, based on its internal multilayer perceptron structure, processes the hidden matrix output by Transformer module 4 of the large language model. Perform mapping to generate a hidden matrix. The corresponding weight coefficient vector Its dimensions are .

[0127] After obtaining the weight coefficient vector Then, gating module 2 will transfer the weight coefficient vector and hidden matrix Input to Transformer module 2 of the scene model. Hidden matrix. After modulation using the sigmoid activation function, the modulated hidden matrix is ​​obtained. For the modulated hidden matrix With scene feature matrix Tensor concatenation is performed to obtain the hierarchical prediction feature matrix of Transformer module 2 of the scene model. Transformer module 2 of the scene model outputs this hierarchical prediction feature matrix to Transformer module 3 of the scene model.

[0128] The above steps are executed iteratively by the Transformer modules of each scene model until the Transformer module M of the scene model outputs the final predicted feature matrix.

[0129] According to the above embodiments, by introducing cross-layer semantic modulation based on a gating mechanism into the multi-layer Transformer structure of the scene model, the scene model can selectively receive semantic features of different depths from the large language model in a layer-by-layer injection manner. By applying weight coefficients to the hidden state matrix output by the large language model and then modulating it after sigmoid activation, dynamic control of the semantic contribution of each layer can be achieved, avoiding the situation where high-level or low-level features dominate the expression of the scene model, thereby improving the utilization efficiency of key semantic information. By concatenating the modulated hidden state matrix with the scene feature matrix generated internally by the scene model along the hidden vector dimension, the scene model's feature expression capability for specific business contexts is enhanced, enabling the scene model to obtain higher feature richness and discriminative ability in the process of multi-source semantic fusion.

[0130] In some embodiments, such as Figure 2 As shown, the Transformer module M of the scene model outputs the final predicted feature matrix to the Multilayer Perceptron (MLP) Block. The large language model outputs the final hidden state matrix to the MLP Block.

[0131] The multilayer perceptron module performs vector aggregation on the final predicted feature matrix and the final hidden state matrix to generate predicted sample-level scene feature vectors. After obtaining the sample-level scene feature vectors, feature mapping is performed through the multilayer fully connected structure of the multilayer perceptron module to obtain a one-dimensional scene feature scalar. After applying a sigmoid activation function to this scene feature scalar, it is mapped to... The scene prediction value is obtained by dividing the range. The general prediction value and the scene prediction value are then fused to obtain the prediction result.

[0132] According to the above embodiments, by jointly mapping the second hidden state matrix and the predicted feature matrix through the second multilayer perceptron module, scene prediction values ​​corresponding to the target business scenario can be accurately generated. By fusing general prediction values ​​and scene prediction values, this invention enables the model to simultaneously possess cross-scenario generalization ability and scene-specific discrimination ability, thereby improving the accuracy, stability, and scalability of multi-scenario prediction tasks.

[0133] In some embodiments, when training the scene model, the binary cross-entropy loss function (BCE) can be used to calculate the scene loss separately. and general loss .

[0134] The binary cross-entropy loss function is calculated as shown in the following formula (2):

[0135]

[0136] in, The number of samples used to train the model. For the first The true labels of each training sample This is the predicted probability value for the corresponding sample, which is either the scene prediction value or the general prediction value.

[0137] Substituting the scene prediction value and the general prediction value into the above formula (2) respectively, the scene loss is calculated. and general loss .

[0138] Loss of scene and general loss Substituting into the following formula (3), the total loss function is calculated. .

[0139]

[0140] in, The scene loss is calculated based on the scene prediction values. This is the general loss calculated based on the general predicted value.

[0141] The total loss function is obtained based on the binary cross-entropy loss function. Then, the backpropagation algorithm is used to calculate the loss function. The gradient of each model parameter is calculated, and the model parameters are updated based on the gradient.

[0142] According to the above embodiments, by calculating the scene loss and the general loss separately and jointly optimizing the two, the model can learn both cross-scene shared features and scene-specific features at the same time, thereby improving the stability and convergence efficiency of model training.

[0143] This invention also provides a scene prediction device based on a large language model, as described in the following embodiments. Since the principle by which this device solves the problem is similar to that of the scene prediction method based on a large language model, the implementation of this device can refer to the implementation of the scene prediction method based on a large language model; repeated details will not be elaborated further.

[0144] like Figure 8 As shown, the scene prediction device based on a large language model includes: a prompt word encoding module 801, a general prediction generation module 802, a scene feature construction module 803, a scene prediction generation module 804, and a result fusion module 805.

[0145] The prompt word encoding module 801 is used to input the acquired prompt word sequence into a pre-established large language model and output the hidden state matrix of each first Transformer layer. The prompt word sequence is obtained through data preprocessing based on the acquired scene information, user information, and material information. The hidden state matrix includes a first hidden state matrix and a second hidden state matrix output by the last first Transformer layer.

[0146] The general prediction generation module 802 is used to perform a fully linked mapping on the second hidden state matrix using the first multilayer perceptron module to obtain a general prediction value.

[0147] The scene feature construction module 803 is used to input the first hidden state matrix into the pre-established scene model to obtain the prediction feature matrix.

[0148] The scene prediction generation module 804 is used to input the prediction feature matrix and the second hidden state matrix into the second multilayer perceptron module of the scene model for fully linked mapping to obtain the scene prediction value.

[0149] The result fusion module 805 is used to obtain prediction results based on general predicted values ​​and scenario predicted values.

[0150] In some embodiments, the scene prediction device based on a large language model further includes a data preprocessing module. This data preprocessing module includes a data acquisition submodule, a field extraction submodule, and a prompt word construction submodule.

[0151] The data acquisition submodule is used to acquire scene information, user information, and material information.

[0152] The field extraction submodule is used to extract fields from scene information, user information, and material information to obtain common fields.

[0153] The prompt word construction submodule is used to process the general fields according to the preset prompt word template to obtain the prompt word sequence.

[0154] In some embodiments, the prompt word encoding module 801 includes: a word segmentation processing submodule, a word list mapping submodule, a vectorization processing submodule, and a semantic encoding submodule.

[0155] The word segmentation submodule is used to segment the prompt word sequence to obtain multiple word groups and the length of the prompt word sequence.

[0156] The vocabulary mapping submodule is used to generate integer numbers corresponding to each word group based on the vocabulary of a large language model.

[0157] The vectorization processing submodule is used to vectorize integer numbers to obtain the embedding vectors corresponding to each word group.

[0158] The semantic encoding submodule is used to input each embedding vector into the first Transformer layer of the large language model for iterative semantic encoding processing to generate the hidden state matrix.

[0159] In some embodiments, the general prediction generation module 802 includes: a vector aggregation submodule, a feature mapping submodule, and a general prediction output submodule.

[0160] The vector aggregation submodule is used to select the embedding vector at a preset position from the hidden state matrix to obtain the sample-level hidden vector.

[0161] The feature mapping submodule is used to perform linear transformations and nonlinear activation processing on the sample-level hidden vectors to obtain feature scalars.

[0162] The general prediction output submodule is used to normalize the feature scalars through an activation function to obtain a general prediction value.

[0163] In some embodiments, the scene prediction device based on a large language model further includes a gating configuration module.

[0164] The gating configuration module is used to set gating modules in the scene model according to preset frequency parameters. Each gating module corresponds one-to-one with at least a portion of the first Transformer layer of the large language model.

[0165] In some embodiments, the scene feature construction module 803 includes: a weight generation submodule and a hidden state weighting submodule.

[0166] The weight generation submodule is used to assign weights to the first hidden state matrix output by the corresponding first Transformer layer through each gating module, so as to obtain the weight coefficients corresponding to the first hidden state matrix.

[0167] The hidden state weighting submodule is used to input the first hidden state matrix and its corresponding weight coefficients into the corresponding second Transformer layer in the scene model for weighting processing to obtain the predicted feature matrix.

[0168] In some embodiments, the hidden state weighting submodule includes: a weight modulation unit, a hidden state modulation unit, a feature splicing unit, and a hierarchical feature iteration unit.

[0169] The weight modulation unit is used to modulate the weight coefficients through the sigmoid activation function of the second Transformer layer to obtain the modulated weight coefficients.

[0170] The hidden state modulation unit is used to perform a multiplication operation on the first hidden state matrix and its corresponding modulated weight coefficients to obtain the modulated first hidden state matrix.

[0171] The feature concatenation unit is used to concatenate the modulated first hidden state matrix with the hierarchical scene feature matrix output by the second Transformer layer to obtain the hierarchical prediction feature matrix.

[0172] The hierarchical feature iteration unit is used to iteratively execute the above steps through each second Transformer layer to generate the prediction feature matrix.

[0173] It should be noted that the scene prediction method based on a large language model provided in this embodiment of the invention can be used in the financial field, or in any technical field other than the financial field. This embodiment of the invention does not limit the application field of the virtual reality-based bank training system and its training method.

[0174] Figure 9 This is a schematic diagram of the physical structure of a computer device provided in an embodiment of the present invention, such as... Figure 9 As shown, the computer device includes a processor 901, a memory 902, and a bus 903.

[0175] The processor 901 and the memory 902 communicate with each other via the bus 903.

[0176] The processor 901 is used to call program instructions in the memory 902 to execute the methods provided in the above-described method embodiments.

[0177] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described scene prediction method based on a large language model.

[0178] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described scene prediction method based on a large language model.

[0179] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0180] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0181] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0182] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0183] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for predicting a scene based on a large language model, the method comprising: The method comprises the following steps: input the obtained prompt word sequence into a pre-established large language model to output a hidden state matrix of each first Transformer layer; wherein the prompt word sequence is obtained by data preprocessing based on obtained scene information, user information and material information; the hidden state matrix comprises a first hidden state matrix and a second hidden state matrix output by the last first Transformer layer; perform full link mapping on the second hidden state matrix by using a first multi-layer perception module to obtain a general prediction value; input the first hidden state matrix into a pre-established scene model to obtain a predicted feature matrix; perform full link mapping on the predicted feature matrix and the second hidden state matrix by using a second multi-layer perception module of the scene model to obtain a scene prediction value; obtain a prediction result based on the general prediction value and the scene prediction value.

2. The method of claim 1, wherein, The data preprocessing based on obtained scene information, user information and material information to obtain a prompt word sequence comprises the following steps: obtain the scene information, user information and material information; extract fields from the scene information, user information and material information to obtain general fields; perform data processing on the general fields according to a preset prompt word template to obtain a prompt word sequence.

3. The method of claim 1, wherein, The inputting of the obtained prompt word sequence into a pre-established large language model to output a hidden state matrix of each first Transformer layer comprises the following steps: perform word segmentation processing on the prompt word sequence to obtain a plurality of word groups and a prompt word sequence length; generate an integer number corresponding to each word group based on a word table of the large language model; perform vectorization processing on the integer number to obtain an embedding vector corresponding to each word group; input each embedding vector into a first Transformer layer of the large language model to perform iterative semantic coding processing to generate the hidden state matrix.

4. The method of claim 1, wherein, The full link mapping on the second hidden state matrix by using a first multi-layer perception module to obtain a general prediction value comprises the following steps: select an embedding vector at a preset position from the hidden state matrix to obtain a sample-level hidden vector; perform linear transformation and nonlinear activation processing on the sample-level hidden vector to obtain a feature scalar; perform normalization processing on the feature scalar by using an activation function to obtain a general prediction value.

5. The method of claim 3, wherein, The method further comprises the following steps: set a gate module in the scene model according to a preset frequency parameter; each gate module corresponds to at least part of the first Transformer layers of the large language model in one-to-one correspondence.

6. The method of claim 5, wherein, The inputting of the first hidden state matrix into a pre-established scene model to obtain a predicted feature matrix comprises the following steps: perform weight distribution on the first hidden state matrix output by the corresponding first Transformer layer by using each gate module to obtain a weight coefficient corresponding to the first hidden state matrix; input the first hidden state matrix and the weight coefficient corresponding thereto into the corresponding second Transformer layer in the scene model to perform weighting processing to obtain a predicted feature matrix.

7. The method of claim 6, wherein, The first hidden state matrix and its corresponding weight coefficient are input into the corresponding second Transformer layer of the scene model for weighted processing to obtain a predicted feature matrix, including: The weight coefficient is modulated by an S-shaped activation function of the second Transformer layer to obtain a modulated weight coefficient; The first hidden state matrix and its corresponding modulated weight coefficient are multiplied to obtain a modulated first hidden state matrix; The modulated first hidden state matrix and the hierarchical scene feature matrix output by the second Transformer layer are spliced to obtain a hierarchical predicted feature matrix; The above steps are iteratively executed by each second Transformer layer to generate a predicted feature matrix. 8.A scenario prediction apparatus based on a large language model, characterized in that, including: The prompt word encoding module inputs the obtained prompt word sequence into a pre-established large language model to output a hidden state matrix of each first Transformer layer; wherein the prompt word sequence is obtained based on data preprocessing of obtained scene information, user information and material information; the hidden state matrix includes a first hidden state matrix and a second hidden state matrix output by the last first Transformer layer; The general prediction generation module uses a first multi-layer perception module to perform full-link mapping on the second hidden state matrix to obtain a general prediction value; The scene feature construction module inputs the first hidden state matrix into a pre-established scene model to obtain a predicted feature matrix; The scene prediction generation module inputs the predicted feature matrix and the second hidden state matrix into a second multi-layer perception module of the scene model for full-link mapping to obtain a scene prediction value; The result fusion module obtains a prediction result based on the general prediction value and the scene prediction value.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-7.

11. A computer program product, characterised in that, The computer program product includes a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-7.

Images