Model training method and data processing method

Abstract

The invention discloses a model training method and a data processing method. Relates to the technical field of artificial intelligence, and the method comprises the following steps: processing article request information through an initial model to obtain a plurality of pieces of recommendation information and action track information corresponding to the plurality of pieces of recommendation information, the action track information being composed of a plurality of action events, the plurality of action events are respectively composed of a current action executed by the initial model, input state information before the initial model executes the current action and environment feedback information corresponding to the current action; according to the action track information, target score values corresponding to characters in the multiple pieces of recommendation information are determined; according to the target score value, the initial model is iteratively updated, a target model is obtained, and the target model outputs recommended article information based on an article request input by a target object. According to the method and the device, the technical problem of relatively low accuracy of processing the user article request by a model in related technologies is solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a model training method and a data processing method. Background Technology

[0002] In the search and recommendation domain, users often submit complex item requests with multi-dimensional attributes, such as height, capacity, brand, and condition (new or used). However, existing technologies exhibit significant shortcomings in handling these requests: software development has evolved from static programming to large-model-driven automatic generation; however, the accuracy of these models in processing specific user item requests remains unsatisfactory. While models can initially utilize query information to perceive the environment and plan actions, they cannot flexibly integrate external knowledge and product database resources, resulting in relatively low accuracy in processing user item requests.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a model training method and a data processing method to at least solve the technical problem in the related art that the model has low accuracy in processing user item requests.

[0005] According to one aspect of the embodiments of this application, a model training method is provided, comprising: processing item request information through an initial model to obtain multiple recommendation information and action trajectory information corresponding to the multiple recommendation information, wherein the action trajectory information consists of multiple action events, each of the multiple action events consisting of a current action executed by the initial model, input state information of the initial model before executing the current action, and environmental feedback information corresponding to the current action; determining a target score value corresponding to a character in the multiple recommendation information based on the action trajectory information; and iteratively updating the initial model based on the target score value to obtain a target model, wherein the target model outputs recommended item information based on the item request input by the target object.

[0006] Further, determining the target score corresponding to the character in the plurality of recommended information based on the action trajectory information includes: obtaining the first score corresponding to the action event in the action trajectory information; obtaining the second score corresponding to each of the plurality of recommended information; and obtaining the target score based on the first score and the second score.

[0007] Further, obtaining the first score corresponding to the action event in the action trajectory information includes: for any action event in the action trajectory information, calculating a first sub-score based on the item request information and the input state information in the action event; calculating a second sub-score based on the item request information, the action in the action event, and the environmental feedback information in the action event; calculating a third sub-score based on the target tool called by the initial model in the action event and the interaction rules corresponding to the target tool; and calculating the first score based on the first sub-score, the second sub-score, and the third sub-score.

[0008] Further, obtaining the second score corresponding to each of the multiple recommendation information includes: for any one of the multiple recommendation information, calculating a fourth score based on the recommendation information and the item request information; calculating a fifth score based on a preset rule and the recommendation information; calculating a sixth score based on the product parameter information in the recommendation information; and calculating the second score based on the fourth score, the fifth score, and the sixth score.

[0009] Further, obtaining the target score based on the first score and the second score includes: calculating a target response score based on the first score and the second score; calculating an action level score corresponding to the action event in the action trajectory information based on the target response score; and obtaining the target score based on the target response score and the action level score.

[0010] Further, the calculation of the target response score based on the first score and the second score includes: for a target action event, obtaining multiple first scores for the target action event corresponding to the multiple recommendation information, and performing standardized calculation based on the multiple first scores to obtain a first standardized score for the target action event, wherein the target action event is any action event in the action trajectory information; for target recommendation information, obtaining a second standardized score for the target recommendation information based on multiple second scores corresponding to the multiple recommendation information, and performing standardized calculation based on the multiple second scores; calculating the initial response score corresponding to the target recommendation information based on the first standardized score, the second standardized score, and a preset discount factor; and performing standardized calculation based on the initial response score corresponding to the multiple recommendation information to obtain the target response score.

[0011] Further, the calculation based on the target response score to obtain the action-level score corresponding to the action event in the action trajectory information includes: calculating the first weight value corresponding to the target action event based on the first standardized score value corresponding to the target action event and the first standardized score value corresponding to the action event in the action trajectory information; and calculating the action-level score corresponding to the target action event based on the initial response score value and the first weight value.

[0012] Further, obtaining the target score based on the target response score and the action level score includes: for any character in the target recommendation information, if the character is the last character of the target action event in the action trajectory information corresponding to the target recommendation information, then the initial score corresponding to the character is determined based on the action level score of the target action event; if the character is the last character in the target recommendation information, then the initial score corresponding to the character is determined based on the target response score; if the character is neither the last character in the target recommendation information nor the last character of the target action event in the action trajectory information corresponding to the target recommendation information, then the initial score corresponding to the character is determined based on a preset value; and the target score is obtained based on the initial score corresponding to the character.

[0013] Further, obtaining the target score based on the initial score corresponding to the character includes: calculating the divergence penalty value corresponding to the character based on the first probability value of the character output by the initial model under the current policy and the second probability value of the character output under the historical policy; and calculating the target score based on the divergence penalty value and the initial score.

[0014] Further, based on the target score, the initial model is iteratively updated to obtain the target model, including: calculating a cumulative advantage function based on the target scores of multiple characters in the target recommendation information; for the target character in the target recommendation information, calculating a second weight value based on the third probability value of the target character output by the initial model under the current strategy, the fourth probability value of the target character output under the historical strategy, and the total number of characters in the action event in which the target character is located; and iteratively updating the initial model based on the cumulative advantage function, the second weight value, and a preset update threshold to obtain the target model.

[0015] According to another aspect of the embodiments of this application, a data processing method is also provided, comprising: receiving target item request information input by a target object; processing the target item request information through a target model to obtain recommended item information, wherein the target model is trained using any of the model training methods described above; and returning the recommended item information to the target object.

[0016] According to another aspect of the embodiments of this application, a model training apparatus is also provided, comprising: a first processing unit, configured to process item request information through an initial model to obtain multiple recommendation information and action trajectory information corresponding to the multiple recommendation information, wherein the action trajectory information consists of multiple action events, and the multiple action events consist of the current action executed by the initial model, the input state information of the initial model before executing the current action, and the environmental feedback information corresponding to the current action; a determining unit, configured to determine the target score value corresponding to the characters in the multiple recommendation information based on the action trajectory information; and an updating unit, configured to iteratively update the initial model based on the target score value to obtain a target model, wherein the target model outputs recommended item information based on the item request input by the target object.

[0017] Further, the determining unit includes: a first acquisition subunit, used to acquire a first score value corresponding to the action event in the action trajectory information; a second acquisition subunit, used to acquire a second score value corresponding to each of the multiple recommendation information; and a determining subunit, used to obtain the target score value based on the first score value and the second score value.

[0018] Further, the first acquisition subunit includes: a first calculation module, used to calculate, for any action event in the action trajectory information, a first fractional value based on the item request information and the input state information in the action event; a second calculation module, used to calculate, a second fractional value based on the item request information, the action in the action event, and the environmental feedback information in the action event; a third calculation module, used to calculate, a third fractional value based on the target tool called by the initial model in the action event and the interaction rules corresponding to the target tool; and a fourth calculation module, used to calculate, a first fractional value based on the first fractional value, the second fractional value, and the third fractional value.

[0019] Further, the second acquisition subunit includes: a fifth calculation module, used to calculate a fourth fractional value for any one of the multiple recommendation information and the item request information; a sixth calculation module, used to calculate a fifth fractional value based on preset rules and the recommendation information; a seventh calculation module, used to calculate a sixth fractional value based on the product parameter information in the recommendation information; and an eighth calculation module, used to calculate a second fractional value based on the fourth fractional value, the fifth fractional value, and the sixth fractional value.

[0020] Further, the determining subunit includes: a ninth calculation module, used to calculate based on the first score and the second score to obtain a target response score; a tenth calculation module, used to calculate based on the target response score to obtain an action level score corresponding to the action event in the action trajectory information; and a determining module, used to obtain the target score based on the target response score and the action level score.

[0021] Further, the ninth calculation module includes: a first calculation submodule, used to obtain multiple first scores of the target action event corresponding to the multiple recommendation information for the target action event, and perform standardized calculation based on the multiple first scores to obtain a first standardized score of the target action event, wherein the target action event is any action event in the action trajectory information; a second calculation submodule, used to obtain a second standardized score of the target recommendation information based on multiple second scores corresponding to the multiple recommendation information, and perform standardized calculation based on the multiple second scores; a third calculation submodule, used to calculate based on the first standardized score, the second standardized score, and a preset discount factor to obtain an initial response score corresponding to the target recommendation information; and a fourth calculation submodule, used to perform standardized calculation based on the initial response scores corresponding to the multiple recommendation information to obtain the target response score.

[0022] Furthermore, the tenth calculation module includes: a fifth calculation submodule, used to calculate based on the first standardized score value corresponding to the target action event and the first standardized score value corresponding to the action event in the action trajectory information to obtain the first weight value corresponding to the target action event; and a sixth calculation submodule, used to calculate based on the initial response score value and the first weight value to obtain the action level score corresponding to the target action event.

[0023] Further, the determining module includes: a first determining submodule, used to determine an initial score value corresponding to any character in the target recommendation information if the character is the last character of the target action event in the action trajectory information corresponding to the target recommendation information, based on the action level score of the target action event; a second determining submodule, used to determine an initial score value corresponding to the character if the character is the last character in the target recommendation information, based on the target response score value; a third determining submodule, used to determine an initial score value corresponding to the character based on a preset value if the character is neither the last character in the target recommendation information nor the last character of the target action event in the action trajectory information corresponding to the target recommendation information; and a fourth determining submodule, used to obtain the target score value based on the initial score value corresponding to the character.

[0024] Furthermore, the fourth determining submodule includes: a first calculation submodule, used to calculate the divergence penalty value corresponding to the character based on the first probability value of the character output under the current strategy and the second probability value of the character output under the historical strategy of the initial model; and a second calculation submodule, used to calculate the target score value based on the divergence penalty value and the initial score value.

[0025] Furthermore, the update unit includes: a first calculation subunit, used to calculate a cumulative advantage function based on the target scores of multiple characters in the target recommendation information; a second calculation subunit, used to calculate a second weight value for the target character in the target recommendation information based on the third probability value of the target character output by the initial model under the current strategy, the fourth probability value of the target character output under the historical strategy, and the total number of characters in the action event in which the target character is located; and iteratively updating the initial model based on the cumulative advantage function, the second weight value, and a preset update threshold to obtain the target model.

[0026] According to another aspect of the embodiments of this application, a data processing apparatus is also provided, comprising: a receiving unit for receiving target item request information input by a target object; a second processing unit for processing the target item request information through a target model to obtain recommended item information, wherein the target model is trained using any of the model training methods described above; and a returning unit for returning the recommended item information to the target object.

[0027] In this embodiment, the following steps are employed: An initial model processes item request information to obtain multiple recommendation messages and corresponding action trajectory information for each recommendation message. The action trajectory information consists of multiple action events, each composed of the current action executed by the initial model, the input state information of the initial model before executing the current action, and the environmental feedback information corresponding to the current action. Based on the action trajectory information, target scores are determined for the characters in the multiple recommendation messages. Based on the target scores, the initial model is iteratively updated to obtain a target model. The target model outputs recommended item information based on the item request input by the target object, thus solving the technical problem of low accuracy in processing user item requests in related technologies.

[0028] In this application, an initial model is used to parse and process user item request information, thereby generating multiple recommendation messages. Simultaneously, the model's action trajectory information during the decision-making process is recorded. The action trajectory includes the initial model's current action, as well as input state information before the action and environmental feedback information after the action, forming a three-in-one event chain of action, state, and feedback. Based on the generated action trajectory information, quantified target scores are generated for the characters in the recommendation information (i.e., the characters output by the model). Using the obtained target scores, the parameters and strategies of the initial model are fine-tuned and optimized through repeated iterative updates until the model can more accurately understand and respond to complex item requests from users. By introducing the recording and analysis of action trajectory information, combined with the quantitative evaluation of target scores, fine-tuning of the model's output quality is achieved, effectively improving the e-commerce shopping guide system's ability to understand and satisfy users' personalized and multi-dimensional item requests. The target model can output recommended item information that better matches user expectations and needs, thereby improving the accuracy and efficiency of processing user item requests. Attached Figure Description

[0029] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0030] Figure 1 This is a hardware structure block diagram of a computer terminal provided according to Embodiment 1 of this application;

[0031] Figure 2 This is a flowchart of the model training method provided in Embodiment 1 of this application;

[0032] Figure 3 This is a flowchart of the data processing method provided according to Embodiment 2 of this application;

[0033] Figure 4This is a schematic diagram of the model training apparatus provided according to Embodiment 3 of this application;

[0034] Figure 5 This is a schematic diagram of a data processing apparatus provided according to Embodiment 4 of this application;

[0035] Figure 6 This is a structural block diagram of an electronic device provided according to Embodiment 5 of this application. Detailed Implementation

[0036] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0037] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0038] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0039] Example 1

[0040] According to an embodiment of this application, a model training method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0041] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a model training method is shown. Figure 1 As shown, the computer terminal (or mobile device) 10 may include a processor set 102 (the processor set 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA, and the processor set 102 may include a processor set, Figure 1 The data is illustrated using 102a, 102b, ..., 102n. A memory 104 is used for storing data, and a transmission module 106 is used for communication functions. In addition, it may include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0042] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0043] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the model training method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned model training method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0044] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0045] The display may be a touchscreen LCD display that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0046] Under the aforementioned operating environment, this application provides the following: Figure 2 The model training method shown. Figure 4 This is a flowchart of a model training method according to Embodiment 1 of this application. The model training method includes:

[0047] Step S201: Process the item request information through the initial model to obtain multiple recommendation information and action trajectory information corresponding to the multiple recommendation information. The action trajectory information consists of multiple action events, which are composed of the current action executed by the initial model, the input state information of the initial model before executing the current action, and the environmental feedback information corresponding to the current action.

[0048] Optionally, the system receives a user's item request (query), processes the request using an initial model, and generates multiple recommendations for the item request, along with corresponding action trajectory information for each recommendation. It should be noted that each recommendation corresponds to one action trajectory, which is composed of multiple action events chained together. Each action event includes at least three key components:

[0049] Current action: This refers to the immediate decision or action taken by the initial model when processing item request information. For example, in an e-commerce shopping guide scenario, the current action could be the model deciding to retrieve products with specific attributes based on the user's query, or choosing to obtain information from different data sources.

[0050] Input state information refers to the model's state before executing the current action, including but not limited to user-inputted item request information, the model's existing knowledge state, and any background information related to the current decision. Input state information provides the model with the context of the processing environment, helping it to make more reasonable decisions.

[0051] Environmental feedback information refers to the direct feedback received from the environment after the model performs the current action, regarding the effect of the action. In e-commerce scenarios, environmental feedback information can come from search results in the product database, user satisfaction assessments, or changes in task metrics (such as conversion rates). This feedback directly reflects the accuracy and effectiveness of the model's decisions and is a key data source for subsequent model optimization.

[0052] By linking these action events together to form action trajectory information, rich details are provided for subsequent model evaluation and iteration. In-depth analysis of this action trajectory information reveals the logical chain, decision path, and feedback mechanisms of the model in processing item request information, thus guiding the direction of model improvement. This enables the model to learn more efficient and accurate decision-making strategies, ultimately improving the accuracy of item request processing and the quality of recommendations.

[0053] It should be noted that the above items can be goods from e-commerce platforms, courses, tutoring materials and educational resources in the education field, or hotels, attractions and travel destinations in the travel field.

[0054] In an optional embodiment, a simulation environment can be built, including a target model, toolchain, and interaction rules, forming a controllable task execution closed loop. Based on the user's complex needs, model interaction is driven within the simulation environment, and data is continuously recorded.<state,action,observation> Action events, etc., form action trajectory information.

[0055] It's important to clarify that a toolchain refers to a series of tools and services that a model can invoke when performing tasks in its environment, such as database retrieval, attribute filtering, inventory queries, and price comparisons. These toolchains serve as the interface between the model and its environment, helping the model obtain the information needed to complete the task. Interaction rules define the logic of how the agent interacts with the toolchains in the environment, including action sequences, feedback mechanisms, and reward structures. Rules ensure that the model's decision-making process remains consistent with the task objective, while providing timely feedback and reward signals to guide the model towards better strategies.

[0056] In an optional embodiment, the initial model first receives the user's item request information, which can be in text form, such as: "I am looking for a new 2023 French door refrigerator, preferably from the brand xx, with a capacity of over 450 liters and a height not exceeding 1850 mm."

[0057] Intent understanding: Through the encoder of the initial model, the user's needs are deeply analyzed to understand key attributes such as height, capacity, brand, and time to market.

[0058] Decision planning: Based on the understood needs, the model plans the next action, for example, deciding to retrieve a refrigerator of brand xx from the product database that meets both height and capacity requirements.

[0059] Execution Action: Based on the planning decision, the decoder or condition generation module generates specific action instructions, such as constructing a query string "xx brand AND xx brand AND capacity >= 450 AND height <= 1850 AND manufacturing year = 2023", and then sending it to the product database for retrieval.

[0060] Environmental interaction: After the model performs the current action, it will obtain feedback information from the environment (such as a database). This information will be used as part of the environmental feedback information for subsequent decision adjustments.

[0061] Results evaluation and trajectory recording: Based on the retrieval results and environmental feedback, the model calculates the effect of the current action and records the input state information, current action and environmental feedback information during this process, forming part of the action trajectory.

[0062] Iterative generation of recommendation information: The above process may be repeated multiple times until recommendation information that meets the user's needs is generated.

[0063] Step S202: Based on the action trajectory information, determine the target score value corresponding to the characters in the multiple recommendation information.

[0064] Optionally, the quality of the tokens in the recommended information and their relevance to user needs can be quantitatively evaluated based on action trajectory information to obtain a corresponding target score. For example, this assesses whether the recommended information accurately reflects the user's needs. For instance, are the attributes mentioned in the user's query (brand, capacity, height, etc.) reflected in the recommended information? The overall quality of the recommended products is evaluated, including price, reviews, sales volume, and brand reputation. This ensures that the recommended information not only meets the user's basic needs but also provides high-value product options. The naturalness and fluency of the recommended information's expression are assessed, along with the use of appropriate professional terminology and expressions to enhance user trust and satisfaction. Finally, the comprehensiveness of the recommended information's coverage of multiple product options is evaluated to provide users with a more complete selection and avoid the limitations of a single recommendation.

[0065] In an optional embodiment, the matching degree can be calculated based on the action events in the action trajectory information to obtain the target score value corresponding to the character in the recommendation information.

[0066] In an alternative embodiment, a pre-trained text matching model can be used to evaluate the degree of matching to the needs, or a task-based metric model can be used to evaluate the quality of the recommendations. For any given recommendation, the scoring model can evaluate its quality character by character. For example, the text can be segmented into words or fragments, and then scores can be calculated for each. For instance, when the model generates "xx refrigerator", the scoring model will evaluate whether this phrase meets the requirements for matching to the needs.

[0067] Step S203: Based on the target score, the initial model is iteratively updated to obtain the target model, wherein the target model outputs recommended item information based on the item request input by the target object.

[0068] Optionally, the character-level target score values ​​corresponding to the recommendation information obtained in step S202 are converted into training signals that can be utilized by the model. Using the target score values ​​as reward signals, the model parameters are updated through a policy optimization algorithm. For example, an advantage function is calculated based on the target score values, and the gradient of the model parameters is calculated based on the advantage function. This gradient indicates the direction of model parameter adjustment to maximize long-term rewards (i.e., the target score values). Based on the calculated gradient, the model parameters are updated using an optimizer (such as Adam), thus obtaining the final target model.

[0069] In summary, an initial model is used to parse and process user item request information, generating multiple recommendation messages while recording the model's action trajectory information during the decision-making process. This action trajectory includes the initial model's current action, as well as input state information before the action and environmental feedback information after the action, forming a three-dimensional event chain of action, state, and feedback. Based on the generated action trajectory information, quantified target scores are generated for the characters in the recommendation information (i.e., the characters output by the model). Using the obtained target scores, the parameters and strategies of the initial model are fine-tuned and optimized through repeated iterative updates until the model can more accurately understand and respond to complex item requests from users. By introducing the recording and analysis of action trajectory information, combined with the quantitative evaluation of target scores, fine-tuning of the model's output quality is achieved, effectively improving the e-commerce shopping guide system's ability to understand and meet users' personalized and multi-dimensional item requests. The target model can output recommended item information that better matches user expectations and needs, thereby improving the accuracy and efficiency of processing user item requests.

[0070] To improve the accuracy of calculating the target score, the model training method provided in Embodiment 1 of this application determines the target score corresponding to the characters in multiple recommendation information based on the action trajectory information by: obtaining the first score corresponding to the action event in the action trajectory information; obtaining the second score corresponding to each of the multiple recommendation information; and obtaining the target score based on the first score and the second score.

[0071] Optionally, assume that the initial model outputs G recommendation information, and the action trajectory information corresponding to the i-th recommendation information includes T action events.

[0072] For each of the T action events, an immediate score is assigned based on its direct effect; this score is the first value. For example, the first value is obtained by evaluating whether the current action and the user's query meet the user's needs. The first value of an action event reflects the immediate quality of the model's decision at that point in time.

[0073] Then, the second score corresponding to the i-th recommendation is obtained, for example, calculated based on the quality of the recommendation, its match with user needs, and recommendation diversity. The first scores of the T action events are standardized to eliminate differences in units between different scoring dimensions and ensure comparability between scores. The second score of the i-th recommendation is also standardized to reflect its relative position among all recommendations. Finally, the target score is calculated based on the processed first and second scores.

[0074] For example, based on task requirements and scenario characteristics, the weights of the first and second scores can be pre-set. If immediate feedback (the first score) is more crucial in guiding the model's strategy during the early stages of training, it can be given a higher weight. As the model matures, the weight of the second score can be gradually increased to ensure that the final recommendation is strictly aligned with user needs. The target score is obtained by weighted summing of the first and second scores. In this way, the model can simultaneously focus on immediate action effects and long-term recommendation goals, thus achieving a balance in the decision-making process.

[0075] By combining the first and second scores, the model can make more robust decisions when dealing with challenging, ambiguous needs or long-tail product recommendations. Immediate feedback helps the model correct potential biases in a timely manner, while long-term feedback ensures continuous optimization of the strategy, enabling it to provide high-quality recommendations in various scenarios.

[0076] To improve the accuracy of calculating the first score, in the model training method provided in Embodiment 1 of this application, obtaining the first score corresponding to the action event in the action trajectory information includes: for any action event in the action trajectory information, calculating a first sub-score based on the item request information and the input state information in the action event; calculating a second sub-score based on the item request information, the action in the action event, and the environmental feedback information in the action event; calculating a third sub-score based on the target tool called by the initial model in the action event and the interaction rules corresponding to the target tool; and calculating the first score based on the first sub-score, the second sub-score, and the third sub-score.

[0077] Optionally, for the i-th action trajectory information corresponding to the i-th recommendation information, a first score is calculated for any action event of the i-th action trajectory information. For example, taking the t-th action event as an example, the first score is calculated based on the item request information query and the input state information in the event. The first score reflects the model's understanding and responsiveness to user needs in the current state. For example, the relevance and rationality of the retrieval task can be evaluated using a natural language model based on the user query and the preceding process to obtain the aforementioned first score.

[0078] Then, based on the item request information, the actions in the action event (such as calling a product search tool), and environmental feedback information (such as search results), a second sub-score is calculated. This second sub-score is used to evaluate the immediate effectiveness of the model's actions. For example, using a search quality scoring model, a quality score for the search results is calculated to measure the effectiveness of the search steps, which helps improve the quality of the search actions.

[0079] Furthermore, a third fractional value is calculated based on the target tool invoked by the initial model during the action event and the corresponding interaction rules of the target tool. This third fractional value is used to evaluate the compliance and effectiveness of the model when invoking the tool, particularly its ability to follow predefined interaction rules, to ensure the safety and rationality of the model's operations. For example, when invoking a product search tool, the model may need to follow a series of rules, such as "prioritizing products with sufficient stock" and "avoiding recommending items that have been marked as uninteresting by the user." The third fractional value may be calculated by checking whether the model's actions comply with the interaction rules, thus encouraging the model to make more compliant decisions in complex tasks.

[0080] Finally, the first fractional value is calculated based on the first, second, and third fractional values.

[0081] In an optional embodiment, ,in, This is the first score. The first fractional value, The second fractional value, It is the third fractional value.

[0082] The first score can comprehensively evaluate the model's performance in a single action event, including its ability to understand needs, the effectiveness of actions, and its adherence to predefined rules, thus providing richer guidance for the iterative optimization of the model.

[0083] To improve the rationality of calculating the second score, in the model training method provided in Embodiment 1 of this application, obtaining the second score values ​​corresponding to multiple recommendation information includes: for any one of the multiple recommendation information, calculating the recommendation information and item request information to obtain a fourth score sub-value; calculating based on preset rules and recommendation information to obtain a fifth score sub-value; calculating the product parameter information in the recommendation information to obtain a sixth score sub-value; and calculating based on the fourth score sub-value, fifth score sub-value, and sixth score sub-value to obtain the second score value.

[0084] Optionally, taking the i-th recommendation as an example to explain the calculation of the second score, first, the matching degree between the recommendation and the item request information is calculated to obtain the fourth sub-score. For example, a natural language model can be used to evaluate the overall quality of product recommendation copy based on the user's query and final answer.

[0085] Suppose a user query is "finding an electric toothbrush suitable for beginners". The model-generated recommendations may include descriptions and links to multiple electric toothbrushes. The fourth sub-score is calculated by comparing the degree to which each electric toothbrush's features (such as whether it includes a tutorial mode, whether it is easy to use, etc.) match the needs of beginners. Information with a higher match will receive a higher fourth sub-score, thus gaining a favorable position in the evaluation of recommendation effectiveness.

[0086] Then, based on the rule-based evaluation, rules are designed to score the final answer, ensuring that the answer conforms to the task specifications, such as including a product recommendation structure. For example, the recommendation information may need to adhere to the rule of "not recommending products whose price is more than twice the average price of the user's historical purchase records." The fifth score can be calculated by checking whether the price of the recommended product exceeds this limit. If the price of the recommended product is within a reasonable range, the fifth score will be higher; otherwise, it will be lower.

[0087] Furthermore, the task indicators for the recommended products, including the rationality of the recommendation and the existence of the product, are used as the basis for the result score of the product recommendation effect. That is, the sixth sub-value is obtained by calculating the product parameter information in the recommendation information.

[0088] Finally, the second fraction value is obtained by calculating based on the fourth, fifth, and sixth fraction values.

[0089] In an optional embodiment, ,in, This is the second score. The fourth fractional value, The fifth fractional value, It is the sixth fractional value.

[0090] The second-score calculation method can more reasonably and comprehensively evaluate the overall value of recommendation information, thereby improving the model's performance and user experience in e-commerce shopping guide scenarios.

[0091] To improve the accuracy of calculating the target score, in the model training method provided in Embodiment 1 of this application, obtaining the target score based on the first score and the second score includes: calculating the target response score based on the first score and the second score; calculating the action level score corresponding to the action event in the action trajectory information based on the target response score; and obtaining the target score based on the target response score and the action level score.

[0092] Optionally, the target response score is calculated based on the first score (instant action feedback) and the second score (comprehensive evaluation of recommendation information). In an optional embodiment, the target response score can be calculated using the following formula:

[0093]

[0094] in, The target response score, This is a discount factor. The discount factor can adjust the exploration tendency: a larger value can be set in the early stages to encourage multi-step exploration, and a smaller value can be set in the later stages to focus on aligning with the task objectives.

[0095] Then, based on the target response score, the action-level score corresponding to the action event in the action trajectory information can be further calculated. The action-level score redistributes the rating of the action event, making it more specific and helpful in guiding subsequent policy learning. For example, with a specific recommendation as the target, the action-level score for each action event is calculated through a dynamic allocation mechanism based on its target response score. For instance, the softmax function can be used to assign weights based on the first score of each action event. In the case of positive feedback (higher target response scores), the action-level score emphasizes those action events that contribute significantly; in the case of negative feedback, action events that might lead to poor recommendation performance are correspondingly downweighted.

[0096] Finally, the target score is obtained based on the target response score and the action-level score. For example, the target score can be calculated by a weighted sum of the target response score and the action-level score. The weighting coefficients can be adjusted according to the task objective and the training phase to ensure that the model simultaneously focuses on the immediate rewards of immediate feedback and the final rewards of long-term recommendation performance, thus improving the model's learning ability and decision-making quality in complex tasks.

[0097] By following the steps described above, the model's performance in e-commerce shopping guide scenarios can be evaluated more accurately. This ensures that the training process considers both immediate action quality and long-term recommendation goals, thereby improving the model's overall performance and task alignment. This multi-level score calculation strategy not only improves the accuracy of target score calculations but also enhances the model's understanding of user needs and the precision of product recommendations.

[0098] To improve the accuracy of the target response score, the model training method provided in Embodiment 1 of this application calculates the target response score based on the first score and the second score, including: for a target action event, obtaining multiple first scores for the target action event corresponding to multiple recommendation information, and performing standardized calculation based on the multiple first scores to obtain a first standardized score for the target action event, wherein the target action event is any action event in the action trajectory information; for target recommendation information, obtaining a second standardized score for the target recommendation information based on multiple second scores corresponding to multiple recommendation information, and performing standardized calculation based on the multiple second scores; calculating the initial response score corresponding to the target recommendation information based on the first standardized score, the second standardized score, and a preset discount factor; and performing standardized calculation based on the initial response score corresponding to multiple recommendation information to obtain the target response score.

[0099] Optionally, to further improve the effectiveness of subsequent training, for the action event at step t (i.e., the target action event mentioned above), the action event at step t is standardized based on multiple first scores of the target action event corresponding to multiple recommendation information, resulting in a first standardized score for the target action event. The standardization process typically includes removing the mean and dividing by the standard deviation to ensure that the scores of the action event in different recommendation information can be compared on the same scale.

[0100] For example,

[0101]

[0102] in, The first standardized score for step t, Let j represent the average of the first scores in step t among the G recommendations, and j represent the j-th recommendation. Let be the variance of the first score value in step t among the G recommendation results.

[0103] For the i-th recommendation (i.e. the target recommendation), the second score of the i-th recommendation is standardized based on the multiple second score values ​​corresponding to the multiple recommendation information to obtain the second standardized score value.

[0104] For example,

[0105]

[0106] in, This is the second standardized score. Let j be the average of the second scores of the G recommendations, and j represent the j-th recommendation. Let be the variance of the second score values ​​of the G recommendation results.

[0107] Then, based on the first standardized score, the second standardized score, and the preset discount factor, the initial response score corresponding to the target recommendation information is calculated. For example...

[0108]

[0109] Finally, the target response score is obtained by standardizing the initial response scores corresponding to multiple recommendation information. For example,

[0110]

[0111] in, Let G be the average of the initial response scores of the G recommendations. Let G be the variance of the initial response scores for the G recommendations.

[0112] Standardization ensures that scores for different recommendation information and action events are compared on the same scale, avoiding numerical instability during model training and improving training reliability and effectiveness. The use of standardization and preset discount factors enhances the efficiency and stability of model training, preventing learning oscillations caused by score differences during training, thus enabling the model to converge to a better strategy more quickly.

[0113] To improve the accuracy of action event score redistribution, the model training method provided in Embodiment 1 of this application calculates the action-level score corresponding to the action event in the action trajectory information based on the target response score, including: calculating the first weight value corresponding to the target action event based on the first standardized score value corresponding to the target action event and the first standardized score value corresponding to the action event in the action trajectory information; and calculating the action-level score corresponding to the target action event based on the initial response score value and the first weight value.

[0114] Optionally, for the t-th step action event (i.e., the target action event mentioned above), the first weight value corresponding to the target action event is calculated based on the first standardized score value corresponding to the action event and the set of first standardized scores value corresponding to all action events in the entire action trajectory information. This weight value reflects the relative importance of the target action event within the overall action sequence.

[0115] For example, ,in, This is a temperature parameter used to control the sharpness of the dispensed material.

[0116] The action-level score corresponding to the target action event is calculated based on the initial response score and the first weight value. The score contribution of each action event can be dynamically adjusted according to the positive or negative value of the target response score, thereby protecting actions that perform well even in failed trajectories from penalties, while reinforcing the importance of key actions in successful trajectories.

[0117] For example, the action level score corresponding to the target action event can be calculated using the following formula:

[0118] If the recommendation information corresponding to the target action event is a positive example (i.e., the target response score is greater than or equal to 0), then the action level score is:

[0119] .

[0120] If the recommendation information corresponding to the target action event is a negative example (i.e., the target response score is less than 0), then the action level score is:

[0121]

[0122]

[0123] in, The action level score corresponding to the target action event.

[0124] The first weight value, calculated using the first standardized score and the target response score, more accurately reflects the actual contribution of each action event to the final recommendation effect, avoiding the "collective punishment" of high-quality sub-processes in traditional reinforcement training. In positive feedback scenarios, the first weight value is used as a multiplier, giving more rewards to action events that play a key role in the success trajectory, thus promoting the model's learning to make better choices in the decision-making process.

[0125] Obtaining the target score corresponding to a character is crucial. In the model training method provided in Embodiment 1 of this application, the target score is obtained based on the target response score and the action level score, including: for any character in the target recommendation information, if the character is the last character of the target action event in the action trajectory information corresponding to the target recommendation information, then the initial score corresponding to the character is determined based on the action level score of the target action event; if the character is the last character in the target recommendation information, then the initial score corresponding to the character is determined based on the target response score; if the character is neither the last character in the target recommendation information nor the last character of the target action event in the action trajectory information corresponding to the target recommendation information, then the initial score corresponding to the character is determined based on a preset value; and the target score is obtained based on the initial score corresponding to the character.

[0126] Optionally, for the recommendation information to be a specific sequence of a certain length output by the initial model, the initial score for any character token in the sequence can be determined according to the following rules:

[0127] If a token is the last token of an action event, its initial score is determined based on the action-level score corresponding to that action event. This rule ensures that the immediate feedback for each specific action is directly associated with the ending character of that action, prompting the model to learn the correct immediate decision.

[0128] If the token is the last token in the target recommendation information, its initial score is determined based on the target response score. This rule considers the overall effect of the recommendation information, ensuring that the model focuses not only on the quality of immediate decisions but also on the comprehensive performance of the final recommendation information.

[0129] For tokens that are neither the last token of the target action event nor the last token of the target recommendation information, their initial scores are determined based on a preset value. The preset value can be a small positive number or zero, used to avoid these characters being completely ignored in the calculation, while ensuring that they do not incorrectly affect the training process.

[0130] Finally, the target score is obtained based on the initial score corresponding to the character.

[0131] In an optional embodiment, the initial score is obtained using the following formula:

[0132]

[0133] in, The length of the token sequence corresponding to the recommendation information. This is the initial score for the entire token sequence. This is the initial score for the k-th character in the token sequence.

[0134] By assigning scores to characters, especially with special handling for characters ending action events and recommendation information, the model can gain a more nuanced understanding of the details in the recommendation generation process, avoiding the coarseness that may result from traditional training methods based on sequences or overall responses. In scenarios such as e-commerce shopping guides, user needs often require multiple steps of reasoning and decision-making to be met. The above method ensures that the model receives accurate feedback at each decision step, thereby better optimizing the entire decision-making chain and improving recommendation efficiency and quality.

[0135] To prevent the model generation from deviating from the trajectory of the initial strategy, in the model training method provided in Embodiment 1 of this application, the target score is obtained based on the initial score value corresponding to the character by: calculating the divergence penalty value corresponding to the character based on the first probability value of the character output by the initial model under the current strategy and the second probability value of the character output under the historical strategy; and calculating the target score value based on the divergence penalty value and the initial score value.

[0136] Optionally, in the context of Reinforcement Learning (RL), especially when training large language models for complex decision-making tasks, maintaining the continuity of the model's policy and preventing the policy from deviating from its initial settings is crucial. If the model excessively pursues high rewards while neglecting policy consistency during text generation or action taking, the generated trajectory may deviate from the expected behavioral pattern, thus affecting the stability and reliability of the recommender system. To address this issue, Embodiment 1 of this application provides a method for introducing a Kullback-Leibler (KL) divergence penalty value to prevent the model from generating trajectories that deviate from the initial policy.

[0137] For the characters generated by the model under the current strategy The algorithm calculates the first probability value of the character under the current policy and the second probability value under the historical policy. Based on the first probability value of the character output by the initial model under the current policy and the second probability value of the character output under the historical policy, it calculates the divergence penalty value corresponding to the character to quantify the difference between it and the historical policy. After obtaining the KL divergence penalty value for each character, it needs to be fused with the initial score value to obtain the final target score value.

[0138] In an optional embodiment, the final target score is calculated using the following formula:

[0139]

[0140] in, The target score for the k-th token.

[0141] Let KL divergence penalty value be used. This represents the probability of the action under the current strategy (i.e., the first probability value). This represents the action probability under the historical strategy (i.e., the second probability value). This is the penalty coefficient.

[0142] The divergence penalty term encourages the model to maintain continuity with historical policies while exploring higher-reward strategies, avoiding instability and unpredictability caused by drastic policy changes. By avoiding drastic policy fluctuations and ineffective exploration, this mechanism improves training efficiency, enabling the model to converge to a stable and high-performing policy more quickly.

[0143] To improve the training effect of the model, in the model training method provided in Embodiment 1 of this application, the initial model is iteratively updated based on the target score value to obtain the target model, including: calculating the cumulative advantage function based on the target score values ​​of multiple characters in the target recommendation information; for the target character in the target recommendation information, calculating the second weight value based on the third probability value of the target character output by the initial model under the current strategy, the fourth probability value of the target character output under the historical strategy, and the total number of characters in the action event in which the target character is located; and iteratively updating the initial model based on the cumulative advantage function, the second weight value, and the preset update threshold to obtain the target model.

[0144] Optionally, a cumulative advantage function is calculated based on the target scores of multiple characters in the target recommendation information. The cumulative advantage function measures the contribution of a character to the overall recommendation effect and is used to guide model updates.

[0145] For the target character in the target recommendation information, the second weight value is calculated based on the third probability value of the target character output by the initial model under the current strategy, the fourth probability value of the target character output under the historical strategy, and the total number of characters in the action event in which the target character is located. That is, for each step, the geometric mean of the probability ratio of its internal tokens is taken as the importance sampling of that step, so as to avoid the explosion / disappearance of the ratio across tokens and unify the optimization direction at the step level.

[0146] Finally, the initial model is iteratively updated based on the cumulative advantage function, the second weight value, and the preset update threshold to obtain the target model.

[0147] In an optional embodiment, the cumulative advantage function is: The second weight value is calculated using the following formula:

[0148]

[0149] Where θ represents the model parameters. This represents the importance sampling weight of the i-th sequence at step t. This represents the probability of an action under the old strategy. This represents the k-th token in step t. This represents the number of tokens included in step t.

[0150] In an optional embodiment, the optimization objective is:

[0151]

[0152] in, When, it indicates the action currently being performed. The generated token is better than other actions, therefore it needs to be improved. However, considering the possibility of insufficient sampling, and The distribution difference cannot be too large, therefore There is an upper limit; one cannot blindly believe everything. However, it has been improved to an excessive degree. Assuming this ratio At that time, this ratio is clipped and fixed at... .

[0153] when When, it indicates the action currently being performed. It's worse than other movements, so it needs to be reduced. However, considering the possibility of insufficient sampling, and The distribution difference cannot be too large, therefore There is a lower limit; one cannot blindly believe everything. However, excessive reduction Assuming this ratio At that time, this ratio is clipped and fixed at... .

[0154] By controlling the update magnitude and direction, drastic changes in policy distribution are avoided, thus improving the stability of model training.

[0155] In one optional embodiment, a user initiates a query: "Looking for a 2025 model refrigerator from brand XX, priced below 5000 yuan, and with energy-saving certification." This query includes multi-dimensional attribute requirements and brand preferences. The model begins constructing an action trajectory, including: intent recognition: identifying the core needs and preferences in the user's query; product retrieval: retrieving potentially matching products from the product database based on the identified needs; attribute verification: checking the attributes of the retrieved products to ensure all conditions are met; and recommendation copy generation: generating recommendation copy for the matching products, including product name, price, and feature descriptions. For each action event in the trajectory, a standardized score is calculated. For example, LLM-based assessment evaluates the match between the product and the user's needs, while rule-based assessment ensures the rationality and professionalism of the recommendations. The calculation of standardized scores ensures the comparability and consistency of scores from different assessment sources.

[0156] For each action event in the trajectory (e.g., step t), a first weight value is calculated. This weight is based on the standardized score of the action event and the global standardized score, and the sharpness of the assignment is controlled by a temperature parameter. Using this first weight, an action-level score can be calculated for each action event, reflecting its contribution to the final recommendation effect. Different formulas are used to calculate action-level scores for positive and negative examples, ensuring that high-quality actions are reinforced while low-quality actions are penalized.

[0157] For each character in the recommendation information, its first probability value under the current policy and its second probability value under historical policies are calculated to obtain the divergence penalty value. Combined with the action-level score, a target score is calculated for each character, ensuring that the training signal considers both the quality of immediate actions and the coherence and policy consistency of the entire recommendation information.

[0158] Based on the target score and cumulative advantage function, a second weight value is calculated to adjust the direction and magnitude of model updates. This second weight value considers the model's action probabilities under the current and historical strategies, as well as the number of characters in the action events. A preset update threshold controls the update range, avoiding drastic changes in strategy distribution and ensuring the stability and efficiency of model iteration. Through this process, the model gradually learns and optimizes its decision-making process, ultimately not only accurately understanding and meeting users' diverse and complex query needs, but also ensuring that recommended products are available on the platform, and that the recommendation text is professional, engaging, and enhances the user experience.

[0159] In the model training method provided in Embodiment 1 of this application, the item request information is processed by an initial model to obtain multiple recommendation information and action trajectory information corresponding to each recommendation information. The action trajectory information consists of multiple action events, which are respectively composed of the current action executed by the initial model, the input state information of the initial model before executing the current action, and the environmental feedback information corresponding to the current action. Based on the action trajectory information, the target score value corresponding to the characters in the multiple recommendation information is determined. Based on the target score value, the initial model is iteratively updated to obtain the target model. The target model outputs recommended item information based on the item request input by the target object, which solves the technical problem of low accuracy of the model in processing user item requests in related technologies.

[0160] In this application, an initial model is used to parse and process user item request information, thereby generating multiple recommendation messages. Simultaneously, the model's action trajectory information during the decision-making process is recorded. The action trajectory includes the initial model's current action, as well as input state information before the action and environmental feedback information after the action, forming a three-in-one event chain of action, state, and feedback. Based on the generated action trajectory information, quantified target scores are generated for the characters in the recommendation information (i.e., the characters output by the model). Using the obtained target scores, the parameters and strategies of the initial model are fine-tuned and optimized through repeated iterative updates until the model can more accurately understand and respond to complex item requests from users. By introducing the recording and analysis of action trajectory information, combined with the quantitative evaluation of target scores, fine-tuning of the model's output quality is achieved, effectively improving the e-commerce shopping guide system's ability to understand and satisfy users' personalized and multi-dimensional item requests. The target model can output recommended item information that better matches user expectations and needs, thereby improving the accuracy and efficiency of processing user item requests.

[0161] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0163] Example 2

[0164] According to embodiments of this application, a data processing method is also provided, such as... Figure 3 As shown, the data processing method includes:

[0165] Step S301: Receive the target item request information input by the target object;

[0166] Step S302: Process the target item request information through the target model to obtain recommended item information, wherein the target model is trained using any of the above-mentioned model training methods;

[0167] Step S303: Return the recommended item information to the target object.

[0168] Optionally, when a user submits a query through the search box or other interactive interface of an e-commerce platform, the system receives the user's input query as item request information. The target model first parses the user query to identify key needs, such as brand, attributes, and price range. Based on the parsed information, the model calls a product retrieval tool to filter a list of products that meet the criteria from the product database. It can also continuously verify the matching degree between product attributes and user needs during the recommendation process and dynamically adjust its decision-making strategy based on process and result monitoring signals to ensure the accuracy of the recommendations. The generated recommended item information is then presented to the user. This information includes not only a specific list of products but also detailed reasons for recommendation and key attribute information for each product. For example: "Recommended: XX brand refrigerator, model XX, depth less than 600mm, dual-system cooling, new product in 2025, meets all your requirements."

[0169] By using the training method proposed in Example 1, the target model can more accurately understand and respond to the subtleties of user needs, avoid recommending products that do not meet the needs, and generate attractive and professional recommendation text to improve the user experience.

[0170] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0171] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0172] Example 3

[0173] According to embodiments of this application, a model training apparatus for implementing the above-described model training method is also provided, such as... Figure 4 As shown, the device includes: a first processing unit 401, a determination unit 402, and an update unit 403.

[0174] The first processing unit 401 is used to process the item request information through the initial model to obtain multiple recommendation information and action trajectory information corresponding to the multiple recommendation information. The action trajectory information consists of multiple action events, and the multiple action events consist of the current action executed by the initial model, the input state information of the initial model before executing the current action, and the environmental feedback information corresponding to the current action.

[0175] The determining unit 402 is used to determine the target score value corresponding to the characters in multiple recommendation information based on the action trajectory information;

[0176] The update unit 403 is used to iteratively update the initial model based on the target score to obtain the target model, wherein the target model outputs recommended item information based on the item request input by the target object.

[0177] In the model training method provided in Embodiment 3 of this application, the first processing unit 401 processes the item request information through an initial model to obtain multiple recommendation information and action trajectory information corresponding to each recommendation information. The action trajectory information consists of multiple action events, which are respectively composed of the current action executed by the initial model, the input state information of the initial model before executing the current action, and the environmental feedback information corresponding to the current action. The determining unit 402 determines the target score value corresponding to the characters in the multiple recommendation information based on the action trajectory information. The updating unit 403 iteratively updates the initial model based on the target score value to obtain the target model. The target model outputs recommended item information based on the item request input by the target object, which solves the technical problem of low accuracy of the model in processing user item requests in related technologies.

[0178] In this application, an initial model is used to parse and process user item request information, thereby generating multiple recommendation messages. Simultaneously, the model's action trajectory information during the decision-making process is recorded. The action trajectory includes the initial model's current action, as well as input state information before the action and environmental feedback information after the action, forming a three-in-one event chain of action, state, and feedback. Based on the generated action trajectory information, quantified target scores are generated for the characters in the recommendation information (i.e., the characters output by the model). Using the obtained target scores, the parameters and strategies of the initial model are fine-tuned and optimized through repeated iterative updates until the model can more accurately understand and respond to complex item requests from users. By introducing the recording and analysis of action trajectory information, combined with the quantitative evaluation of target scores, fine-tuning of the model's output quality is achieved, effectively improving the e-commerce shopping guide system's ability to understand and satisfy users' personalized and multi-dimensional item requests. The target model can output recommended item information that better matches user expectations and needs, thereby improving the accuracy and efficiency of processing user item requests.

[0179] Optionally, in the model training method provided in Embodiment 3 of this application, the determining unit includes: a first acquisition subunit, used to acquire the first score value corresponding to the action event in the action trajectory information; a second acquisition subunit, used to acquire the second score value corresponding to multiple recommendation information respectively; and a determining subunit, used to obtain the target score value based on the first score value and the second score value.

[0180] Optionally, in the model training method provided in Embodiment 3 of this application, the first acquisition subunit includes: a first calculation module, used to calculate a first fractional value for any action event in the action trajectory information based on the item request information and the input state information in the action event; a second calculation module, used to calculate a second fractional value based on the item request information, the action in the action event, and the environmental feedback information in the action event; a third calculation module, used to calculate a third fractional value based on the target tool called by the initial model in the action event and the interaction rules corresponding to the target tool; and a fourth calculation module, used to calculate a first score based on the first fractional value, the second fractional value, and the third fractional value.

[0181] Optionally, in the model training method provided in Embodiment 3 of this application, the second acquisition subunit includes: a fifth calculation module, used to calculate the recommendation information and item request information for any one of the multiple recommendation information to obtain a fourth fractional value; a sixth calculation module, used to calculate the fifth fractional value based on preset rules and recommendation information; a seventh calculation module, used to calculate the product parameter information in the recommendation information to obtain a sixth fractional value; and an eighth calculation module, used to calculate the second score based on the fourth fractional value, the fifth fractional value, and the sixth fractional value.

[0182] Optionally, in the model training method provided in Embodiment 3 of this application, the determining sub-unit includes: a ninth calculation module, used to calculate based on the first score and the second score to obtain the target response score; a tenth calculation module, used to calculate based on the target response score to obtain the action level score corresponding to the action event in the action trajectory information; and a determining module, used to obtain the target score based on the target response score and the action level score.

[0183] Optionally, in the model training method provided in Embodiment 3 of this application, the ninth calculation module includes: a first calculation submodule, used to obtain multiple first scores of the target action event corresponding to multiple recommendation information for the target action event, and perform standardized calculation based on the multiple first scores to obtain a first standardized score of the target action event, wherein the target action event is any action event in the action trajectory information; a second calculation submodule, used to obtain a second standardized score of the target recommendation information based on multiple second scores corresponding to multiple recommendation information, and perform standardized calculation based on the multiple second scores; a third calculation submodule, used to calculate based on the first standardized score, the second standardized score, and a preset discount factor to obtain an initial response score corresponding to the target recommendation information; and a fourth calculation submodule, used to perform standardized calculation based on the initial response scores corresponding to multiple recommendation information to obtain a target response score.

[0184] Optionally, in the model training method provided in Embodiment 3 of this application, the tenth calculation module includes: a fifth calculation submodule, used to calculate based on the first standardized score value corresponding to the target action event and the first standardized score value corresponding to the action event in the action trajectory information to obtain the first weight value corresponding to the target action event; and a sixth calculation submodule, used to calculate based on the initial response score value and the first weight value to obtain the action level score corresponding to the target action event.

[0185] Optionally, in the model training method provided in Embodiment 3 of this application, the determining module includes: a first determining submodule, used to determine the initial score value corresponding to any character in the target recommendation information if the character is the last character of the target action event in the action trajectory information corresponding to the target recommendation information, based on the action level score of the target action event; a second determining submodule, used to determine the initial score value corresponding to the character if the character is the last character in the target recommendation information, based on the target response score value; a third determining submodule, used to determine the initial score value corresponding to the character based on a preset value if the character is neither the last character in the target recommendation information nor the last character of the target action event in the action trajectory information corresponding to the target recommendation information; and a fourth determining submodule, used to obtain the target score value based on the initial score value corresponding to the character.

[0186] Optionally, in the model training method provided in Embodiment 3 of this application, the fourth determining submodule includes: a first calculation submodule, used to calculate the divergence penalty value corresponding to the character based on the first probability value of the character output by the initial model under the current policy and the second probability value of the character output under the historical policy; and a second calculation submodule, used to calculate the target score value based on the divergence penalty value and the initial score value.

[0187] Optionally, in the model training method provided in Embodiment 3 of this application, the update unit includes: a first calculation subunit, used to calculate the cumulative advantage function based on the target scores of multiple characters in the target recommendation information; a second calculation subunit, used to calculate the second weight value for the target character in the target recommendation information based on the third probability value of the target character output by the initial model under the current strategy, the fourth probability value of the target character output under the historical strategy, and the total number of characters in the action event in which the target character is located; and iteratively updating the initial model based on the cumulative advantage function, the second weight value, and a preset update threshold to obtain the target model.

[0188] It should be noted that the first processing unit 401, the determining unit 402, and the updating unit 403 mentioned above correspond to steps S201 to S203 in Embodiment 1. The three units and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the device, can run on the computer terminal 10 provided in Embodiment 1.

[0189] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0190] Example 4

[0191] According to embodiments of this application, a data processing apparatus for implementing the above-described data processing method is also provided, such as... Figure 5 As shown, the device includes: a receiving unit 501, a second processing unit 502, and a return unit 503.

[0192] The receiving unit 501 is used to receive the target item request information input by the target object;

[0193] The second processing unit 502 is used to process the target item request information through the target model to obtain recommended item information, wherein the target model is trained using any of the above-mentioned model training methods;

[0194] Return unit 503 is used to return the recommended item information to the target object.

[0195] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 2, but are not limited to the schemes provided in Embodiment 2.

[0196] Example 5

[0197] Embodiments of this application may provide an electronic device, which may be any one of a group of electronic device terminals. Optionally, in this embodiment, the aforementioned electronic device may also be replaced by a terminal device such as a mobile terminal.

[0198] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0199] In this embodiment, the aforementioned electronic device can execute the program code for the following steps in the model training method and data processing method: processing the item request information through the initial model to obtain multiple recommendation information and action trajectory information corresponding to the multiple recommendation information, wherein the action trajectory information consists of multiple action events, and the multiple action events consist of the current action executed by the initial model, the input state information of the initial model before executing the current action, and the environmental feedback information corresponding to the current action; determining the target score value corresponding to the characters in the multiple recommendation information based on the action trajectory information; iteratively updating the initial model based on the target score value to obtain the target model, wherein the target model outputs recommended item information based on the item request input by the target object.

[0200] The aforementioned electronic device can execute the following steps in the model training method and data processing method: determining the target score value corresponding to the characters in multiple recommendation information based on the action trajectory information, including: obtaining the first score value corresponding to the action event in the action trajectory information; obtaining the second score value corresponding to each of the multiple recommendation information; and obtaining the target score value based on the first score value and the second score value.

[0201] The aforementioned electronic device can execute the following steps in the model training method and data processing method: obtaining the first score value corresponding to the action event in the action trajectory information includes: for any action event in the action trajectory information, calculating based on the item request information and the input state information in the action event to obtain the first score value; calculating based on the item request information, the action in the action event, and the environmental feedback information in the action event to obtain the second score value; calculating based on the target tool called by the initial model in the action event and the interaction rules corresponding to the target tool to obtain the third score value; and calculating based on the first score value, the second score value, and the third score value to obtain the first score value.

[0202] The aforementioned electronic device can execute the following steps in the model training method and data processing method: obtaining the second score corresponding to multiple recommendation information includes: for any one of the multiple recommendation information, calculating the recommendation information and item request information to obtain a fourth score; calculating based on preset rules and recommendation information to obtain a fifth score; calculating the product parameter information in the recommendation information to obtain a sixth score; and calculating based on the fourth score, fifth score, and sixth score to obtain the second score.

[0203] The aforementioned electronic device can execute the program code for the following steps in the model training method and data processing method: obtaining the target score based on the first score and the second score includes: calculating the target response score based on the first score and the second score; calculating the action level score corresponding to the action event in the action trajectory information based on the target response score; and obtaining the target score based on the target response score and the action level score.

[0204] The aforementioned electronic device can execute the following steps in the model training method and data processing method: Calculating the target response score based on the first and second scores includes: for a target action event, obtaining multiple first scores for the target action event corresponding to multiple recommendation information, and performing standardized calculations based on the multiple first scores to obtain a first standardized score for the target action event, wherein the target action event is any action event in the action trajectory information; for target recommendation information, obtaining a second standardized score for the target recommendation information based on multiple second scores corresponding to multiple recommendation information, and performing standardized calculations based on the multiple second scores; calculating the initial response score corresponding to the target recommendation information based on the first standardized score, the second standardized score, and a preset discount factor; and performing standardized calculations based on the initial response scores corresponding to multiple recommendation information to obtain the target response score.

[0205] The aforementioned electronic device can execute the following steps in the model training method and data processing method: Calculate the action level score corresponding to the action event in the action trajectory information based on the target response score, including: calculate the first weight value corresponding to the target action event based on the first standardized score value corresponding to the target action event and the first standardized score value corresponding to the action event in the action trajectory information; calculate the action level score corresponding to the target action event based on the initial response score value and the first weight value.

[0206] The aforementioned electronic device can execute the following steps in the model training method and data processing method: Obtaining the target score based on the target response score and action level score includes: for any character in the target recommendation information, if the character is the last character of the target action event in the action trajectory information corresponding to the target recommendation information, then determine the initial score corresponding to the character based on the action level score of the target action event; if the character is the last character in the target recommendation information, then determine the initial score corresponding to the character based on the target response score; if the character is neither the last character in the target recommendation information nor the last character of the target action event in the action trajectory information corresponding to the target recommendation information, then determine the initial score corresponding to the character based on a preset value; and obtaining the target score based on the initial score corresponding to the character.

[0207] The aforementioned electronic device can execute the following steps in the model training method and data processing method: obtaining the target score based on the initial score corresponding to the character includes: calculating the divergence penalty value corresponding to the character based on the first probability value of the character output by the initial model under the current policy and the second probability value of the character output under the historical policy; and calculating the target score based on the divergence penalty value and the initial score.

[0208] The aforementioned electronic device can execute the following steps in the model training method and data processing method: Iteratively updating the initial model based on the target score to obtain the target model includes: calculating the cumulative advantage function based on the target scores of multiple characters in the target recommendation information; for the target character in the target recommendation information, calculating the second weight value based on the third probability value of the target character output by the initial model under the current policy, the fourth probability value of the target character output under the historical policy, and the total number of characters in the action event in which the target character is located; iteratively updating the initial model based on the cumulative advantage function, the second weight value, and a preset update threshold to obtain the target model.

[0209] The aforementioned electronic device can execute the following steps in the model training method and data processing method: receiving target item request information input by the target object; processing the target item request information through the target model to obtain recommended item information, wherein the target model is trained using any of the above-mentioned model training methods; and returning the recommended item information to the target object.

[0210] Optionally, Figure 6 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 6 As shown, the electronic device 60 may include: one or more ( Figure 6 (Only one is shown in the image) Processor 602 and memory 604. The electronic device 60 may also include a memory controller to control and manage the memory 604; the electronic device 60 may also include a peripheral interface to connect to a radio frequency module, an audio module, and a display screen, etc.

[0211] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the model training method and data processing method and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned model training method and data processing method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the electronic device 20 via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0212] The processor can access the information and application program stored in the memory via a transmission device to perform the following steps: Processing the item request information using an initial model to obtain multiple recommendations and corresponding action trajectory information for each recommendation. The action trajectory information consists of multiple action events, each composed of the current action executed by the initial model, the input state information of the initial model before executing the current action, and the environmental feedback information corresponding to the current action. Based on the action trajectory information, determining the target score value corresponding to the characters in the multiple recommendations. Iteratively updating the initial model based on the target score value to obtain a target model, whereby the target model outputs recommended item information based on the item request input by the target object.

[0213] Optionally, the processor may also execute program code that performs the following steps: determining the target score value corresponding to the characters in multiple recommendation information based on the action trajectory information, including: obtaining the first score value corresponding to the action event in the action trajectory information; obtaining the second score value corresponding to each of the multiple recommendation information; and obtaining the target score value based on the first score value and the second score value.

[0214] Optionally, the processor may also execute program code with the following steps: obtaining the first score value corresponding to the action event in the action trajectory information includes: for any action event in the action trajectory information, calculating based on the item request information and the input state information in the action event to obtain the first score value; calculating based on the item request information, the action in the action event, and the environmental feedback information in the action event to obtain the second score value; calculating based on the target tool called by the initial model in the action event and the interaction rules corresponding to the target tool to obtain the third score value; and calculating based on the first score value, the second score value, and the third score value to obtain the first score value.

[0215] Optionally, the processor may also execute program code that performs the following steps: obtaining the second score corresponding to multiple recommendation information includes: for any one of the multiple recommendation information, calculating the recommendation information and item request information to obtain a fourth score; calculating based on preset rules and recommendation information to obtain a fifth score; calculating the product parameter information in the recommendation information to obtain a sixth score; and calculating based on the fourth score, fifth score, and sixth score to obtain the second score.

[0216] Optionally, the processor may also execute program code that performs the following steps: obtaining a target score based on a first score and a second score, including: calculating a target response score based on the first score and the second score; calculating an action level score corresponding to the action event in the action trajectory information based on the target response score; and obtaining the target score based on the target response score and the action level score.

[0217] Optionally, the processor may also execute program code with the following steps: calculating the target response score based on the first score and the second score, including: for a target action event, obtaining multiple first scores for the target action event corresponding to multiple recommendation information, and performing standardized calculation based on the multiple first scores to obtain a first standardized score for the target action event, wherein the target action event is any action event in the action trajectory information; for target recommendation information, obtaining a second standardized score for the target recommendation information based on multiple second scores corresponding to multiple recommendation information, and performing standardized calculation based on the multiple second scores; calculating the initial response score corresponding to the target recommendation information based on the first standardized score, the second standardized score, and a preset discount factor; and performing standardized calculation based on the initial response score corresponding to multiple recommendation information to obtain the target response score.

[0218] Optionally, the processor may also execute program code that performs the following steps: calculating the action level score corresponding to the action event in the action trajectory information based on the target response score, including: calculating the first weight value corresponding to the target action event based on the first standardized score value corresponding to the target action event and the first standardized score value corresponding to the action event in the action trajectory information; and calculating the action level score corresponding to the target action event based on the initial response score value and the first weight value.

[0219] Optionally, the processor may also execute program code with the following steps: obtaining the target score based on the target response score and action level score includes: for any character in the target recommendation information, if the character is the last character of the target action event in the action trajectory information corresponding to the target recommendation information, then determine the initial score corresponding to the character based on the action level score of the target action event; if the character is the last character in the target recommendation information, then determine the initial score corresponding to the character based on the target response score; if the character is neither the last character in the target recommendation information nor the last character of the target action event in the action trajectory information corresponding to the target recommendation information, then determine the initial score corresponding to the character based on a preset value; and obtaining the target score based on the initial score corresponding to the character.

[0220] Optionally, the processor may also execute program code with the following steps: obtaining the target score based on the initial score corresponding to the character includes: calculating the divergence penalty value corresponding to the character based on the first probability value of the character output by the initial model under the current policy and the second probability value of the character output under the historical policy; and calculating the target score based on the divergence penalty value and the initial score.

[0221] Optionally, the processor may also execute program code with the following steps: Iteratively updating the initial model based on the target score to obtain the target model includes: calculating the cumulative advantage function based on the target scores of multiple characters in the target recommendation information; for the target character in the target recommendation information, calculating the second weight value based on the third probability value of the target character output by the initial model under the current policy, the fourth probability value of the target character output under the historical policy, and the total number of characters in the action event in which the target character is located; iteratively updating the initial model based on the cumulative advantage function, the second weight value, and a preset update threshold to obtain the target model.

[0222] Optionally, the processor may also execute program code that performs the following steps: receiving target item request information input by the target object; processing the target item request information through the target model to obtain recommended item information, wherein the target model is trained using any of the above-mentioned model training methods; and returning the recommended item information to the target object.

[0223] Those skilled in the art will understand that Figure 6 The structure shown is for illustrative purposes only. Electronic device 60 can also be a smartphone, tablet computer, handheld computer, mobile internet device (MID), PAD and other terminal devices. Figure 6 This does not limit the structure of the aforementioned electronic device. For example, electronic device 60 may also include components that are more... Figure 6 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 6 The different configurations shown.

[0224] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0225] Example 6

[0226] Embodiments of this application also provide a computer program product. Optionally, in this embodiment, the computer program product can be used to store the program code executed by the model training method and data processing method provided in Embodiment 1.

[0227] Optionally, in this embodiment, the computer program product may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0228] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0229] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0230] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0231] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0232] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0233] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0234] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A model training method, characterized in that, include: The initial model processes the item request information to obtain multiple recommendation information and the action trajectory information corresponding to the multiple recommendation information. The action trajectory information consists of multiple action events, which are respectively composed of the current action executed by the initial model, the input state information of the initial model before executing the current action, and the environmental feedback information corresponding to the current action. Based on the action trajectory information, the target score value corresponding to the character in the plurality of recommendation information is determined; Based on the target score, the initial model is iteratively updated to obtain the target model, wherein the target model outputs recommended item information based on the item request input by the target object.

2. The method according to claim 1, characterized in that, Based on the action trajectory information, the target score value corresponding to the character in the plurality of recommendation information is determined as follows: Obtain the first score value corresponding to the action event in the action trajectory information; Obtain the second score value corresponding to each of the multiple recommendation information; The target score is obtained based on the first score and the second score.

3. The method according to claim 2, characterized in that, Obtaining the first score value corresponding to the action event in the action trajectory information includes: For any action event in the action trajectory information, a first fractional value is obtained by calculating based on the item request information and the input state information in the action event; The second fractional value is calculated based on the item request information, the action in the action event, and the environmental feedback information of the action event; The third fractional value is calculated based on the target tool invoked by the initial model in the action event and the interaction rules corresponding to the target tool. The first fraction value is obtained by calculating based on the first fraction value, the second fraction value, and the third fraction value.

4. The method according to claim 2, characterized in that, Obtaining the second score corresponding to each of the multiple recommendation information items includes: For any one of the multiple recommendation information, the fourth fractional value is calculated by combining the recommendation information and the item request information. The fifth fractional value is obtained by calculating based on preset rules and the recommended information; The sixth fractional value is obtained by calculating the product parameter information in the recommended information; The second fractional value is obtained by calculating based on the fourth, fifth, and sixth fractional values.

5. The method according to claim 2, characterized in that, Based on the first score and the second score, the target score is obtained as follows: The target response score is obtained by calculating based on the first score and the second score. Based on the target response score, the action level score corresponding to the action event in the action trajectory information is calculated. The target score is obtained based on the target response score and the action level score.

6. The method according to claim 5, characterized in that, The target response score is calculated based on the first score and the second score, including: For a target action event, multiple first scores of the target action event corresponding to the multiple recommendation information are obtained, and standardized calculation is performed based on the multiple first scores to obtain the first standardized score of the target action event, wherein the target action event is any one of the action trajectory information; For the target recommendation information, the second standardized score of the target recommendation information is obtained by standardizing the calculation based on the multiple second score values ​​corresponding to the multiple recommendation information. The initial response score corresponding to the target recommendation information is obtained by calculating based on the first standardized score, the second standardized score, and the preset discount factor. The target response score is obtained by standardizing the initial response scores corresponding to the multiple recommendation information.

7. The method according to claim 6, characterized in that, The action level score corresponding to the action event in the action trajectory information is calculated based on the target response score, including: The first weight value corresponding to the target action event is calculated based on the first standardized score value corresponding to the target action event and the first standardized score value corresponding to the action event in the action trajectory information. The action level score corresponding to the target action event is obtained by calculating based on the initial response score and the first weight value.

8. The method according to claim 5, characterized in that, Based on the target response score and the action level score, the target score is obtained as follows: For any character in the target recommendation information, if the character is the last character of the target action event in the action trajectory information corresponding to the target recommendation information, then the initial score value corresponding to the character is determined based on the action level score of the target action event; If the character is the last character in the target recommendation information, then the initial score corresponding to the character is determined based on the target response score. If the character is not the last character in the target recommendation information and is not the last character in the target action event in the action trajectory information corresponding to the target recommendation information, then the initial score value corresponding to the character is determined according to the preset value. The target score is obtained based on the initial score corresponding to the character.

9. The method according to claim 8, characterized in that, Based on the initial score corresponding to the character, the target score is obtained as follows: The divergence penalty value corresponding to the character is calculated based on the first probability value of the character output under the current policy and the second probability value of the character output under the historical policy, according to the initial model. The target score is obtained by calculating based on the divergence penalty value and the initial score.

10. The method according to claim 1, characterized in that, Based on the target score, the initial model is iteratively updated to obtain the target model, which includes: The cumulative advantage function is calculated based on the target score values ​​of multiple characters in the target recommendation information. For the target character in the target recommendation information, the second weight value is calculated based on the third probability value of the target character output by the initial model under the current strategy, the fourth probability value of the target character output under the historical strategy, and the total number of characters in the action event in which the target character is located. Based on the cumulative advantage function, the second weight value, and the preset update threshold, the initial model is iteratively updated to obtain the target model.

11. A data processing method, characterized in that, include: Receive the target item request information input by the target object; The target item request information is processed by the target model to obtain recommended item information, wherein the target model is trained using the model training method described in any one of claims 1 to 10; The recommended item information is returned to the target object.

12. A model training device, characterized in that, include: The first processing unit is used to process the item request information through an initial model to obtain multiple recommendation information and action trajectory information corresponding to the multiple recommendation information respectively. The action trajectory information consists of multiple action events, and the multiple action events are respectively composed of the current action executed by the initial model, the input state information of the initial model before executing the current action, and the environmental feedback information corresponding to the current action. The determining unit is used to determine the target score value corresponding to the character in the plurality of recommendation information based on the action trajectory information; The update unit is used to iteratively update the initial model based on the target score to obtain the target model, wherein the target model outputs recommended item information based on the item request input by the target object.

13. A data processing apparatus, characterized in that, include: The receiving unit is used to receive the target item request information input by the target object; The second processing unit is used to process the target item request information through the target model to obtain recommended item information, wherein the target model is trained using the model training method described in any one of claims 1 to 10; The return unit is used to return the recommended item information to the target object.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein when the program is executed, it controls the device on which the storage medium is located to perform the model training method of any one of claims 1 to 10, or the data processing method of claim 11.

15. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the model training method according to any one of claims 1 to 10, or the data processing method according to claim 11.

16. A computer program product, characterized in that, Includes a computer program or instructions that, when executed by a processor, implement the model training method of any one of claims 1 to 10, or the data processing method of claim 11.

Images