Operation response method and device, electronic equipment and storage medium
Through the goal decision model trained by deep reinforcement learning, combining the state characteristics of the target object and real-time business data, the response strategy is generated and adjusted, which solves the problem of insufficient flexibility of response strategy in the existing technology, and improves the execution effect and adaptation of response strategy Compatibility.
Patent Information
- Application Number
- CN202310595892.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art relies on the matching method of the rule engine when responding to different operations of related objects, resulting in insufficient flexibility and scalability of the response strategy, it is difficult to assist related objects in completing business processing, and insufficient decision adaptability.
A target decision model trained by deep reinforcement learning is adopted, combining the state characteristics of the target object and real-time business data, and a response strategy is generated and adjusted to improve the execution effect of the response strategy.
Through the deep reinforcement learning model, we learn the decision preferences of the target object and generate high adaptability response strategies, which improves the execution effectiveness and accuracy of the response strategy and enhances the scalability and flexibility of the strategy.
Smart Images

Figure CN119025616A_ABST
Abstract
Description
Background Art
[0002] In the related art, when business interactions are performed with related objects, adaptive responses are usually made according to different operations of the related objects to assist the related objects in completing business processing.
[0003] At present, when adaptively responding to different operations on related objects, a rule engine is usually used to preset corresponding response strategies for various possible operations on various types of objects; then, in response to the operation instructions of the related objects, after determining the matching target response strategy based on the operation information of the related objects, the corresponding target response strategy is executed for the related objects to achieve the operation response to the related objects.
[0004] However, in the existing operation response methods, the target response strategy to be executed can only be rigidly determined based on the matching of the operation information and the response strategy. Therefore, it is very easy to execute a target response strategy that is not very helpful to the relevant objects, and it is difficult to assist in completing the business processing of the relevant objects. Summary of the invention
[0005] The embodiments of the present application provide an operation response method, device, electronic device and storage medium for determining a response strategy that can obtain positive feedback from relevant objects and improve the execution effect of the response strategy.
[0006] In a first aspect, an operation response method is proposed, comprising:
[0007] In response to an operation indication triggered by a target object, a target state characteristic associated with the target object is obtained, wherein the target state characteristic includes: an operation characteristic used to describe the operation status of the target object in a specified business scenario, and an attribute characteristic publicly displayed by the target object;
[0008] The trained target decision model is used to obtain the corresponding target decision result based on the target state characteristics, wherein the target decision model is trained by using a deep reinforcement learning training method, after presetting each state characteristic and each decision result for the specified business scenario, based on the sample objects corresponding to each state characteristic, and based on the historical feedback of each decision result;
[0009] Generate a corresponding target response strategy based on the target decision result, the decision prompt information stored for the object group to which the target object belongs, and the real-time updated business data;
[0010] The target response strategy is executed for the target object.
[0011] In a second aspect, an operation response device is provided, comprising:
[0012] A response unit, configured to obtain target state characteristics associated with the target object in response to an operation indication triggered by the target object, wherein the target state characteristics include: operation characteristics used to describe the operation status of the target object in a specified business scenario, and attribute characteristics publicly displayed by the target object;
[0013] An obtaining unit is used to adopt a trained target decision model to obtain a corresponding target decision result based on the target state feature, wherein the target decision model is trained by adopting a deep reinforcement learning training method, after presetting each state feature and each decision result for the specified business scenario, based on the sample objects corresponding to each state feature, and trained for the historical feedback of each decision result;
[0014] A generating unit, configured to generate a corresponding target response strategy based on the target decision result, the decision prompt information stored for the object group to which the target object belongs, and the business data updated in real time;
[0015] An execution unit is used to execute the target response strategy for the target object.
[0016] Optionally, when acquiring the target state feature associated with the target object, the response unit is used to:
[0017] Acquire business data of the target object in a specified business scenario, wherein the business data includes historical operation data, current operation data, and currently publicly displayed attribute information;
[0018] The business data is normalized, and data content irrelevant to the operation of the target object is deleted from the business data. Feature extraction is performed based on the processed business data to obtain target state features associated with the target object.
[0019] Optionally, when generating a corresponding target response strategy based on the target decision result, the decision prompt information stored for the object group to which the target object belongs, and the real-time updated business data, the generating unit is used to:
[0020] Based on the attribute characteristics, determining the object group to which the target object belongs, and acquiring decision prompt information stored for the object group;
[0021] Based on the decision prompt information, the target decision result is adjusted to obtain an adjusted target decision result, and based on the real-time updated business data and the adjusted target decision result, a target response strategy applicable to the target object in the specified business scenario is generated.
[0022] Optionally, the decision prompt information records: each negative sub-decision result with negative feedback, each positive sub-decision result with positive feedback and the corresponding weight, and the mutually exclusive relationship between the sub-decision results; when adjusting the target decision result based on the decision prompt information, the generation unit is used to perform any one or combination of the following:
[0023] Deleting the negative decision contents in the target decision result that match the negative sub-decision results from the target decision result;
[0024] Determine the positive decision contents in the target decision result that match the positive sub-decision results, and delete the positive sub-decision result with a lower weight in every two positive sub-decision results that have a mutually exclusive relationship from the target decision result.
[0025] Optionally, in the process of training to obtain the target decision model, the device further includes a training unit, and the training unit is used to perform the following operations:
[0026] Generate training samples based on the historical business data of each sample object in the specified business scenario; a training sample includes: information groups associated with a sample object in a time-series arrangement, each information group is used to describe a process in which a sample object is guided to process a target business, and an information group includes a sample state feature, a sample decision result, and a sample feedback result;
[0027] The constructed initial decision model is trained multiple times using the training samples until a preset convergence condition is met, thereby obtaining a trained target decision model.
[0028] Optionally, during a round of iterative training, the training unit is used to perform the following operations:
[0029] Reading a training sample to obtain information groups arranged in time sequence in the training sample;
[0030] Obtain each information group in turn in the described chronological order and perform the following operations:
[0031] The sample state features in the information group are input into the initial decision model to obtain the predicted feedback results of the initial decision model on the corresponding sample decision results, and the model parameters of the initial decision model are adjusted according to the result difference between the predicted feedback results and the corresponding sample feedback results.
[0032] Optionally, when generating each training sample based on the historical business data of each sample object in the specified business scenario, the training unit is used to:
[0033] Obtain the historical business data of each sample object in a specified business scenario within a preset time period;
[0034] Each historical business data is normalized to obtain each process data, wherein one process data includes: data describing the state time series change process of a sample object in the process of being guided to process a target business;
[0035] For each process data, each information group involved in the temporal change process of the sample state feature of the corresponding sample object is determined, and a training sample is generated based on the information groups.
[0036] Optionally, when normalizing each historical business data to obtain each process data, the training unit is used to:
[0037] In each historical business data, delete the historical business data with abnormal values of the specified content, and for the remaining historical business data, correct the values of other contents and adjust the value range of each numerical content;
[0038] For each processed historical business data, perform the following operations respectively:
[0039] Natural language processing technology is used to perform semantic identification processing on each text content in the processed historical business data to obtain corresponding process data.
[0040] Optionally, when executing the target response strategy for the target object, the execution unit is used to:
[0041] Storing feedback results of the target object under the target state characteristics for the target response strategy, so that when it is determined that the update cycle of the target decision model has been reached, training samples are regenerated based on the stored correspondence between the target state characteristics, the target response strategy, and the feedback results;
[0042] Based on the feedback result, the decision prompt information is updated.
[0043] In a third aspect, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the program.
[0044] In a fourth aspect, a computer-readable storage medium is proposed, on which a computer program is stored, and the computer program implements the above method when executed by a processor.
[0045] In a fifth aspect, a computer program product is proposed, comprising a computer program, wherein the computer program implements the above method when executed by a processor.
[0046] The beneficial effects of this application are as follows:
[0047] In an embodiment of the present application, an operation response method, device, electronic device and storage medium are proposed. In response to an operation instruction triggered by a target object, a target state feature representing the current operation status of the target object in a specified business scenario and the attribute information of the target object are obtained, and then a trained target decision model is used to determine a target decision result applicable to the target object for the state of the target object represented by the target state feature. The target decision model is trained in a deep reinforcement learning manner. After presetting each state feature and each decision result for a specified business scenario, the model is trained based on the sample objects corresponding to each state feature and the historical feedback of each decision result. Thereafter, based on the target decision result, the decision prompt information stored for the object group to which the target object belongs, and the business data updated in real time, a target response strategy to be executed on the target object is determined, and then the target response strategy is executed on the target object.
[0048] In this way, since the target decision model used to determine the target decision result is trained based on the feedback of sample objects on the decision results in different states by using deep reinforcement learning, the trained target decision model learns the preferences of different types of objects for the decision results; furthermore, after executing the target response strategy generated based on the target decision result on the target object, it is possible to obtain positive feedback from the relevant objects with a high probability, thereby improving the adaptability between the target decision result and the target object; moreover, the obtained target response strategy can play a positive role in promoting the business processing of the target object, thereby improving the effectiveness of the execution of the target response strategy;
[0049] In addition, when generating a target response strategy based on the target decision result, since it combines the decision prompt information stored for the object group to which the target object belongs and the real-time updated business data, when generating the target response strategy at different times, on the one hand, it can incorporate the real-time updated business data to improve the scalability and flexibility of the target response strategy; on the other hand, considering the impact of the decision prompt information, it can adjust the target response strategy according to the existing decision feedback, thereby improving the accuracy and feasibility of the target response strategy. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 A schematic diagram of possible application scenarios in the embodiments of the present application;
[0051] Figure 2 A schematic diagram of the operation response process in an embodiment of the present application;
[0052] Figure 3A This is a schematic diagram of the process of training and obtaining a target decision model in an embodiment of the present application;
[0053] Figure 3B A schematic diagram of the process of generating various training samples in an embodiment of the present application;
[0054] Figure 3C This is a schematic diagram of historical business data obtained during training in an embodiment of the present application;
[0055] Figure 4A A schematic diagram of the process of constructing a negative sub-decision in an embodiment of the present application;
[0056] Figure 4B This is a schematic diagram of the process of continuing to add active sub-decisions in the embodiment of the present application;
[0057] Figure 5A A schematic diagram showing the content of the response to the operation executed in the implementation of this application;
[0058] Figure 5B A schematic diagram of the process of obtaining feedback results from the processing device in the implementation of this application;
[0059] Figure 6 This is a schematic diagram of the logical structure of the operation response device in the embodiment of the present application;
[0060] Figure 7 A schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application;
[0061] Figure 8 Schematic diagram of the structure of a computing device in an embodiment of the present application. DETAILED DESCRIPTION
[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.
[0063] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.
[0064] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.
[0065] Target state characteristics: refers to the characteristics of the target object that needs to respond to the operation during the operation response process in actual application. It is used to describe the current operation status of the target object in the specified business scenario and the attribute information exposed by the target object.
[0066] Training samples: In an embodiment of the present application, the constructed training samples are used to train a target decision model. The data in a training sample describes the time series change process of each sample state feature in the process of a sample object being guided to process a target business, wherein a sample state feature is associated with a sample decision result and a sample feedback result, and the historical business data is characterized by the following records: when the sample object is in a state described by a sample state feature and an operation response corresponding to a sample decision result is executed, a historical feedback operation described by the sample feedback result is performed.
[0067] Deep Reinforcement Learning (DRL): is a method that combines deep learning and reinforcement learning. It learns the best strategy by training a neural network model to maximize the cumulative reward in a certain environment.
[0068] Adaptive Rule Engine (ARE): It can be understood as a system that can automatically adjust and optimize rules based on input data. It uses machine learning technology to improve prediction accuracy and solve complex problems.
[0069] Artificial Intelligence (AI): It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.
[0070] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0071] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology generally includes text processing, semantic understanding, machine translation, robot question and answer, knowledge graph and other technologies; in the embodiment of the present application, the text content is semantically identified with the help of natural language processing, and the processing involved in the process of semantic identification includes word segmentation, part-of-speech tagging, and word sense disambiguation.
[0072] Machine Learning (ML): It is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.
[0073] The following is a brief introduction to the design concept of the embodiment of the present application:
[0074] In the related art, when business interactions are performed with related objects, it is usually necessary to make adaptive responses according to different operations of the related objects to assist the related objects in completing business processing.
[0075] Currently, when making adaptive responses to different operations of related objects, a rule engine is usually used to first determine a matching target response strategy based on the current operation information of the related objects, and then execute the target response strategy for the related objects.
[0076] However, the existing response methods lack flexibility and scalability when selecting target response strategies, and can only determine the target response strategy to be executed based on the matching between operation information and response strategies. Moreover, the decision-making adaptability of the target response strategy is insufficient, and it is very easy to execute a target response strategy that is not very helpful to the relevant objects, thereby reducing the execution effect of the response strategy and failing to meet the business needs of the relevant objects.
[0077] In view of this, in an embodiment of the present application, an operation response method, device, electronic device and storage medium are proposed, which respond to the operation indication triggered by the target object, obtain the target state characteristics that characterize the current operation status of the target object in the specified business scenario, and the attribute information of the target object, and then use the trained target decision model to determine the target decision result applicable to the target object for the state of the target object represented by the target state characteristics, wherein the target decision model is trained in a deep reinforcement learning manner, and after presetting each state characteristic and each decision result for the specified business scenario, it is trained based on the sample objects corresponding to each state characteristic and the historical feedback of each decision result; thereafter, based on the target decision result, the decision prompt information stored for the object group to which the target object belongs, and the business data updated in real time, the target response strategy to be executed on the target object is determined, and then the target response strategy is executed on the target object.
[0078] In this way, since the target decision model used to determine the target decision result is trained based on the feedback of sample objects on the decision results in different states by using deep reinforcement learning, the trained target decision model learns the preferences of different types of objects for the decision results; furthermore, after executing the target response strategy generated based on the target decision result on the target object, it is possible to obtain positive feedback from the relevant objects with a high probability, thereby improving the adaptability between the target decision result and the target object; moreover, the obtained target response strategy can play a positive role in promoting the business processing of the target object, thereby improving the effectiveness of the execution of the target response strategy;
[0079] In addition, when generating a target response strategy based on the target decision result, since it combines the decision prompt information stored for the object group to which the target object belongs and the real-time updated business data, when generating the target response strategy at different times, on the one hand, it can incorporate the real-time updated business data to improve the scalability and flexibility of the target response strategy; on the other hand, considering the impact of the decision prompt information, it can adjust the target response strategy according to the existing decision feedback, thereby improving the accuracy and feasibility of the target response strategy.
[0080] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.
[0081] See also Figure 1 , which is a schematic diagram of a possible application scenario in an embodiment of the present application, includes a processing device 110 , a client device 120 , and an optional auxiliary device 130 .
[0082] In some possible embodiments, the processing device 110 may obtain a trained target decision model from the auxiliary device 130 . In other words, in a possible embodiment, the training of the target decision model is performed by the auxiliary device 130 .
[0083] In some other possible embodiments, the processing device 110 may train itself to obtain a target decision model; wherein the target decision model is used to output a decision result based on the state characteristics.
[0084] In the embodiments of the present application, the processing device 110 and the auxiliary device 130, and the processing device 110 and the client device 120 can communicate through a wired network or a wireless network; in the process of performing an operation response, the processing device 110 can respond to the operation of the target object on the operable page to implement an operation response, wherein the operable page can belong to an application, a web page, or a mini-program, and the present application does not impose any specific restrictions on this.
[0085] The processing device 110 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms; or, it can be a terminal device with certain processing capabilities.
[0086] The client device 120 may be a computer device such as a personal computer, a mobile phone, a tablet computer, a notebook, an e-book reader, a smart home, and a vehicle-mounted terminal.
[0087] The auxiliary device 130 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms; or, it can be a terminal device with certain processing capabilities.
[0088] The technical solution proposed in this application can realize operation response in various business scenarios. The following describes the relevant operation response process in combination with several possible business scenarios:
[0089] Business scenario 1: Shopping item recommendations.
[0090] In the shopping item recommendation scenario proposed in business scenario 1, items that meet the purchase needs can be recommended.
[0091] Specifically, the processing device can obtain the state characteristics of the target object in response to the operation instructions triggered by the target object in the shopping application, or the shopping webpage, or the shopping mini-program; then, with the help of the trained target decision model, based on the state characteristics of the target object, output the corresponding target decision result; thereafter, based on the target decision result, the decision prompt information stored for the object group to which the target object belongs, and the real-time updated purchasable items, generate a response strategy corresponding to the target object; then, based on the determined response strategy, determine the content to be presented to the target object.
[0092] Afterwards, with the help of step-by-step operational responses and strategy execution, the target object is assisted in completing the purchase of the item.
[0093] Business scenario 2: Multimedia data recommendation.
[0094] In the multimedia data recommendation scenario proposed in business scenario 2, multimedia data that meets viewing needs can be recommended, wherein the multimedia data can specifically be obtained from at least one or a combination of audio data, video data, and text data.
[0095] Specifically, the processing device can obtain the state characteristics of the target object in response to the operation instructions triggered by the target object on the multimedia data playback application, or in response to the operation instructions triggered by the target object on the multimedia data playback web page; then, with the help of the trained target decision model, based on the state characteristics of the target object, output the corresponding target decision result; then, based on the target decision result, the decision prompt information stored for the object group to which the target object belongs, and the real-time updated multimedia data, generate a response strategy corresponding to the target object; then, based on the determined response strategy, determine the content to be presented to the target object.
[0096] Afterwards, with the help of step-by-step operation responses and policy execution, the target object is assisted in finding the multimedia data of interest.
[0097] Business scenario three: abnormal behavior analysis.
[0098] In the abnormal behavior analysis scenario proposed in business scenario three, it is possible to determine whether related objects have abnormalities and perform adaptive processing.
[0099] Specifically, the processing device can obtain the state characteristics of the target object in response to the operation instructions triggered by the target object in an application such as implementing financial business; then, with the help of a trained target decision model, based on the state characteristics of the target object, output the corresponding target decision result; thereafter, based on the target decision result, the decision prompt information stored for the object group to which the target object belongs, and the real-time updated financial operation exception handling policy, generate a response strategy corresponding to the target object; then, based on the determined response strategy, determine the content to be presented to the target object.
[0100] Afterwards, with the help of step-by-step operational response and policy execution, the target object is assisted to conduct financial business processing safely and legally.
[0101] Business scenario four: autonomous driving planning.
[0102] In the autonomous driving planning scenario proposed in business scenario four, route planning can be adaptively performed based on the collected road conditions.
[0103] Specifically, the processing device can obtain the state characteristics of the target object in response to the driving instructions triggered by the target object on the navigation application; then, with the help of the trained target decision model, based on the state characteristics of the target object, output the corresponding target decision result; thereafter, based on the target decision result, the decision prompt information stored for the object group to which the target object belongs, and the road condition information updated in real time, generate the route planning result corresponding to the target object; then, guide the target object to travel according to the route planning result.
[0104] Afterwards, with the help of step-by-step operational responses and strategy execution, the target object is assisted to reach the destination safely and quickly. The operational response in this scenario may be determined by detecting the vehicle's driving conditions, or it may be manually operated by the target object, which may be driving according to the plan, changing lanes, manually triggering re-routing, etc.
[0105] In conjunction with the accompanying drawings, from the perspective of a processing device, the processing process involved in implementing an operation response is described by taking the processing device executing the training of a target decision model and performing an operation response as an example:
[0106] See also Figure 2 As shown, it is a schematic diagram of the process of operating response in the embodiment of the present application. Figure 2 , explain the relevant operation response process:
[0107] Step 201: The processing device obtains a target state feature associated with the target object in response to an operation instruction triggered by the target object.
[0108] In an embodiment of the present application, the processing device responds to the operation indication triggered by the target object in the operable page, wherein the method of triggering the operation indication may be: manually clicking the page component, clicking the page component with the mouse, automatically triggering when entering the specified page, and automatically triggering when browsing to the specified content location in the specified page, etc.; the operable page may be a page in the client application, or it may be a page in the web page, or it may be a page in the mini program, and the present application does not impose any specific restrictions on this.
[0109] Furthermore, in order to achieve the response, the processing device obtains the target state characteristics associated with the target object, wherein the target state characteristics include: operation characteristics used to describe the operation status of the target object in a specified business scenario, and attribute characteristics publicly displayed by the target object.
[0110] In the process of obtaining the target state characteristics, the processing device obtains the business data of the target object in the specified business scenario, wherein the business data includes historical operation data, current operation data, and currently publicly displayed attribute information; normalizes the business data, deletes data content irrelevant to the operation of the target object from the business data, and extracts features based on the processed business data to obtain the target state characteristics associated with the target object.
[0111] Specifically, the processing device can provide operation responses for different businesses, or the processing device can provide operation responses for a single business; based on this, when the processing device responds to the operation instruction of the target object, it is necessary to determine a specified business scenario in which the target object triggers the operation instruction; then, in order to restore the operation status of the target object in the specified business scenario, the processing device obtains the business data of the target object in the specified business scenario.
[0112] It should be noted that, in the embodiment of the present application, the business data includes: during the current business processing of the target object, all operation information from the start of the business processing to the target object triggering the operation indication, the historical operation data before the current business processing of the target object, and the attribute information of the target object, wherein the start of the business processing may specifically refer to: the target object triggering access to any one of the applications, web pages, and mini-programs developed for the specified business scenario.
[0113] In addition, the target object's publicly displayed attribute information may be any of the following: the target object sets publicly displayed attribute information in the account registered in the specified business scenario; the target object sets publicly displayed attribute information in other applications when the target object performs business processing in the specified business scenario based on the account in other applications; the target object sets and displays attribute information in other applications when the target object authorizes the collection. The publicly displayed attribute information may specifically be the target object's publicly displayed gender information, age information, and region information.
[0114] In the embodiment of the present application, the acquisition of data related to the object is carried out with the authorization of the relevant object, and the acquisition and use of the data complies with the provisions of relevant laws and regulations.
[0115] Furthermore, after acquiring the business data associated with the target object, in the process of normalizing the business data, the processing device pre-processes the business data to achieve data cleaning and data conversion of the business data. In the data cleaning stage, the missing values in the business data are processed, and the processing method can be to fill the missing content with set fields; in the data conversion stage, the numerical content is subjected to value constraint processing, and the processing method can be to perform standardization or normalization operations so that the value of the numerical content is within the specified range.
[0116] After completing the preprocessing, the processing device uses natural language processing technology to perform semantic identification processing on the text content in the business data and complete the normalization processing. The processing involved in semantic identification includes at least one or a combination of word segmentation, part-of-speech tagging, and semantic disambiguation; word segmentation is to decompose the text into independent words or tags; part-of-speech tagging is to identify the language role of each word, that is, the part-of-speech information; word sense disambiguation is to determine the specific meaning of a word in a specific context and perform semantic interpretation on the word.
[0117] Furthermore, the processing device deletes data content irrelevant to the operation of the target object from the normalized business data according to actual processing needs.
[0118] It should be noted that the data content related to the operation of the target object retained in this application specifically refers to the content that can have an important impact on the model decision. The relevant content can be obtained based on actual processing experience, or all business data can be directly used as data content related to the operation of the target object.
[0119] For example, in the model training stage before the operation response, the processing device can try to use different operation information to participate in the generation of training samples, and then determine the operation information that is most effective for decision analysis based on the processing effect of the model; thereafter, when performing an operation response based on the target decision model, the operation information that can effectively indicate the operation situation is determined based on the training experience and trial results obtained during the training process of the target decision model.
[0120] For another example, assuming that in a shopping recommendation scenario, the business data acquired by the processing device may include data content that is presented indiscriminately for different objects, as well as data generated by direct operations on the target object. In this case, it can be clearly determined that content that is presented indiscriminately and for which the target object has no operations is not helpful for model decision-making, so the data content that is presented indiscriminately can be filtered out.
[0121] Afterwards, the processing device extracts features based on the processed business data to obtain target state features associated with the target object. The feature extraction technology is used to extract text features for the text content in the processed business data, wherein the extracted features may be term frequency-inverse document frequency (TF-IDF) features, word embedding features, etc. At the same time, the processing device extracts corresponding statistical features for the statistically numerical content in the processed business data, wherein the statistical features include at least one or a combination of average value, maximum value, minimum value, and standard deviation.
[0122] It should be noted that since the technology for feature extraction for text is a conventional technology in this field, in possible implementation methods, the processing device can adopt models such as bag-of-words model, N-gram model (N-gram), text to vector model (wordto vector, word2vec), etc. to realize text feature extraction. This application will not specifically limit the extraction method used to extract text features.
[0123] Furthermore, the obtained text features and statistical features are fused to obtain target state features associated with the target object, wherein for different target objects, the constructed target state features have the same dimension.
[0124] In this way, the target state characteristics obtained for the target object can characterize the corresponding essential attributes of the target object and the operation status of the target object, so that the state of the target object in the specified business scenario can be fully described; that is, the processing device can respond to the operation instructions triggered by the target object in the specified business scenario, and obtain the target state characteristics that characterize the current state of the target object in the specified business scenario, so that with the help of the target state characteristics, the operation status of the target object with corresponding attribute characteristics in the specified business scenario can be characterized, which provides a processing basis for subsequent response decisions based on the state of the target object; moreover, since data normalization and data screening operations are performed in the process of generating the target state characteristics, it can facilitate the feature extraction process and improve the feature extraction efficiency and the target state feature acquisition efficiency.
[0125] Step 202: The processing device uses the trained target decision model to obtain a corresponding target decision result based on the target state characteristics.
[0126] After the processing device obtains the target state characteristics associated with the target object, it uses the trained target decision model to obtain the corresponding target decision results based on the target state characteristics. The target decision model is trained using a deep reinforcement learning method. After presetting each state characteristic and each decision result for a specified business scenario, it is trained based on the sample objects corresponding to each state characteristic and the historical feedback of each decision result.
[0127] The following is a description of the process of training and obtaining the target decision model with reference to the accompanying drawings:
[0128] See also Figure 3A As shown, it is a schematic diagram of the process of training to obtain the target decision model in the embodiment of the present application. Figure 3A , the relevant training process is explained:
[0129] Step 301: The processing device generates each training sample based on the historical business data of each sample object in a specified business scenario.
[0130] It should be noted that, according to actual processing needs, the processing device can be configured to periodically retrain the target decision model, or can be configured to make real-time adjustments to the target decision model after the target decision model is trained in the initial stage of the model being put into use; or, can be configured to make real-time adjustments to the target decision model after the target decision model is trained in the case of insufficient training samples, and the processing device can provide operational responses for different business scenarios; based on this, the processing device can train a corresponding target decision model for different business scenarios. In the following description, only the example of training the target decision model for a specified business scenario is used for schematic explanation.
[0131] See also Figure 3B As shown, it is a schematic diagram of the process of generating each training sample in the embodiment of the present application. Figure 3B , the process of generating each training sample is explained:
[0132] Step 3011: The processing device obtains the historical business data of each sample object in a specified business scenario within a preset time period.
[0133] Specifically, since the application adopts the deep reinforcement learning training method in the process of training the target decision model, it is necessary to implement the model training adjustment based on the feedback results of the decision results based on sample objects in different states; based on this, in order to meet the training needs, the processing device obtains the historical business data of each sample object in the specified business scenario within a preset time period.
[0134] It should be noted that the preset time period can be selected according to actual processing needs. For each historical business data within the preset time period, it can be the historical business data of each sample object up to the training time point; or, the preset time period can be a historical time period selected based on training needs, and this application does not impose specific restrictions on this. In addition, the historical business data includes: the historical operation data generated by the sample object's most recent processing of the target business up to the training time point, other historical operation data, and attribute information of the sample object.
[0135] For example, see Figure 3C As shown, it is a schematic diagram of historical business data obtained during training in the embodiment of the present application. Figure 3C As can be seen from the schematic content, model training is performed at time point 1 and time point 2, respectively. During the model training process at time point 1, various historical business data for generating training samples are obtained, wherein one historical business data obtained includes: historical operation data when the sample object last processed the target business as of time point 1, other historical operation data, and attribute information of sample object 1; similarly, during the model training process at time point 2, when various historical business data for generating training samples are obtained, one historical business data obtained includes: historical operation data when the sample object last processed the target business as of time point 2, other historical operation data, and attribute information of the sample object.
[0136] For another example, considering that historical business data that is very far away from the present is not very helpful for data analysis, the preset time period can be set to the most recent year based on actual processing experience; or, if you want to analyze all historical business data, you can set the preset time period to a larger value so that all historical business data of each sample object can be covered.
[0137] In addition, in an embodiment of the present application, each sample object refers to an object that has performed business processing in a specified business scenario. For the sample object to process the target business, the processing operations of the sample object on the business can be predefined. Then, when it is determined that the operations performed under the specified business meet the predefined requirements, it can be determined that the target business has been processed.
[0138] For example, assuming that the specified business scenario is online shopping, then each sample object may refer to a user who is registered on the shopping application and has historical operations, where the operation methods corresponding to the historical operations may be: searching, browsing, purchasing, participating in discussions, evaluating, and other operations on items; the processing corresponding to the predefined target business may be: purchasing specific items, viewing the details of specific items, and so on.
[0139] Step 3012: The processing device performs normalization processing on each historical business data to obtain each process data.
[0140] After the processing device obtains the historical business data corresponding to each sample object, it normalizes each historical business data to obtain each process data, where one historical business data includes: historical operation data representing the most recent target business processing process, other historical operation data, and attribute information of the sample object.
[0141] In an embodiment of the present application, the processing device deletes historical business data with abnormal values of specified content in each historical business data, and corrects the values of other contents and adjusts the value range of each numerical content for the remaining historical business data; then, for each processed historical business data, the following operations are performed respectively: natural language processing technology is used to perform semantic identification processing on each text content in the processed historical business data to obtain a corresponding process data.
[0142] Specifically, the processing device can adopt the same processing method as the normalization processing recorded in step 201 to achieve normalization processing of each historical business data. The processing process involved includes: data cleaning of each historical business data, removing historical business data including abnormal content, wherein the historical business data including abnormal content may be: repeated historical business data with the same content as other historical business data; historical business data whose corresponding value result exceeds a preset normal threshold after directly determining or statistically determining the value result for the specified field name.
[0143] For example, when determining historical business data with abnormal values of specified content, if it is determined that the frequency of access operations of the target object in the specified business scenario reaches a preset normal threshold based on the number of accesses to the target object recorded in the historical business data, then it can be considered that the historical business data includes abnormal content, and therefore it is necessary to delete this historical business data from each historical business data.
[0144] It should be noted that in the embodiment of the present application, according to actual processing needs, when deleting various historical business data, a set of field names that need to be focused on can be pre-set, so that when there are abnormal values of field names included in the field name set, the corresponding historical business data will be directly deleted; in particular, when there are field names with abnormal values that are not included in the field name set, the processing device can correct the value corresponding to the field name to eliminate the impact of the abnormal value.
[0145] That is to say, for the value content corresponding to the field name that is not the focus of attention, the processing device can perform value correction processing on the content with incorrect or empty values, thereby filling in missing values and adjusting abnormal values.
[0146] Afterwards, the processing device adjusts the value range of the numerical content for the remaining historical business data. Specifically, it may perform operations such as standardization or normalization so that the value of the numerical content is within a specified range.
[0147] Based on this, it is equivalent to completing the preprocessing for each historical business data. Then, the processing device uses natural language processing technology to perform semantic identification processing on the text content in the processed historical business data, and finally obtains a process data corresponding to the target business processing process under consideration, wherein the processing involved in semantic identification includes at least one or a combination of word segmentation, part-of-speech tagging, and semantic disambiguation; thereafter, for each processed historical business data, a corresponding process data is obtained, wherein a process data includes: data describing the state time series change process of a sample object being guided to process a target business.
[0148] In this way, by preprocessing and semantic identification of the historical business data of the sample object, the adverse effects brought by the historical business data with abnormal specified content values can be eliminated, thereby ensuring the validity of the historical business data; moreover, based on the obtained process data, it is possible to characterize not only the target business processing process closest to the training time point, but also the processing situations of other target businesses that occurred before the target business processing process, so that the status of the target object can be more comprehensively characterized based on the historical data accumulated by the sample object in the future, thereby helping to improve the accuracy of analytical decisions.
[0149] Moreover, the processing equipment can better understand and parse the text content by integrating natural language processing technology, thereby assisting the model in making effective decisions later.
[0150] Step 3013: The processing device determines, for each process data, each information group involved in the temporal change of the sample state characteristics of the corresponding sample object, and generates a training sample based on each information group.
[0151] Specifically, after the processing equipment processes each piece of historical business data to obtain the corresponding process data, it constructs a corresponding training sample for each process data, wherein the content described in a training sample is: the time series change of the state of the sample object in the most recent target business processing process up to the training time point, as well as the decision results corresponding to each state of the sample object and the feedback results of the sample object for the decision results.
[0152] Based on this, the processing device describes the time-series changes of the state in the form of information groups arranged in time sequence, and characterizes the state of the sample object by constructing a sample state feature; therefore, a constructed information group includes: a sample state feature, a sample decision result, and a sample feedback result; the content in an information group represents the feedback result that can be obtained when the corresponding decision result is executed for a sample object in a state.
[0153] It should be noted that when generating the sample state features of the sample object, feature extraction can be performed on the process data corresponding to the sample object. Specifically, text feature extraction can be performed on the text content in the process data, and statistical features can be extracted from the numerical content in the process data, such as the average value, maximum value, minimum value, standard deviation, etc.; the obtained text features and statistical features are then fused to obtain the sample state features associated with the sample object. When performing statistical feature extraction and text feature extraction, the same processing method as in step 201 can be adopted, which is not described in detail in this application.
[0154] Furthermore, the processing device constructs each information group with a time sequence for each state of the time sequence change described in each process data, and generates a corresponding training sample based on each constructed information group. Similarly, each training sample can be obtained corresponding to each process data.
[0155] In this way, by obtaining the historical business data of each sample object and screening and processing it in a targeted manner, it is ultimately possible to extract the state change process of the most recent target business processing from each historical business data, and to extract features that reflect the essential attributes and relationships of the data, which helps to improve the learning effect of subsequent models; moreover, since the sample state features that describe the state include statistical features for various types of numerical content, as well as text features extracted from text content, it is possible to use the accumulated historical data to describe the state of the sample object from a continuous perspective, thereby being able to better construct training samples.
[0156] Step 302: The processing device uses each training sample to perform multiple rounds of iterative training on the constructed initial decision model until a preset convergence condition is met, thereby obtaining a trained target decision model.
[0157] In an embodiment of the present application, after the processing device constructs each training sample, it performs multiple rounds of iterative training on the constructed initial decision model based on each training sample until a preset convergence condition is met, and finally obtains a trained target decision model, wherein the preset convergence condition may be: the number of training rounds reaches a set value, or the number of times the loss value is continuously lower than the first set value reaches a second set value, etc., wherein the values of the first set value and the second set value are set according to actual processing needs.
[0158] Specifically, when constructing the initial decision model, the processing device can construct the initial decision model as a classification model or a regression model according to the actual processing goal; if the processing goal is to predict specific feedback results (for example, in a content recommendation scenario, determine the probability that the user will click on each preset decision result for feedback), then the initial decision model can be constructed as a regression model; and, if the processing goal is to predict the category of feedback (for example, in a content recommendation scenario, determine whether the user will click on each preset decision result for feedback), then the initial decision model can be constructed as a classification model, wherein, when constructing the initial decision model, a general classification model structure or regression model structure can be used to implement model construction.
[0159] When the processing device uses the deep reinforcement learning method to train the initial decision model, it first defines a state space and an action space, where the state space includes preset state features and the action space includes preset decision results; then, it uses the deep Q network (Deep QNetwork, DQN) or other deep reinforcement learning algorithms for training.
[0160] In the possible training process, for DQN, a Q function is defined, which represents the expected reward for taking action a in state s. The goal is to find a strategy that maximizes the cumulative reward for taking actions according to this strategy starting from any state. Use the following Bellman equation for update:
[0161]
[0162] Among them, s t and a t Respectively represent the state at time t and the decision result of execution, r t is the immediate reward, α is the learning rate, γ is the discount factor, It is an expectation of future rewards. The relevant reward calculation process is a mature technology in this field and will not be described in detail here.
[0163] The following takes the training operation performed during a round of iterative training as an example to illustrate the training process performed during the process of training to obtain the target decision model:
[0164] Read a training sample to obtain each information group arranged in time sequence in the training sample; then obtain each information group in turn according to the time sequence order, and perform the following operations: input the sample state characteristics in the information group into the initial decision model to obtain the predicted feedback result of the initial decision model on the corresponding sample decision result, and adjust the model parameters of the initial decision model according to the difference between the predicted feedback result and the corresponding sample feedback result.
[0165] Specifically, in the initial round of iterative training, the processing device trains the initial decision model through interactive learning. In the specific training process, interactive learning is performed according to the order of each information group with a time sequence. In a round of training, according to the corresponding time sequence in the first information group, the initial decision model is used to predict the feedback probability of each preset decision result based on the sample state characteristics in the corresponding information group, and then the predicted decision result is determined according to the feedback probability, and then the model parameters are adjusted according to the difference between the decision result and the sample decision result in the corresponding information group. Similarly, for other subsequent information groups in a training sample, the above operations are performed respectively to achieve model parameter adjustment.
[0166] It should be noted that in the embodiments of the present application, in deep reinforcement learning, if the initial decision model constructed is a regression model, the loss function used is the mean squared error loss function (MSE); if the initial decision model constructed is a classification model, the loss function used is the cross entropy loss function (Cross Entropy Loss).
[0167] In addition, it should be clear that since the technical solution adopted in this application is an improvement based on the original decision matching method, when initially constructing training samples for the model training process, the sample construction can be achieved based on the historical business data generated by the decision-making method indicated by the background technology; and then through interactive learning, the actual needs of users can be mined, so that a reasonable decision-making method can be determined for users.
[0168] In this way, by adopting the deep reinforcement learning method to train the model, the model learning can be guided according to the actual feedback results of the sample objects. When the training model faces different states with time-series relationships, it gradually predicts the decision results that can obtain positive feedback, so that the decision model can conduct targeted learning in combination with the actual feedback results of the decision results, thus providing a guarantee for the training effect of the model; moreover, with the help of deep reinforcement learning, the target decision model can learn and maximize the cumulative reward, thus ensuring the decision effect.
[0169] Then, the processing device executes rounds of iterative training processes similar to the above process, and finally outputs the trained target decision model.
[0170] In this way, the trained target decision model has the ability to predict decision results with positive feedback results based on state characteristics. The trained target decision model learns the preferences of different types of objects for decision results, which helps to improve the effectiveness and adaptability of response decisions.
[0171] During the processing of step 202, the processing device uses the trained target decision model to obtain a corresponding target decision result based on the target state characteristics associated with the target object.
[0172] It should be noted that when determining the target decision result, the prediction result of the target decision model is converted into a specific decision rule. Specifically, if the target decision model is essentially a classification model, the decision result corresponding to the output result of the specified category can be directly determined as the target decision result based on the output category of the target decision model; if the target decision model is essentially a regression model, a threshold can be set, the output value of the model can be compared with the threshold, and the decision result with the corresponding output value higher than the threshold can be determined as the target decision result.
[0173] In this way, compared with the function of the rule engine under the prior art, the present application uses the prediction of the target decision model to provide a processing basis for the decision of the rule engine, which is equivalent to proposing an adaptive rule engine in the solution proposed in the present application, which can use the target decision model to assist in decision processing.
[0174] Step 203: The processing device generates a corresponding target response strategy based on the target decision result, the decision prompt information stored for the object group to which the target object belongs, and the real-time updated business data.
[0175] When executing step 203, the processing device determines the object group to which the target object belongs based on the attribute characteristics of the target object, and obtains the decision prompt information stored for the object group; then adjusts the target decision result based on the decision prompt information to obtain the adjusted target decision result, and generates a target response strategy applicable to the target object in the specified business scenario based on the real-time updated business data and the adjusted target decision result.
[0176] Specifically, the processing device can store corresponding decision prompt information for different object groups, wherein the decision prompt information is constructed based on the historical feedback results of previous objects on the decision, and records the situations that need to be paid attention to when determining the response strategy for this type of object group; the processing device can divide the object into different groups based on its attribute information, such as determining the object group based on the public age information and regional information.
[0177] Taking the example that the decision prompt information records: each negative sub-decision result with negative feedback, each positive sub-decision result with positive feedback and the corresponding weights, and the mutually exclusive relationship between the sub-decision results, when adjusting the target decision result based on the decision prompt information, there are the following possible operations:
[0178] It should be noted that in the embodiment of the present application, a negative sub-decision result refers to: a sub-decision result for which the subject has given negative feedback, wherein the negative result may specifically refer to operations such as blocking operations and deleting display operations that express dissatisfaction; a sub-decision result refers to a portion of the decision results included in the decision result.
[0179] For example, if the decision result is: recommending displaying type A books, type B books, and type C books, then displaying type A books can be understood as a sub-decision result.
[0180] In addition, positive sub-decision results refer to: sub-decision results for which the subjects have given positive feedback, where positive results can specifically be operations such as viewing, collecting, and purchasing that represent satisfactory emotions; the mutually exclusive relationship between sub-decision results represents the relationship between decision contents that are configured not to be executed at the same time.
[0181] For example, assuming that according to actual business needs, the configuration cannot recommend both A1 items and Am items at the same time, then for the decision result recommending Category A items, the sub-decision result: recommending item A1 and the sub-decision result: recommending item A2 are mutually exclusive.
[0182] It should be understood that, according to actual processing needs, when constructing decision prompt information, the sub-decision results for which the number of times positive results are fed back exceeds the first threshold value can be recorded as positive sub-decision, and the sub-decision results for which the number of times negative results are fed back exceeds the second threshold value can be recorded as negative sub-decision, wherein the first threshold value and the second threshold value are set according to actual processing needs and are not specifically described in this application.
[0183] See also Figure 4A As shown, it is a schematic diagram of the process of constructing a negative sub-decision in the embodiment of the present application. Figure 4A From the illustrated process, it can be seen that when the processing device constructs decision prompt information for object group 1, it is assumed that objects 1-n are included in object group 1 based on historical business data. By cumulatively counting the feedback results of each object, it is determined that the number of occurrences of "blocking item A" in the feedback results reaches the second threshold value. Then, "recommending item A" can be determined as a negative sub-decision and written into the decision prompt information corresponding to object group 1.
[0184] For example, see Figure 4B As shown, it is a schematic diagram of the process of continuing to add positive sub-decisions in an embodiment of the present application. When the processing device continues to construct decision prompt information for object group 1, it is assumed that objects 1-n are included in object group 1 based on historical data. By cumulatively counting the feedback results of each object, it is determined that the number of occurrences of "purchase C item" in the feedback results reaches a first threshold value, then "recommend item C" can be determined as a positive sub-decision and written into the decision prompt information corresponding to object group 1.
[0185] Operation 1: Delete the negative decision contents in the target decision result that match each negative sub-decision result from the target decision result.
[0186] In the embodiment of the present application, since the negative sub-decision results correspond to negative decision content that is likely to receive negative feedback, for this type of negative decision content, the processing device can directly delete the corresponding content from the obtained target decision result.
[0187] Operation 2: Determine the positive decision contents that match the positive sub-decision results in the target decision result, and delete the positive decision contents with lower weights in every two mutually exclusive positive decision contents from the target decision result.
[0188] In the embodiment of the present application, since the positive sub-decision result corresponds to the positive decision content that is likely to receive positive feedback, the processing device will try to retain this type of positive decision content in the target decision result; however, considering that there may be mutually exclusive relationships between different positive decision contents, the processing device can delete the positive sub-decision result with a lower weight among every two positive sub-decision results that have a mutually exclusive relationship from the target decision result.
[0189] It should be noted that the weight corresponding to the positive sub-decision result can be specifically determined according to the number of positive feedbacks corresponding to the positive sub-decision result, and the number of positive feedbacks and the value of the weight are positively correlated.
[0190] In this way, in the process of determining the corresponding target response strategy based on the target decision result, the processing device can take into account the feedback of the previous response strategy, and based on the decision prompt information determined by the previous feedback result, adjust the decision content in the direction of obtaining more positive feedback from the target object, which helps to increase the adaptability of the response strategy to the target object and improve the execution effect of the response strategy.
[0191] It should be understood that in the embodiment of the present application, the maintenance of decision prompt information by the processing device is equivalent to optimizing the corresponding method of strategy generation. Decisions with poor feedback effects are deleted from the determined target decision results. Moreover, by adjusting the weight of the decision according to the actual feedback results, etc., the generation performance of the response strategy can be improved.
[0192] Moreover, the maintenance of decision prompt information mainly depends on feedback information and the prediction results of the model. By analyzing the feedback information, the effect of the decision results output by the model can be evaluated; and by analyzing the prediction results of the model, the applicability of the decision results can be evaluated. Based on these evaluation results, it can be decided whether to optimize the decision results and how to optimize them. The optimization methods involved may include: deleting decision content with poor feedback effects, avoiding the execution of decisions with poor feedback effects, and deleting decision content that conflicts with other decisions; adjusting the weight of decision content, increasing the weight of decision content with positive feedback, and reducing the weight of relatively poor decision content, so that when making decisions, more consideration is given to decision content that can obtain positive feedback results.
[0193] In addition, it should be noted that decision optimization needs to ensure the effectiveness of decisions while maintaining the interpretability of decisions. Interpretability is an important feature of rule engines, which can help understand and trust the decisions of rule engines. In order to maintain the interpretability of decisions, some principles need to be followed when optimizing decisions, such as: try to keep decisions simple and avoid overly complex decisions; try to keep decisions consistent and avoid conflicting decisions; try to keep decisions complete and ensure that there are corresponding decisions to handle all possible input data.
[0194] Therefore, this application can build an adaptive rule engine that can adaptively generate and execute decision rules by combining natural language processing and deep reinforcement learning. This adaptive rule engine can process various types of input data, generate interpretable decision results, and then execute corresponding response strategies; in addition, by collecting feedback information, continuously optimizing models and decision results, it can continuously improve the effectiveness of decisions.
[0195] In an embodiment of the present application, the processing device not only considers the decision prompt information, but also considers the business data updated in real time, and then generates a corresponding target response strategy based on the target decision result, the matching decision prompt information, and the business data updated in real time.
[0196] Specifically, considering the extremely fast speed of iterative updates of information data, in order to ensure that the target object can access new data, after adjusting the target response strategy based on the decision prompt information, the target response strategy can be filled with content based on the real-time updated business data to generate a target response strategy that can be executed on the target object.
[0197] It should be noted that, in some possible implementations, in order to take into account the needs of target objects at different levels, the content filling ratio can be adaptively set for the same type of content.
[0198] For example, if the target object searches for "suspense novels" in an e-commerce shopping scenario, then the specific novels recommended to the target object can be determined based on the real-time updated novels and historical novels according to a certain proportion. For example, if a total of 10 books are pushed, at least 30% of the novels published this year; 50% of the novels published in the past 3 years; 70% of the novels published in the past 5 years; and 100% of the novels published in the past 10 years;
[0199] For another example, when making e-commerce product recommendations, assuming that the target object searches for suspense novels, then when generating a target response strategy, the recommended content can be determined based on the popularity of the writers of suspense novels that are updated in real time; or, the recommended content can be determined based on the novel popularity ranking that is updated in real time.
[0200] In this way, with the help of the dual influence of decision-making prompt information and real-time updated business data, it is possible to adaptively generate target response strategies with a high probability of obtaining positive feedback results for the target object. In the process of generating the target response strategy, on the one hand, adjustments are made with the help of historical feedback from similar groups, and on the other hand, the effectiveness of the data is improved with the help of real-time updated business data, which is conducive to generating target response strategies that are adapted to the target object and improves the scalability and flexibility of the target response strategy.
[0201] Step 204: The processing device executes a target response strategy for the target object.
[0202] Specifically, after the processing device determines the target response strategy for the target object, it executes the target response strategy for the target object, so that the target object can obtain a feedback response result after triggering the operation instruction, and then further perform status analysis based on the feedback result, and further determine the target response strategy that needs to be executed.
[0203] Furthermore, after obtaining the feedback results of the target object, the processing device stores the feedback results of the target object for the target response strategy under the target state characteristics, so that when it is determined that the update cycle of the target decision model has been reached, the training samples are regenerated based on the stored target state characteristics, target response strategy, and the correspondence between the feedback results; at the same time, the decision prompt information is updated based on the feedback results.
[0204] Specifically, after the processing device executes the target response strategy for the target object and obtains the feedback results from the target object, there are two aspects of processing. On the one hand, the decision prompt information is updated based on the feedback results, and on the other hand, the feedback results are stored to provide a basis for sample generation for subsequently re-updating the target decision model.
[0205] In other words, after the decision is executed, feedback information will be collected, including the decision results and the corresponding feedback results. This information will be used as a reward to strengthen the learning of the target decision model and to further optimize the model.
[0206] It should be noted that in the embodiments of the present application, the collection of feedback information mainly depends on the operating environment of the target object and the behavior of the target object. For example, for a recommendation system, feedback information can be obtained through user clicks, purchases, evaluations, etc.; for an autonomous driving system, feedback information can be obtained through the operating status of the vehicle, the driver's operation, etc.
[0207] In this way, not only can the collected feedback information be used to update the model parameters, optimize the decision rules, and improve the decision-making effect, but the matching decision prompt information can also be adaptively adjusted, which helps to better generate response strategies; moreover, the processing equipment, with the help of natural language processing technology and deep reinforcement learning technology, is equivalent to realizing adaptive adjustment of the response strategy, which helps to improve decision-making accuracy and the ability to solve complex problems.
[0208] The following is a specific example, taking the operation response in the product recommendation scenario of an e-commerce platform as an example to illustrate the relevant processing process:
[0209] See also Figure 5A As shown, it is a schematic diagram of the content presentation of the operation response executed in the implementation of this application, combined with the attached Figure 5A It can be seen that after the target object Z accesses the application Q on its corresponding terminal device, a search operation is triggered based on "e-book" in the presented page 1.
[0210] The processing device obtains historical business data of the target object Z, data generated by the target object Z's current access to the application Q, and attribute information of the target object Q in response to the operation instruction triggered by the target object Z on the electronic book.
[0211] When the processing device generates training samples for the product recommendation scenario, the processing method of the data generates the target state features corresponding to the target object Q based on the acquired data.
[0212] It should be noted that when training the corresponding target decision model for the product recommendation scenario of the e-commerce platform, from the perspective of generating training samples, it includes the data preprocessing stage and the feature engineering stage. In the data preprocessing stage, after obtaining the original data such as the browsing history, purchase history, and search history of the sample user, the data is cleaned, invalid data and duplicate data are deleted, and missing values are corrected. In addition, the data is converted into a format acceptable to the model (such as normalized data); after completing the preprocessing, cleaned and standard format data can be obtained.
[0213] Furthermore, in the feature engineering stage, the processing device performs feature extraction based on the preprocessed data to extract data features (such as the length of browsing history, purchase frequency, search keywords, etc.); thus, a state vector (i.e., state feature) that can describe the behavior and preferences of the sample users can be obtained.
[0214] Afterwards, during the model training process, the processing device uses the sample user's feedback information (such as purchasing behavior) as a reward and uses a deep reinforcement learning method to train the model, so that the model can learn the user's behavior and preferences, and finally obtain a trained target decision model.
[0215] Continue to combine Figure 5A For explanation, please see the attached Figure 5A In the illustrated content, the processing device uses the trained target decision model to output the target decision result according to the target state characteristics of the currently determined target object. Figure 5A As can be seen from the illustrated content, the processing device determines to recommend 'science fiction novels' and 'suspense novels' for the target object Z that searches for 'electronic books'.
[0216] Furthermore, in the decision execution phase, the processing setting determines the specific target response strategy to be executed based on the current state of the target object Z, the input data (such as search keywords, browsed products, etc.), and the target decision results output by the model.
[0217] Specifically, the processing device can use the internal adaptive rule engine to combine the current state and input data to determine the response strategy, such as Figure 5A In the illustrated content, specific results corresponding to science fiction novels and specific results corresponding to suspense novels were found, and then recommended results were obtained.
[0218] Based on this, in the process of determining the recommendation results, adaptive recommendations can be achieved by considering the historical behavior, attribute information, and current context of the target object Z.
[0219] Correspondingly, the processing device executes the recommendation result for the target object Z, and displays the specific recommended content on page 2 of the application Q associated with the target object.
[0220] For further information, see Figure 5B As shown, it is a schematic diagram of the process of the processing device obtaining feedback results in the implementation of this application. Figure 5B , the process of processing based on the feedback results is explained:
[0221] Combined with Figure 5A and 5B From the recorded technical contents, it can be known that after the processing device executes the target response strategy for the target object, the recommendation results are presented on page 2 of the application Q associated with the target object Z; thereafter, depending on the different operations performed by the target object Z on the recommendation results, there are positive feedback results or negative feedback results.
[0222] Continue to combine Figure 5BTo illustrate the schematic content, when the processing device determines that the target object Z clicks on a specific book, it can be determined that the target object Z is interested in the content in the recommendation result. At this time, it can be considered that positive feedback from the target object Z is obtained, that is, the recommended content is clicked; or, when the target object Z views the recommendation results presented in page 2 and triggers a return operation, it can be considered that the current recommendation results do not contain the content that the target object 2 is interested in, and therefore the feedback result obtained is a negative feedback result.
[0223] Furthermore, after the processing device determines the feedback result of the target object Z based on the target response strategy, feedback learning and decision optimization are performed based on the obtained feedback result.
[0224] Specifically, in the process of feedback learning, the object's positive feedback on the recommendation results, such as clicks, purchases, collections, etc., can be used as rewards for reinforcement learning to further optimize the model and decision results, so that an optimized model and decision can be obtained.
[0225] In addition, during the process of strategy optimization, decisions can be optimized based on feedback results, such as deleting decision content with poor results, adjusting the weight of decision content, etc. Specifically, by constructing decision prompt information, the target decision results output by the model can be adjusted to obtain optimized decision results.
[0226] In the technical solution disclosed in the present application, it is possible to respond to the operation instructions of the target object, generate an adaptive response strategy according to the state of the target object, display the recommendation results to the target object in the e-commerce shopping scenario, and continuously optimize the model and decision content according to the feedback of the target object to improve the recommendation effect.
[0227] In summary, in the embodiments of the present application, the functional implementation includes the following parts, namely, the implementation of an adaptive rule engine, the integration of natural language processing, and the application of deep reinforcement learning. When implementing an adaptive rule engine, the target decision model obtained by deep reinforcement learning training can automatically adjust the product strategy according to the user's real-time behavior and feedback, which makes the operational response of the present application highly adaptive and dynamic; moreover, the processing device uses natural language processing technology to extract richer features. The present application proposes to use natural language processing technology to process text information such as search queries and comments of related objects, and extract useful features from them. In the field of recommendation / risk control, this can not only improve the accuracy of recommendation and recognition, but also realize the diversity and novelty of feature considerations; in addition, unlike the traditional recommendation / risk control system, which ignores the sequence behavior, the present application can effectively deal with the sequence problem by adopting deep reinforcement learning, and can improve the accuracy of recommendation and identification of malicious users in scenarios such as content recommendation and risk control.
[0228] Based on the same inventive concept, refer to Figure 6 As shown, it is a schematic diagram of the logical structure of the operation response device in the embodiment of the present application. The operation response device 600 includes a response unit 601, an acquisition unit 602, a generation unit 603, and an execution unit 604, wherein:
[0229] The response unit 601 is used to obtain target state characteristics associated with the target object in response to the operation instruction triggered by the target object, wherein the target state characteristics include: operation characteristics used to describe the operation status of the target object in a specified business scenario, and attribute characteristics publicly displayed by the target object;
[0230] The obtaining unit 602 is used to use the trained target decision model to obtain the corresponding target decision result based on the target state characteristics, wherein the target decision model is trained by using a deep reinforcement learning training method, after presetting each state characteristic and each decision result for a specified business scenario, based on the sample objects corresponding to each state characteristic, and based on the historical feedback of each decision result;
[0231] A generating unit 603 is used to generate a corresponding target response strategy based on the target decision result, the decision prompt information stored for the object group to which the target object belongs, and the business data updated in real time;
[0232] The execution unit 604 is used to execute the target response strategy for the target object.
[0233] Optionally, when acquiring the target state feature associated with the target object, the response unit 601 is used to:
[0234] Obtain the business data of the target object in the specified business scenario, where the business data includes historical operation data, current operation data, and currently publicly displayed attribute information;
[0235] The business data is normalized, and data content irrelevant to the operation of the target object is deleted from the business data. Feature extraction is performed based on the processed business data to obtain the target state feature associated with the target object.
[0236] Optionally, when generating a corresponding target response strategy based on the target decision result, the decision prompt information stored for the object group to which the target object belongs, and the real-time updated business data, the generating unit 603 is used to:
[0237] Based on the attribute characteristics, determine the object group to which the target object belongs, and obtain the decision prompt information stored for the object group;
[0238] Based on the decision prompt information, the target decision result is adjusted to obtain the adjusted target decision result, and based on the real-time updated business data and the adjusted target decision result, a target response strategy applicable to the target object in the specified business scenario is generated.
[0239] Optionally, the decision prompt information records: each negative sub-decision result with negative feedback, each positive sub-decision result with positive feedback and the corresponding weight, and the mutually exclusive relationship between the sub-decision results; when adjusting the target decision result based on the decision prompt information, the generation unit 603 is used to perform any one or combination of the following:
[0240] Deleting the negative decision contents in the target decision result that match the negative sub-decision results from the target decision result;
[0241] In determining the target decision result, each positive decision content that matches each positive sub-decision result is deleted from the target decision result in which the positive sub-decision result with the lower weight is deleted from every two positive sub-decision results that are mutually exclusive.
[0242] Optionally, during the process of training to obtain the target decision model, the device further includes a training unit 605, and the training unit 605 is used to perform the following operations:
[0243] Generate training samples based on historical business data of each sample object in a specified business scenario; a training sample includes: information groups associated with a sample object in a time-series arrangement, each information group is used to describe a process in which a sample object is guided to process a target business, and an information group includes a sample state feature, a sample decision result, and a sample feedback result;
[0244] The constructed initial decision model is trained for multiple rounds of iterations using various training samples until the preset convergence conditions are met, thereby obtaining the trained target decision model.
[0245] Optionally, during a round of iterative training, the training unit 605 is configured to perform the following operations:
[0246] Read a training sample and obtain each information group in a time sequence in the training sample;
[0247] Get each group of information in chronological order and do the following:
[0248] The sample state features in the information group are input into the initial decision model to obtain the predicted feedback results of the initial decision model on the corresponding sample decision results, and the model parameters of the initial decision model are adjusted according to the difference between the predicted feedback results and the corresponding sample feedback results.
[0249] Optionally, when generating each training sample based on the historical business data of each sample object in a specified business scenario, the training unit 605 is used to:
[0250] Obtain the historical business data of each sample object in a specified business scenario within a preset time period;
[0251] Each historical business data is normalized to obtain each process data, wherein one process data includes: data describing the state time series change process of a sample object in the process of being guided to process a target business;
[0252] For each process data, each information group involved in the temporal change of the sample state feature of the corresponding sample object is determined, and a training sample is generated based on each information group.
[0253] Optionally, when normalization processing is performed on each historical business data to obtain each process data, the training unit 605 is used to:
[0254] In each historical business data, delete the historical business data with abnormal values of the specified content, and for the remaining historical business data, correct the values of other contents and adjust the value range of each numerical content;
[0255] For each processed historical business data, perform the following operations respectively:
[0256] Natural language processing technology is used to perform semantic identification processing on each text content in the processed historical business data to obtain corresponding process data.
[0257] Optionally, when executing the target response strategy for the target object, the execution unit 604 is used to:
[0258] Storing the feedback results of the target object under the target state characteristics for the target response strategy, so that when it is determined that the update cycle of the target decision model has been reached, the training samples are regenerated based on the stored correspondence between the target state characteristics, the target response strategy, and the feedback results;
[0259] Based on the feedback results, update the decision prompt information.
[0260] After introducing the operation response method and device according to the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.
[0261] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.
[0262] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application, referring to Figure 7 As shown, it is a schematic diagram of a hardware structure of an electronic device using an embodiment of the present application, and the electronic device 700 may include at least a processor 701 and a memory 702. Among them, the memory 702 stores program codes, and when the program codes are executed by the processor 701, the processor 701 performs the steps of any of the above-mentioned operation responses.
[0263] In some possible implementations, the computing device according to the present application may include at least one processor and at least one memory. The memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the operation response according to various exemplary implementations of the present application described above in this specification. For example, the processor may execute the following steps: Figure 2 Follow the steps shown in .
[0264] Refer to the following Figure 8 800 according to this embodiment of the present application. Figure 8 As shown, the computing device 800 is in the form of a general computing device. The components of the computing device 800 may include but are not limited to: at least one processing unit 801, at least one storage unit 802, and a bus 803 connecting different system components (including the storage unit 802 and the processing unit 801).
[0265] Bus 803 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.
[0266] The storage unit 802 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 8021 and / or a cache memory 8022 , and may further include a read-only memory (ROM) 8023 .
[0267] The storage unit 802 may also include a program / utility 8025 having a set (at least one) of program modules 8024, such program modules 8024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0268] The computing device 800 may also communicate with one or more external devices 804 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable objects to interact with the computing device 800, and / or communicate with any device that enables the computing device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 805. In addition, the computing device 800 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 806. As shown, the network adapter 806 communicates with other modules for the computing device 800 via a bus 803. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computing device 800, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0269] Based on the same inventive concept as the above method embodiment, various aspects of the operation response provided by the present application can also be implemented in the form of a program product, which includes a program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the operation response method according to various exemplary embodiments of the present application described above in this specification. For example, the electronic device can execute the following steps: Figure 2 Follow the steps shown in .
[0270] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0271] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0272] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. An operation response method, characterized in that: include: In response to an operation indication triggered by a target object, a target state characteristic associated with the target object is obtained, wherein the target state characteristic includes: an operation characteristic used to describe the operation status of the target object in a specified business scenario, and an attribute characteristic publicly displayed by the target object; The trained target decision model is used to obtain the corresponding target decision result based on the target state characteristics, wherein the target decision model is trained by using a deep reinforcement learning training method, after presetting each state characteristic and each decision result for the specified business scenario, based on the sample objects corresponding to each state characteristic, and based on the historical feedback of each decision result; Generate a corresponding target response strategy based on the target decision result, the decision prompt information stored for the object group to which the target object belongs, and the real-time updated business data; The target response strategy is executed for the target object.
2. The method according to claim 1, characterized in that The acquiring the target state feature associated with the target object includes: Acquire business data of the target object in a specified business scenario, wherein the business data includes historical operation data, current operation data, and currently publicly displayed attribute information; The business data is normalized, and data content irrelevant to the operation of the target object is deleted from the business data. Feature extraction is performed based on the processed business data to obtain target state features associated with the target object.
3. The method according to claim 1, characterized in that Based on the target decision result, the decision prompt information stored for the object group to which the target object belongs, and the real-time updated business data, a corresponding target response strategy is generated, including: Based on the attribute characteristics, determining the object group to which the target object belongs, and acquiring decision prompt information stored for the object group; Based on the decision prompt information, the target decision result is adjusted to obtain an adjusted target decision result, and based on the real-time updated business data and the adjusted target decision result, a target response strategy applicable to the target object in the specified business scenario is generated.
4. The method according to claim 3, characterized in that The decision prompt information records: each negative sub-decision result with negative feedback, each positive sub-decision result with positive feedback and the corresponding weight, and the mutually exclusive relationship between the sub-decision results; when adjusting the target decision result based on the decision prompt information, any one or combination of the following is performed: Deleting the negative decision contents in the target decision result that match the negative sub-decision results from the target decision result; Determine the positive decision contents in the target decision result that match the positive sub-decision results, and delete the positive sub-decision result with a lower weight in every two positive sub-decision results that have a mutually exclusive relationship from the target decision result.
5. The method according to claim 1, characterized in that During the training process to obtain the target decision model, the following operations are performed: Generate each training sample based on the historical business data of each sample object in the specified business scenario; A training sample includes: information groups associated with a sample object in a time-sequential arrangement, wherein each information group is used to describe a process in which a sample object is guided to process a target service, and an information group includes a sample state feature, a sample decision result, and a sample feedback result; The constructed initial decision model is trained multiple times using the training samples until a preset convergence condition is met, thereby obtaining a trained target decision model.
6. The method according to claim 5, characterized in that During one round of training iterations, the following operations are performed: Reading a training sample to obtain information groups arranged in time sequence in the training sample; Obtain each information group in turn in the described chronological order and perform the following operations: The sample state features in the information group are input into the initial decision model to obtain the predicted feedback results of the initial decision model on the corresponding sample decision results, and the model parameters of the initial decision model are adjusted according to the result difference between the predicted feedback results and the corresponding sample feedback results.
7. The method according to claim 5, characterized in that The generating of each training sample based on the historical business data of each sample object in the specified business scenario includes: Obtain the historical business data of each sample object in a specified business scenario within a preset time period; Each historical business data is normalized to obtain each process data, wherein one process data includes: data describing the state time series change process of a sample object in the process of being guided to process a target business; For each process data, each information group involved in the temporal change process of the sample state feature of the corresponding sample object is determined, and a training sample is generated based on the information groups.
8. The method according to claim 7, characterized in that The above-mentioned normalization processing is performed on each historical business data to obtain each process data, including: In each historical business data, delete the historical business data with abnormal values of the specified content, and for the remaining historical business data, correct the values of other contents and adjust the value range of each numerical content; For each processed historical business data, perform the following operations respectively: Natural language processing technology is used to perform semantic identification processing on each text content in the processed historical business data to obtain corresponding process data.
9. The method according to any one of claims 1 to 8, characterized in that The executing the target response strategy for the target object includes: Storing feedback results of the target object under the target state characteristics for the target response strategy, so that when it is determined that the update cycle of the target decision model has been reached, training samples are regenerated based on the stored correspondence between the target state characteristics, the target response strategy, and the feedback results; Based on the feedback result, the decision prompt information is updated.
10. An operation response device, characterized in that: include: A response unit, configured to obtain target state characteristics associated with the target object in response to an operation indication triggered by the target object, wherein the target state characteristics include: operation characteristics used to describe the operation status of the target object in a specified business scenario, and attribute characteristics publicly displayed by the target object; An obtaining unit is used to adopt a trained target decision model to obtain a corresponding target decision result based on the target state feature, wherein the target decision model is trained by adopting a deep reinforcement learning training method, after presetting each state feature and each decision result for the specified business scenario, based on the sample objects corresponding to each state feature, and trained for the historical feedback of each decision result; A generating unit, configured to generate a corresponding target response strategy based on the target decision result, the decision prompt information stored for the object group to which the target object belongs, and the business data updated in real time; An execution unit is used to execute the target response strategy for the target object.
11. The device according to claim 10, characterized in that When acquiring the target state feature associated with the target object, the response unit is used to: Acquire business data of the target object in a specified business scenario, wherein the business data includes historical operation data, current operation data, and currently publicly displayed attribute information; The business data is normalized, and data content irrelevant to the operation of the target object is deleted from the business data. Feature extraction is performed based on the processed business data to obtain target state features associated with the target object.
12. The device according to claim 10, characterized in that When generating a corresponding target response strategy based on the target decision result, the decision prompt information stored for the object group to which the target object belongs, and the real-time updated business data, the generating unit is used to: Based on the attribute characteristics, determining the object group to which the target object belongs, and acquiring decision prompt information stored for the object group; Based on the decision prompt information, the target decision result is adjusted to obtain an adjusted target decision result, and based on the real-time updated business data and the adjusted target decision result, a target response strategy applicable to the target object in the specified business scenario is generated.
13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.
14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.
15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.
