Data poisoning attack and defense method for offline reinforcement learning

Abstract

The invention relates to a data poisoning attack and defense method for offline reinforcement learning, and belongs to the technical field of offline reinforcement learning. The method comprises the following steps: firstly, acquiring a state space, an action space and a reward space in an offline reinforcement learning data set, searching a key moment in an offline reinforcement learning process according to the change condition of a state action pair at each moment in the data set, and adding tiny disturbance smaller than the proportion of the data to the data corresponding to the key moment; and data poisoning is carried out on the offline reinforcement learning data set. By implementing the attack and performing system analysis on the attack and defense process, fragile links which are utilized on the basis of feature distance distribution anomaly exist in an off-line RL data set, a feature distance distribution anomaly-based detection method is further provided and is used for automatically detecting, marking and processing suspicious samples before training, and the detection accuracy of the suspicious samples is improved. Therefore, a closed-loop technical scheme of'providing attacks-discovering vulnerabilities-corresponding protection 'is formed, so that the security and robustness of an offline reinforcement learning system are enhanced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of offline reinforcement learning, and particularly relates to a data poisoning attack and defense method for offline reinforcement learning. BACKGROUND

[0002] Offline reinforcement learning, also known as batch reinforcement learning, is a paradigm in which an agent learns from a previously collected dataset of experiences. Unlike online reinforcement learning, which requires real-time feedback from the environment to improve the policy, offline RL can learn without online interaction with the environment, making it suitable for exploring scenarios that are expensive, time-consuming, or risky, such as autonomous driving, healthcare decision-making, intelligent robot control, and game design.

[0003] Data poisoning attacks have been widely present in the field of machine learning, and early attacks were mainly aimed at basic models such as logistic regression and support vector machine (Marco Barreno, Blaine Nelson, Anthony D. Joseph, and J. D. Tygar. The security of machine learning. Mach. Learn., 81(2):121-148, November 2010. ISSN 0885-6125. doi: 10.1007 / s10994-010-5188-5.). In recent years, with the development of deep learning, attackers have gradually shifted their focus to deep networks (Feng, J., Cai, Q.-Z., and Zhou, Z.-H. Learning to confuse: generating training time adversarial data with autoencoder. Advances in Neural Information Processing Systems, 32, 2019.). These works show that data security has become a problem that cannot be ignored. As an important branch of current machine learning, RL also has data security problems. At present, there are more and more in-depth studies on data poisoning attacks in online RL (Rakhsha, A., Zhang, X., Zhu, X., and Singla, A. Reward poisoning in reinforcement learning: Attacks against unknown learners in unknown environments. arXiv preprint arXiv:2102.08492, 2021.).

[0004] The data poisoning problem exists in online RL, and offline RL also faces such threats. The above work on data poisoning in online RL provides many ideas for the research of offline RL. In 2019, Ma (Ma, Y., Zhang, X., Sun, W., and Zhu, J. Policy poisoning in batch reinforcement learning and control. Advances in Neural Information Processing Systems, 2019.) first attacked the reinforcement learning dataset, taking batch reinforcement learning and controller as victims, and attacking at each time step to affect the performance of the agent, but the cost of attacking each time step is high; In 2020, Rakhsha (Rakhsha, A., Radanovic, G., Devidze, R., Zhu, X., and Singla, A. Policy teaching via environment poisoning: Training-time adversarial attacks against reinforcement learning. In International Conference on Machine Learning, pp. 7974-7984. PMLR, 2020.) turned the attack into an optimization problem, and found the best attack for different attack costs, but the attack proposed did not target more realistic continuous tasks, and the attack modified the reward and state transition function at the same time, increasing the cost of the attack; In 2022, Gong (Gong, C., Yang, Z., Bai, Y., He, J., Shi, J., Sinha, A., Xu, B., Hou, X., Fan, G., and Lo, D. Mind your data! hiding backdoors in offline reinforcement learning datasets. arXiv preprint arXiv:2210.04688, 2022.) proposed a backdoor attack for offline reinforcement learning, which controlled the attack by adding triggers, but the poisoning method was random poisoning, and the poisoning ratio needed to reach 10% for better attack effect, which was costly, and the backdoor attack needed to be continuously operated in the model training and testing phase to take effect, which had high requirements for the attacker; In 2022, Wu (Wu, F., Li, L., Xu, C., Zhang, H.Kailkhura, B., Kenthapadi, K., Zhao, D., and Li, B. *Copa: Certifying robust policies for offline reinforcement learning against poisoning attacks. arXiv preprint arXiv:2203.08398, 2022.* proposed a certification framework for the robustness of offline reinforcement learning policies, which can evaluate the robustness of algorithms. In 2022, Rang (Rangi, A., Xu, H., Tran-Thanh, L., and Franceschetti, M. *Understanding the limits of poisoning attacks in episodic reinforcement learning. arXiv preprint arXiv:2208.13663, 2022.*) theoretically proved the limits of poisoning attacks in reinforcement learning (including offline and online), but existing work lacks analysis of offline datasets and the impact on offline algorithms and models. Research is needed on low-cost and efficient poisoning attack and defense methods, while also deeply analyzing the impact of these attacks. Summary of the Invention

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A data poisoning attack method targeting offline reinforcement learning includes the following steps:

[0007] Step 1: Obtain the state space, action space, and reward space from the offline dataset;

[0008] Step 2: Encode the features of the state-action pairs corresponding to the trajectory at each time step to obtain the feature vector corresponding to each time step;

[0009] Step 3: Input the feature vector into the feature distance calculator to calculate the Euclidean distance between feature vectors at adjacent time points, which is used as the feature distance;

[0010] Step 4: Sort the feature distance values ​​from largest to smallest, and obtain the key point set according to the poisoning ratio;

[0011] Step 5: For each state-action pair in the key point set, add a random perturbation that meets preset constraints to generate a toxic dataset, thus completing the poisoning of the offline dataset.

[0012] Furthermore, for the state-action pair corresponding to the trajectory at each moment... Perform feature encoding, where The state at time t, The action at time t includes the following steps:

[0013] Step 2.1: Process the state corresponding to the trajectory at each time step. Flatten the input state into 1D and convert it into an integer type. Then flatten the input action data into 1D and append it to the state data along dimension 1 to form a new state-action pair. :

[0014]

[0015] Step 2.2: Train a normally behaving agent model using a clean offline reinforcement learning dataset, obtain the model parameters, and use them as the feature encoder. ;

[0016] Step 2.3: Set the new state-action pair value Input to the trained feature encoder In the middle, output feature array That is, the eigenvector.

[0017] Further, the Euclidean distance between feature vectors at adjacent time points is calculated, including the following steps:

[0018] Step 3.1: Add the feature array back into the original dataset as a new column according to the sequence order to obtain the feature vector dataset;

[0019] Step 3.2: For each data point in the dataset, if it is the first data point in the current round, set the feature distance corresponding to that data point to 0. If the data point is not the first data point in the current round, calculate the Euclidean distance between the features of that data point and the data from the previous time step. , as the feature distance;

[0020]

[0021] in, For the eigenvector of the th Dimension; if it is the start time of the sequence, then it is specified that... ;

[0022] Step 3.3: Calculate the Euclidean distance value corresponding to each time step. Store it as a new column in the feature vector dataset to obtain the feature distance dataset.

[0023] Furthermore, the feature distances corresponding to all times are sorted from largest to smallest, and a set of key points is obtained according to the poisoning ratio, including the following steps:

[0024] Step 4.1: Based on the poisoning ratio Determine the number of key points ;

[0025] Step 4.2: Sort the feature distance dataset in descending order of the value of the feature distance column, and obtain the indexes of the key points that meet the specified number of key points K in descending order.

[0026] Further, generating a toxic dataset includes the following steps:

[0027] Step 5.1: Based on the index of the key point obtained in Step 4, obtain the value of the state-action pair in the corresponding key point. ;

[0028] Step 5.2: Set the perturbation ratio to... Calculate the upper limit of perturbation for each key point:

[0029]

[0030] Step 5.3: For each key point, generate the perturbation factor. ,satisfy That is, a random value whose absolute value is less than the upper limit of the disturbance;

[0031] Step 5.4: Generate the perturbation factor The corresponding state-action pair added to each keypoint:

[0032]

[0033] The toxic dataset was obtained.

[0034] Furthermore, it includes the following steps:

[0035] Step 101: Construct a complete dataset containing the normal sample set and the toxic dataset, and calculate the feature vector for each trajectory in the complete dataset at each time step in chronological order. ;

[0036] Step 102: Based on the feature vector described in Step 101, calculate the feature Euclidean distance between adjacent samples in the dataset to form a feature distance sequence. ,

[0037] Step 103: For the aforementioned eigendistance sequence Anomaly detection is performed to identify abnormal samples, and interpolation is performed to repair the state-action pairs corresponding to the identified abnormal samples.

[0038] Furthermore, the identification of anomalous samples includes the following steps:

[0039] Global feature distance sequence for the obtained complete dataset To perform a normality test, the Shapiro-Wilk test is preferred to determine whether the sequence approximately follows a normal distribution. The null hypothesis of the Shapiro-Wilk test is "the sample comes from a normal distribution," and its test statistic is denoted as... and corresponding value ;like ≥ ,in, If a pre-set significance level threshold is set, the normality assumption is accepted; otherwise, if there are signs of significant deviation from normality, the process proceeds to the outlier identification process.

[0040] Calculate the mean of the feature distances. with standard deviation And define a global anomaly threshold. Where k=3, the feature distance sequence > The samples were marked as anomalous.

[0041] Furthermore, the interpolation repair of the state-action pairs corresponding to the identified abnormal samples adopts an automatic correction method based on neighboring time-series interpolation, specifically as follows:

[0042] The original state-action vector of the heterogeneous sample is The vector at the previous moment is Interpolation correction is performed using the following formula:

[0043]

[0044] Among them, the correction factor To reduce the impact of suspected mutations on the model, we used a small positive number, 𝜂=0.05.

[0045] A computer device / apparatus / system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of a data poisoning attack and defense method against offline reinforcement learning.

[0046] A computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of a data poisoning attack and defense method against offline reinforcement learning.

[0047] The beneficial effects of this invention are as follows:

[0048] 1. This invention proposes a data poisoning method for offline reinforcement learning. It can identify trajectories that significantly impact the learning task during offline reinforcement learning and poison key points, greatly reducing attack costs and improving attack efficiency. Furthermore, this invention is universally applicable to different offline reinforcement learning algorithms and tasks. The proposed attack method has been validated on three offline reinforcement learning algorithms: Batch-Constrained Q-learning (BCQ), Batch-Ensemble Actor-Critic with Retrace (BEAR), and Conservative Q-Learning (CQL); and on the Walker2D, Hopper, and Half-Cheetah tasks in the MuJoCo robot simulator, and the Carla-Lane autonomous driving task in the Carla simulator. It achieves an average performance reduction of over 85% for agents trained on poisoned data compared to those trained on clean datasets when the poisoning ratio is only 1%; and an over 85% performance reduction when the poisoning ratio is 5%.

[0049] 2. The perturbation method proposed in this invention can dynamically add tiny perturbations to the dataset in proportion to itself. The changes are very small and difficult to detect compared with the original data. The changes are less than 0.05 times the original data, which improves the concealment of the attack.

[0050] 3. This invention proposes an attack method and a detection and correction method based on anomalies in feature distance distribution, and provides an engineerable data review scheme, enabling defenders to detect and reduce the harm of the attack before training;

[0051] In summary, the data poisoning method for offline reinforcement learning proposed in this patent has good attack effectiveness, significantly impacting different offline reinforcement learning algorithms and tasks, and is highly efficient and covert. Furthermore, it proposes targeted detection and defense methods, achieving a closed-loop security engineering practice of "proposing attacks—corresponding protection." Attached Figure Description

[0052] Figure 1 An overview diagram of the data poisoning method for offline reinforcement learning provided by this invention;

[0053] Figure 2 Feature encoder model diagram;

[0054] Figure 3 Overview diagram of anomaly detection and data repair methods. Detailed Implementation

[0055] The present invention will now be further described with reference to the accompanying drawings.

[0056] The purpose of this invention is to provide a data poisoning attack and defense method for offline reinforcement learning. By adding small perturbations to points that have a significant impact on the learning process, the attack's stealth can be increased while improving its efficiency. Simultaneously, a feature distance distribution-based anomaly detection and defense method is proposed.

[0057] like Figure 1 As shown in Figures 2 and 3, the present invention includes the following steps:

[0058] Step 1: Obtain the state space, action space, and reward space from a clean offline dataset;

[0059] Step 2: Assign state-action pairs to the trajectory at each moment. The input feature encoder is used for encoding to obtain the feature vectors corresponding to each time step;

[0060] 201) Process the state corresponding to the trajectory at each time step, flatten the input state into 1 dimension and convert it into an integer type, then flatten the input action data into 1 dimension and concatenate it along the 1st dimension to the state data to form a new state-action pair input:

[0061]

[0062] 202) Train a normally behaving agent model using a clean offline reinforcement learning dataset, obtain the model parameters of the model, and use them as a feature encoder. ;

[0063] 203) The new state-action pair value obtained in 201) The encoder trained by inputting into 202) In the process, the output feature array is obtained. This is the eigenvector.

[0064] Step 3: Input the feature vector into the feature distance calculator to calculate the Euclidean distance between feature vectors at adjacent time points, which is used as the feature distance;

[0065] 301) Add the feature array back to the original dataset as a new column according to the sequence order to obtain the feature vector dataset;

[0066] 302) For each data point in the dataset, if it is the first data point in the current round, set the feature distance corresponding to that data point to 0. If the data point is not the first data point in the current round, calculate the Euclidean distance between the features of that data point and the data from the previous time step. , as the feature distance;

[0067]

[0068] in, For the eigenvector of the th Dimension; if it is the start time of the sequence, then it is specified that... ;

[0069] 303) Calculate the Euclidean distance value corresponding to each time step. Store it as a new column in the feature vector dataset to obtain the feature distance dataset.

[0070] Step 4: Sort the feature distance values ​​from largest to smallest, and obtain the key point set according to the poisoning ratio;

[0071] 401) Based on the poisoning ratio Determine the number of key points For example, if the total number of data points in the dataset is 1 million, and the poisoning ratio is 1%, it means that the number of key points is 10,000.

[0072] 402) Sort the feature distance dataset in descending order of the value of the feature distance column, and obtain the index of the key points that meet the required number of key points in descending order of the number of key points determined in 401).

[0073] Step 5: For each state-action pair in the key point set, add a random perturbation that meets preset constraints to generate a toxic dataset, thus completing the poisoning of the offline dataset.

[0074] 501) Based on the index of the key point obtained in step 4, obtain the value of the state-action pair in the corresponding key point. ;

[0075] 502) Set the perturbation ratio to Calculate the upper limit of perturbation for each key point:

[0076]

[0077] 503) For each key point, generate a perturbation factor. ,satisfy That is, a random value whose absolute value is less than the upper limit of the disturbance;

[0078] 504) will generate a disturbance factor. The corresponding state-action pair added to each keypoint:

[0079]

[0080] Obtain the toxic dataset and carry out a data poisoning attack.

[0081] Step 6: Construct a complete dataset containing the normal sample set and the toxic dataset, and calculate the feature vector for each trajectory in the complete dataset at each time step in chronological order. ;

[0082] 601) Calculate the feature vector for each time step of each trajectory in the complete dataset in chronological order. ;

[0083] 602) Based on Calculate the feature Euclidean distance between adjacent samples in the dataset to form a feature distance sequence. (hereinafter referred to as "feature distance");

[0084] 603) Regarding the aforementioned distance sequence For anomaly detection, the Shapiro–Wilk test is preferred to determine whether the sequence approximately follows a normal distribution. The null hypothesis of the Shapiro–Wilk test is "the sample comes from a normal distribution," and its test statistic is denoted as [test statistic]. and corresponding value ;like ≥ (For example =0.05), accept the normality hypothesis; otherwise, determine that there are signs of significant deviation from normality and proceed to the outlier identification process.

[0085] 604) Under the baseline distribution model established based on a clean dataset, calculate the statistic of the feature distance: mean. with standard deviation And define a global anomaly threshold:

[0086]

[0087] Where k=3 (i.e., the commonly used three sigma rule). For any time, if... If the sample is marked as an anomalous candidate, then the feature anomaly identification of the complete dataset containing poisoned samples and normal samples can be achieved.

[0088] Step 7: Interpolate and repair the state-action pairs corresponding to the identified abnormal samples;

[0089] For state-action pairs that are judged to be anomalous and have high confidence, an automatic correction method based on neighboring time-series interpolation is proposed, specifically as follows:

[0090] The original state-action vector of the heterogeneous sample is The vector at the previous moment is Interpolation correction is performed using the following formula:

[0091]

[0092] Among them, the correction factor To reduce the impact of suspected mutations on the model, a small positive number is preferred, consistent with the poisoning perturbation range described in this specification. (i.e., the 5% rule), thus obtaining the correction value.

[0093] In particular, in some preferred embodiments of the present invention, a computer device is also provided, including a memory and a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the data poisoning attack and defense method against offline reinforcement learning described in any of the above embodiments.

[0094] In some other preferred embodiments of the present invention, a computer-readable storage medium is also provided, on which a computer program / instruction is stored, wherein when the computer program is executed by a processor, the steps of the data poisoning attack and defense method against offline reinforcement learning described in any of the above embodiments are implemented.

[0095] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above embodiments of the data poisoning attack and defense method for offline reinforcement learning, which will not be repeated here.

[0096] Computer-readable storage media encompass a variety of types, including persistent and non-persistent, portable and fixed. These media store information using different technologies, and the content can be machine instructions, data structures, program modules, or other types of data. Some typical examples of computer storage media include: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), various types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory and other storage technologies, optical storage media such as CD-ROM and digital video disc (DVD), magnetic storage devices such as magnetic tape and disks, and other non-transferable media used to store information accessible to computing devices. It is important to note that the computer-readable media described herein do not include temporary storage media, such as modulated data signals and carrier waves.

[0097] Those skilled in the art will further recognize that the operation of the module can be achieved using existing technical protocols or programs, without relying on new computer programs themselves. The units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0098] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0099] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data poisoning attack method targeting offline reinforcement learning, characterized in that, Includes the following steps: Step 1: Obtain the state space, action space, and reward space from the offline dataset; Step 2: Encode the features of the state-action pairs corresponding to the trajectory at each time step to obtain the feature vector corresponding to each time step; Step 3: Input the feature vector into the feature distance calculator to calculate the Euclidean distance between feature vectors at adjacent time points, which is used as the feature distance; Step 4: Sort the feature distance values ​​from largest to smallest, and obtain the key point set according to the poisoning ratio; Step 5: For each state-action pair in the key point set, add a random perturbation that meets preset constraints to generate a toxic dataset, thus completing the poisoning of the offline dataset.

2. The data poisoning attack method for offline reinforcement learning according to claim 1, characterized in that, In step 2, the state-action pair corresponding to the trajectory at each time point is... Perform feature encoding, where The state at time t, The action at time t includes the following steps: Step 2.1: Process the state corresponding to the trajectory at each time step. Flatten the input state into 1D and convert it into an integer type. Then flatten the input action data into 1D and append it to the state data along dimension 1 to form a new state-action pair. : Step 2.2: Train a normally behaving agent model using a clean offline reinforcement learning dataset, obtain the model parameters, and use them as the feature encoder. ; Step 2.3: Set the new state-action pair value Input to the trained feature encoder In the middle, output feature array That is, the eigenvector.

3. The data poisoning attack method for offline reinforcement learning according to claim 2, characterized in that, Step 3 involves calculating the Euclidean distance between feature vectors at adjacent time points, including the following steps: Step 3.1: Add the feature array back into the original dataset as a new column according to the sequence order to obtain the feature vector dataset; Step 3.2: For each data point in the dataset, if it is the first data point in the current round, set the feature distance corresponding to that data point to 0. If the data point is not the first data point in the current round, calculate the Euclidean distance between the features of that data point and the data from the previous time step. , as the feature distance; in, For the eigenvector of the th Dimension; if it is the start time of the sequence, then it is specified that... ; Step 3.3: Calculate the Euclidean distance value corresponding to each time step. Store it as a new column in the feature vector dataset to obtain the feature distance dataset.

4. The data poisoning attack method for offline reinforcement learning according to claim 3, characterized in that, In step 4, the feature distances corresponding to all times are sorted from largest to smallest, and a set of key points is obtained according to the poisoning ratio, including the following steps: Step 4.1: Based on the poisoning ratio Determine the number of key points ; Step 4.2: Sort the feature distance dataset in descending order of the value of the feature distance column, and obtain the indexes of the key points that meet the specified number of key points K in descending order.

5. A data poisoning attack method for offline reinforcement learning according to claim 4, characterized in that, Step 5, generating the toxic dataset, includes the following steps: Step 5.1: Based on the index of the key point obtained in Step 4, obtain the value of the state-action pair in the corresponding key point. ; Step 5.2: Set the perturbation ratio to... Calculate the upper limit of perturbation for each key point: Step 5.3: For each key point, generate the perturbation factor. ,satisfy That is, a random value whose absolute value is less than the upper limit of the disturbance; Step 5.4: Generate the perturbation factor The corresponding state-action pair added to each keypoint: The toxic dataset was obtained.

6. A defense method against data poisoning attacks targeting offline reinforcement learning, as described in claim 5, characterized in that, Includes the following steps: Step 101: Construct a complete dataset containing the normal sample set and the toxic dataset, and calculate the feature vector for each trajectory in the complete dataset at each time step in chronological order. ; Step 102: Based on the feature vector described in Step 101, calculate the feature Euclidean distance between adjacent samples in the dataset to form a feature distance sequence. ; Step 103: For the aforementioned eigendistance sequence Anomaly detection is performed to identify abnormal samples, and interpolation is performed to repair the state-action pairs corresponding to the identified abnormal samples.

7. A defense method against data poisoning attacks targeting offline reinforcement learning as described in claim 6, characterized in that, The identification of abnormal samples includes the following steps: Global feature distance sequence for the obtained complete dataset To perform a normality test, the Shapiro-Wilk test is preferred to determine whether the sequence approximately follows a normal distribution. The null hypothesis of the Shapiro-Wilk test is "the sample comes from a normal distribution," and its test statistic is denoted as... and corresponding value ;like ≥ ,in, If a pre-set significance level threshold is set, the normality assumption is accepted; otherwise, if there are signs of significant deviation from normality, the process proceeds to the outlier identification process. Calculate the mean of the feature distances. with standard deviation And define a global anomaly threshold. Where k=3, the feature distance sequence The samples were marked as anomalous.

8. A defense method against data poisoning attacks targeting offline reinforcement learning, as described in claim 7, characterized in that, The interpolation repair of the state-action pairs corresponding to the identified abnormal samples adopts an automatic correction method based on adjacent time-time interpolation, specifically as follows: The original state-action vector of the heterogeneous sample is The vector at the previous moment is Interpolation correction is performed using the following formula: Among them, the correction factor Use small positive numbers to reduce the impact of suspected mutations on the model. .

9. A computer device / equipment / system, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that: When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 8.

Images