A method for detecting real people during human-computer dialogue based on a large model

Abstract

This invention provides a method for detecting real users during human-computer dialogue based on a large model, comprising: Step 1, data collection: collecting raw user data and preprocessing it; Step 2, model construction; Step 3, model fine-tuning; Step 4, fine-tuning strategy: trying different strategies to obtain better results; Step 5, evaluation and iteration: continuously improving the model to meet actual needs based on evaluation and user feedback; Step 6, privacy and security considerations: all data is stored on the user's computer, and the large model platform does not store user data. This method can fully capture the context, semantics, and emotional information of the dialogue, thereby effectively distinguishing between real user responses and machine-generated responses. It has high adaptability and can automatically learn and respond to new fake dialogue patterns. This method has a wide range of applications, including online social platforms and customer service systems.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of emerging information technology (cloud computing, big data, AI, blockchain, Internet of Things / Internet of Vehicles, business platform, video communication, IPTV, new ICT, etc.), and particularly relates to a method for real person detection in human-computer dialogue based on a large model. BACKGROUND

[0002] With the development of ICT technology, people's social, business activities, etc. gradually expand, as the environment we live in becomes more and more complex, people's behavior shows diversity and complexity, and it is impossible to detect whether the person you are talking to is a real person or a robot in the process of human-computer dialogue by relying on the experience we already have. Therefore, a method is needed to identify real people in this situation.

[0003] Currently, the method for real person detection in human-computer dialogue process commonly used methods include natural language processing (NLP), knowledge base and rule engine, machine learning, recurrent neural network (RNN), long short-term memory network (LSTM), convolutional neural network (CNN), pre-training model, etc. These model algorithms have a series of problems such as small training sample size, low semantic understanding, poor model interpretability, and poor user experience. SUMMARY

[0004] The purpose of the present application is to solve the problems of the prior art, and the present application proposes a method for real person detection in human-computer dialogue based on a large model. This method combines the use of pre-training models to extract text features, and then uses models such as recurrent neural networks for sequence modeling. In addition, knowledge base and rule engine are also embedded in these methods to assist in determining authenticity.

[0005] The method of the present application comprises the following steps:

[0006] Step 1, data collection: collect the original data of the user personally and pre-process it;

[0007] Step 2, model construction;

[0008] Step 3, model fine-tuning;

[0009] Step 4, fine-tuning strategy: try different strategies, such as introducing diversity guiding questions, controlling the length of generation, etc. (quoted from a blog article by OpenAI titled "Improving Language Models with Human Feedback", published in November 2019. This article discusses a method for improving language models, including strategies such as introducing diversity guiding questions and controlling the length of generation. The article describes the use of human feedback to improve the performance of language models.) to obtain better results;

[0010] Step 5, evaluation and iteration: based on evaluation and user feedback, continuously improve the model to meet actual needs; through regular evaluation, understand the performance of the model in various tests in a timely manner, identify problems and bottlenecks. At the same time, actively collect user feedback to deeply understand user experience and find challenges that the model may encounter in actual application. Based on this information, adjust the fine-tuning strategy, parameter setting or data input method of the model. Such a cycle process can iteratively improve the model to better meet the actual needs of users.

[0011] In step 1, the original data of the user individual includes user voice, face picture, historical dialogue and action behavior data,

[0012] Further, in step 1, the action behavior data includes user interaction, movement, speech, shopping and game behavior data.

[0013] Further, in step 1, the preprocessing includes word segmentation and noise removal.

[0014] Further, in step 2, the model architecture adopts a hybrid cloud architecture mode, and on the basis of a large model platform based on a public cloud, uses the nlp semantic recognition and analysis technology of the large model platform to extract key information; the key information includes: question / request, entity, sentiment and emotion, key point, context, action item, etc., and the uploaded local data is not saved; the local client COPILOT stores a model library and a database, the model library stores various algorithms (such as text sentiment analysis, text generation mode detection, complexity and logic analysis, context consistency analysis, human common sense and knowledge detection, user behavior analysis, etc.), and the database stores the original data of the user individual, if the data cannot be obtained in the model library, the data is queried in the database.

[0015] Further, step 2 includes:

[0016] Step 2-1, multi-modal input through a multi-modal interface (such as an input interface supporting different types of data or information such as text, voice, picture, video, etc.), and the local client COPILOT is responsible for receiving information; the multi-modal input is the original data of the user individual.

[0017] Step 2-2, the local client COPILOT calls a third-party platform to convert unstructured data into structured data, and returns a multi-modal feature vector; the unstructured data includes user voice and face pictures;

[0018] Step 2-3, the local client COPILOT sends the structured data to the large model platform, which identifies the keywords based on semantic recognition technology and returns the recognition results;

[0019] Step 2-4, the local client COPILOT sends the recognition results back to the local model library, calls the corresponding algorithm for secondary recognition and calculation, and returns the results to the local client COPILOT;

[0020] Step 2-5, if the local model library returns empty data, call the local database to return sub-optimal data (such as similar historical words);

[0021] Step 2-6, the local client COPILOT returns the results to the multi-modal interface.

[0022] Further, in step 2-4, the corresponding algorithm includes:

[0023] Feature weighted average algorithm:

[0024]

[0025] where n is the number of application scenarios, f(x) represents the probability of real person detection, Xi represents the feature vector of the i-th modality, and Wi represents the multi-modal weight;

[0026] Multi-modal feature representation learning algorithm: combines features from different modalities (such as text, image, sound, etc.) to obtain more informative and more abstract feature representations.

[0027] Data preparation: based on the data collected in step 1, the user feature vector is pre-set to obtain the modality feature vector Xi, where i is the index of the modality, i takes values from 1 to n, and n is the number of modalities;

[0028] Feature fusion and connection: connect the feature vectors of different modalities to form a new feature vector [X1, X2,..., Xn], where Xn represents the feature vector of the n-th modality;

[0029] Multilayer Perceptron (MLP) model: use the weight matrix and bias vector of the multilayer perceptron model, the weight matrix and bias vector will be updated during the training process; Multilayer Perceptron (MLP) is a basic feedforward neural network structure, commonly used to solve various machine learning tasks, including classification, regression and feature extraction, etc. The weight matrix and bias vector of MLP are key parameters in the model, which will be optimized during the training process to minimize the loss function.

[0030] The first layer of hidden layers: the connected feature vector [x1, x2,... xn] is input to the first hidden layer; the activation function of each neuron uses ReLU, and the calculation formula is:

[0031]

[0032] where hj is the output of the jth hidden layer of the first hidden layer, Wji is the corresponding weight, and bj is the bias;

[0033] The second layer of hidden layers: the output of the first layer is taken as input, and the ReLU activation function is used for transformation:

[0034]

[0035] where hk is the output of the kth neuron of the second hidden layer, and Wkj is the corresponding weight;

[0036] Output layer: the output of the second layer of hidden layers is input to the output layer to obtain the learned multi-modal feature z:

[0037]

[0038] where Wk is the weight of the output layer, and p is the number of neurons of the second hidden layer;

[0039] Weighted fusion algorithm: according to the importance of each modality, dynamically adjust and fuse multiple modal features:

[0040]

[0041] where Pi represents the probability that the target person is determined to be a real person, ni represents the number of groups in which the real person is located, n represents the total number of groups, wi represents the weight of the multi-modal of the ith person, and f(Xi) represents the probability of real person detection of the ith person;

[0042] Threshold setting and real person determination: set a proper threshold, according to the accuracy rate, between 0.2 and 1, and determine whether it is a real person according to the calculation result; adjust the threshold based on the verification result of the training data (step 2-6) to balance the accuracy rate and the recall rate;

[0043] Commitment determination: in the process of human-computer interaction, according to the calculated real person detection probability, increase the real person commitment in the interaction process as one of the reference bases for real person judgment.

[0044] Further, step 3 includes: using the API of the large model platform, performing model fine-tuning by providing the dialogue history, user input and labeled expected output, the API call returns the generated text, and the text is compared with the labeled output to calculate the loss.

[0045] The method of the present application further comprises:

[0046] Step 6: Privacy and security considerations: all data is stored on the user's computer, and the large model platform side does not save user data.

[0047] The present application also provides a storage medium storing a computer program or instructions, which, when executed, implement the method for detecting real people in human-machine conversation based on a large model.

[0048] The present application can more accurately identify false information in human-machine conversation by training on large-scale conversation data. This method has a high level of contextual understanding, which can capture the semantic and emotional changes in the conversation, thereby effectively distinguishing between real people and machine-generated conversations. It not only has the ability to adapt to new false conversation patterns, but also can achieve real-time detection, which helps to prevent the spread of false information and adverse effects, and provides a powerful tool for online social platforms, customer service systems, etc.

[0049] The present application combines deep learning and natural language processing to identify possible false information in human-machine conversation. By training on large-scale conversation data, this method can fully capture the context, semantics and emotional information of the conversation, thereby effectively distinguishing between real users and machine-generated replies. It has a high degree of adaptability and can automatically learn and respond to new false conversation patterns. This method has a wide range of applications and can be used in online social platforms, customer service systems, etc. to identify and filter false information in real time, maintain information quality and user trust. By accurately determining the authenticity of the conversation, it helps to prevent the spread of false information, improves user experience, and builds a healthier and more reliable digital communication environment.

[0050] The present application has the following beneficial effects:

[0051] High accuracy: large pre-training language models have shown excellent performance in natural language processing tasks, so they can effectively identify text that does not conform to the pattern of real human conversation, improving the accuracy of detection.

[0052] Contextual understanding: large models can better understand the context of the conversation, allowing them to more accurately distinguish between machine-generated replies. It can capture semantic and emotional changes in the conversation, further improving the accuracy of detection.

[0053] Real-time: this method can detect real people in real-time conversations, helping to prevent machine-generated false information. This is of great significance to online social platforms, customer service systems, etc.

[0054] Adaptability: Large models can gradually adapt to new false dialogue patterns and strategies through continuous training and optimization, thereby continuously improving detection capabilities.

[0055] Multi-modal support: Large model-based methods can also integrate image, audio, and other multi-modal information to more comprehensively detect real people and enhance the perception of false dialogue.

[0056] Automation: This method can automatically detect in large amounts of dialogue data without human intervention, enabling efficient false dialogue recognition.

[0057] Customizability: Customized adjustments and optimizations can be made according to the needs of different application scenarios, improving the adaptability and effectiveness of the method in specific fields.

[0058] Countermeasures: By identifying false dialogue, appropriate measures can be taken to deal with machine automatic replies, reducing the spread of false information and negative effects.

[0059] Extensibility: It can be embedded as a plug-in into commonly used chat tools as a complementary function. BRIEF DESCRIPTION OF DRAWINGS

[0060] The above and / or other aspects of the present application will become more apparent by describing in detail the following embodiments with reference to the accompanying drawings, in which:

[0061] Figure 1 is the model architecture diagram established by the present application. DETAILED DESCRIPTION

[0062] The present application provides a method for real person detection in human-computer dialogue based on large models. The method is based on the multi-modal feature vectors and weights of people in human-computer interaction process, and judges whether the dialogue is real or not by calculating the probability. The core of this method is to design a recognition architecture and algorithm based on multi-modal behavior characteristics. Among them, feature extraction and feature selection are based on large models to extract multi-modal features of people from a large amount of data obtained from human-computer dialogue, and select the most distinctive feature vectors. Finally, the score of each human-computer dialogue object is calculated to determine the real person. Algorithm optimization is to optimize the algorithm formula for different behavior characteristics and data distribution, to improve the accuracy of recognition. This method can be applied to human behavior recognition in various human-computer dialogue environments, and has important application value.

[0063] The present application includes the following main steps:

[0064] Step 1, data collection: Collect the user's voice, face, history dialogue, action behavior, etc. data, including but not limited to user's interaction, movement, speech, shopping, game behavior data, and perform necessary preprocessing such as word segmentation, noise removal, etc.

[0065] Step 2, build the model, the approximate architecture is as shown in Figure 1

[0066] The model architecture adopts an innovative hybrid cloud architecture mode. On the basis of the large model platform of the public cloud, the nlp semantic recognition and analysis technology of the large model platform is mainly used to extract key information. The local data is not uploaded to the large model platform. The most important model library and database are stored locally. The personal algorithms are stored in the model library, and the personal original data is stored in the local database. If the data cannot be obtained in the local model library, the data query will be performed in the local database. This architecture has two advantages: 1) on the basis of fully utilizing the large model, the local user data privacy security protection is well realized 2) the ability of the large model is used to greatly improve the accuracy of real person detection.

[0067] The main process is as follows:

[0068] Step 2-1, multi-modal interface is used for multi-modal input, and the local client COPILOT is responsible for information reception; step 2-2, the local client COPILOT calls the third-party platform to convert the unstructured data such as voice and picture into structured data, and returns the multi-modal feature vector;

[0069] Step 2-3, the local client COPILOT sends the structured data to the large model, and the large model identifies the key words and other parameters based on the NLP, CNN and other semantic recognition technologies, and returns the recognition result.

[0070] Step 2-4, COPILOT sends the recognition result back to the local model library to call the corresponding model for secondary identification and calculation, and returns the result to COPILOT;

[0071] Step 2-5, if the local model library returns empty data, the local database is called to return suboptimal data.

[0072] Step 2-6, COPILOT returns the result to the multi-modal interface.

[0073] Step 3, model fine-tuning: use the API of the large model to fine-tune the model by providing the dialogue history, user input and labeled expected output. The API call will return the generated text, which will be compared with the labeled output to calculate the loss.

[0074] ​Step 4, Fine-tuning Strategy: Try different strategies such as introducing diversity guiding questions, controlling the length of generation, etc. to get better results.

[0075] Step 5, Evaluation and Iteration: Based on artificial evaluation and user feedback, continuously improve the model to meet actual needs;

[0076] Step 6, Privacy and Security Considerations: All data is stored on the user's computer, and the platform side does not save user data to avoid possible user data leakage.

[0077] Algorithm aspect: In order to improve the accuracy of judgment, a total of 5 algorithms are used.

[0078] Algorithm 1: Feature Weighted Average Algorithm

[0079] Formula: Where f(x) represents the probability of real person detection, Xi represents the multi-modal feature vector, and Wi represents the multi-modal weight.

[0080] Algorithm 2: Multi-modal Feature Representation Learning Algorithm: Combine features from different modalities (such as text, images, sound, etc.) to obtain more informative and abstract feature representations.

[0081] Data preparation: Represent the feature vector from each modality as Xi, where i is the index of the modality.

[0082] Feature fusion and connection: Connect the feature vectors of different modalities to form a new feature vector [x1, x2,..., xn], where n is the number of modalities.

[0083] Multi-layer Perceptron (MLP) Model: Define the weight matrix and bias vector of the multi-layer perceptron model, which will be updated during the training process.

[0084] First hidden layer: Input the connected feature vector [x1, x2,..., xn] into the first hidden layer. The activation function of each neuron uses ReLU, and the calculation formula is Where hj is the output of the jth hidden layer of the first hidden layer, Wji is the corresponding weight, and bj is the bias.

[0085] Second hidden layer: Take the output of the first layer as input and use the ReLU activation function for transformation: Where hk is the output of the kth neuron of the second hidden layer, Wkj is the corresponding weight, and bk is the bias.

[0086] Output layer: Input the output of the second hidden layer into the output layer to obtain the learned multi-modal feature representation z: where Wk is the weight of the output layer, and b is the bias.

[0087] Algorithm 3: Weighted fusion algorithm: dynamically adjust according to the importance of each modality, and fuse multiple modal features. This process is based on the following formula:

[0088]

[0089] where Pi represents the probability that the person is determined to belong to a real person, mi represents the number of groups that the real person belongs to, m represents the total number of groups, wj represents the weight of the multi-modal of the jth person, and f(Xj) represents the probability of real person detection of the jth person.

[0090] Algorithm 4: Threshold setting and real person determination: set appropriate thresholds, and determine whether it is a real person according to the comprehensive score. Adjust the threshold based on the verification results of the training data to balance the accuracy and recall rate.

[0091] Algorithm 5: Commitment determination: in the process of human-computer interaction, according to the real person probability calculated by the algorithm, increase the real person commitment in the interaction process as one of the reference basis for real person determination.

[0092] Embodiment

[0093] The embodiment provides a method for real person detection in human-computer dialogue based on a large model, which includes the following steps:

[0094] Step 1, data collection: collect the original data of the user personally, and perform pretreatment;

[0095] Step 2, model construction;

[0096] Step 3, model fine-tuning;

[0097] Step 4, fine-tuning strategy: try different strategies, such as introducing diversity guiding questions, controlling the length of generation, etc. (Reference from an article by OpenAI titled "Improving Language Models with Human Feedback", published in November 2019. This article discusses a method for improving language models, including strategies such as introducing diversity guiding questions and controlling the length of generation. The article introduces the use of human feedback to improve the performance of language models.), to obtain better results;

[0098] Step 5, evaluation and iteration: based on evaluation and user feedback, continuously improve the model to meet actual needs; through regular evaluation, understand the performance of the model in various tests in a timely manner, identify problems and bottlenecks. At the same time, actively collect user feedback to better understand user experience and identify challenges the model may encounter in actual application. Based on this information, adjust the fine-tuning strategy, parameter settings or data input method of the model. Such a cycle process can iteratively improve the model to better meet the actual needs of users;

[0099] Step 6, privacy and security considerations: all data is stored on the user's computer, and the large model platform side does not save user data.

[0100] In step 1, the user's personal raw data includes user voice, face pictures, historical conversations and action behavior data, and the action behavior data includes user interaction, movement, speech, shopping and game behavior data.

[0101] In step 1, the preprocessing includes word segmentation and noise removal.

[0102] In step 2, the model architecture adopts a hybrid cloud architecture mode, and on the basis of a public cloud large model platform, uses the nlp semantic recognition and parsing technology of the large model platform to extract key information; the key information includes: question / request, entity, sentiment and emotion, key point of view, context, action item, etc., and the uploaded local data is not saved; the local client COPILOT stores model library and database, the model library stores various algorithms (such as text sentiment analysis, text generation mode detection, complexity and logic analysis, context consistency analysis, human common sense and knowledge detection, user behavior analysis, etc.), and the database stores user's personal raw data. If the data cannot be obtained in the model library, the data will be queried in the database.

[0103] Step 2 includes:

[0104] Step 2-1, multi-modal input through a multi-modal interface (such as an input interface that supports different types of data or information such as text, voice, pictures, and videos); the local client COPILOT is responsible for receiving information; the multi-modal input is the user's personal raw data.

[0105] Step 2-2, the local client COPILOT calls a third-party platform to convert unstructured data into structured data and returns a multi-modal feature vector; the unstructured data includes user voice and face pictures;

[0106] Step 2-3, the local client COPILOT sends the structured data to the large model platform, which identifies the keywords based on semantic recognition technology and returns the recognition results;

[0107] Step 2-4, the local client COPILOT sends the recognition results back to the local model library, calls the corresponding algorithm for secondary recognition and calculation, and returns the results to the local client COPILOT;

[0108] Step 2-5, if the local model library returns empty data, call the local database to return sub-optimal data (such as similar historical words);

[0109] Step 2-6, the local client COPILOT returns the results to the multi-modal interface.

[0110] In step 2-4, the corresponding algorithm includes:

[0111] Feature weighted average algorithm:

[0112]

[0113] Where n is the number of application scenarios, f(x) represents the probability of real person detection, Xi represents the feature vector of the i-th modality, and Wi represents the multi-modal weight;

[0114] Multi-modal feature representation learning algorithm: combines features from different modalities (such as text, image, sound, etc.) to obtain more informative and abstract feature representations.

[0115] Data preparation: based on the data collected in step 1, the user feature vector is pre-set, such as the data in Table 1 below, to obtain the modality feature vector Xi, where i is the index of the modality, i takes values from 1 to n, and n is the number of modalities; Table 1 defines the feature vector under different behavior types in different modalities under different human-computer interaction scenarios, where the behavior types include social activities, work meetings, game entertainment, online learning, virtual shopping, sports exercise, digital creation, chat socialization, conference presentation, and virtual tourism.

[0116] Table 1

[0117]

[0118]

[0119] Feature fusion and connection: connect the feature vectors of different modalities to form a new feature vector [X1, X2,..., Xn], where Xn represents the feature vector of the n-th modality;

[0120] Multilayer Perceptron (MLP) model: use the weight matrix and bias vector of the multilayer perceptron model, which will be updated during the training process; Multilayer Perceptron (MLP) is a basic feedforward neural network structure, which is commonly used to solve various machine learning tasks, including classification, regression and feature extraction, etc. The weight matrix and bias vector of MLP are key parameters in the model, which will be optimized during the training process to minimize the loss function.

[0121] First hidden layer: input the connected feature vector [x1, x2,... xn] to the first hidden layer; the activation function of each neuron uses ReLU, and the calculation formula is:

[0122]

[0123] Where hj is the output of the jth hidden layer of the first hidden layer, Wji is the corresponding weight, and bj is the bias;

[0124] Second hidden layer: take the output of the first layer as input, and use ReLU activation function for transformation:

[0125]

[0126] Where hk is the output of the kth neuron of the second hidden layer, and Wkj is the corresponding weight;

[0127] Output layer: input the output of the second hidden layer to the output layer to get the learned multi-modal feature z:

[0128]

[0129] Where Wk is the weight of the output layer, and p is the number of neurons in the second hidden layer;

[0130] Weighted fusion algorithm: dynamically adjust according to the importance of each modality, and fuse multiple modal features:

[0131]

[0132] Where Pi represents the probability that the target person is determined to be a real person, ni represents the number of groups that the real person belongs to, n represents the total number of groups, wi represents the weight of the multi-modal of the ith person, and f(Xi) represents the probability of real person detection of the ith person;

[0133] Threshold setting and real person determination: set a proper threshold, according to the accuracy rate, between 0.2 and 1, and determine whether it is a real person according to the calculation result; adjust the threshold based on the verification result of the training data (step 2-6) to balance the accuracy and recall rate;

[0134] Commitment determination: in the process of human-computer interaction, according to the calculated probability of real person detection, increase the commitment of real person in the interaction process as one of the reference basis of real person judgment.

[0135] Step 3 includes: using the API of the large model platform, model fine-tuning is performed by providing dialogue history, user input and labeled expected output, the API call returns the generated text, and the text is compared with the labeled output to calculate the loss.

[0136] In order to verify the effectiveness of the present application, a sample set consisting of 20 individuals whose behavior data were recorded was used for testing. The weights of text, speech, picture / video and other actions are defined in Table 2 below.

[0137] Table 2

[0138]

[0139] Next, according to the above formula, and the defined feature vector and weight, the real person probability is calculated by the large model, and the calculation result is shown in Table 3 below.

[0140] Table 3

[0141]

[0142] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and the computer program can run the invention content of a method for real person detection in human-computer dialogue process based on a large model and part or all steps in each embodiment when executed by the data processing unit. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0143] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present application can be realized by means of a computer program and its corresponding general hardware platform. Based on such understanding, the technical solutions in the embodiments of the present application can be embodied in the form of a computer program, i.e. a software product, which can be stored in a storage medium, including a plurality of instructions for causing a device (which can be a personal computer, a server, a single-chip microcomputer, a MUU or a network device, etc.) containing a data processing unit to execute the method described in each embodiment or some parts of the embodiments of the present application.

[0144] The application provides a method for real person detection in a man-machine conversation process based on a large model. The method and approach for specifically implementing the technical solution are various, and the above description is only the preferred embodiment of the application. It should be pointed out that, for ordinary skilled persons in the technical field, several improvements and refinements can be made without departing from the principle of the application, and these improvements and refinements should also be regarded as the protection scope of the application. The components not explicitly described in the embodiment can be implemented by using the prior art.

Claims

1. A method for real person detection in a human-machine dialogue process based on a large model, characterized in that, The method comprises the following steps: Step 1, data collection: collect the user's personal original data and perform preprocessing; Step 2, model construction; Step 3, model fine-tuning; Step 4, fine-tuning strategy: try different strategies to get better results; Step 5, evaluation and iteration: continuously improve the model according to the evaluation and user feedback to meet the actual needs; In step 1, the user's personal original data includes user voice, face picture, historical dialogue and action behavior data; In step 1, the action behavior data includes user interaction, movement, speech, shopping and game behavior data; In step 1, the preprocessing includes word segmentation and noise removal; In step 2, the model architecture adopts a hybrid cloud architecture mode, and the nlp semantic recognition and analysis technology of the large model platform based on the public cloud is used to extract key information; the uploaded local data is not saved; the local client COPILOT stores the model library and the database, the model library stores various algorithms, and the database stores the user's personal original data; if the data cannot be obtained in the model library, the database is queried for data; Step 2 comprises: Step 2-1, multi-modal input through a multi-modal interface, and the local client COPILOT is responsible for receiving information; the multi-modal input is the user's personal original data; Step 2-2, the local client COPILOT calls a third-party platform to convert unstructured data into structured data, and returns a multi-modal feature vector; the unstructured data includes user voice and face pictures; Step 2-3, the local client COPILOT sends the structured data to the large model platform, the large model platform performs keyword recognition based on semantic recognition technology, and returns the recognition result; Step 2-4, the local client COPILOT sends the recognition result back to the local model library, calls the corresponding algorithm for secondary recognition and calculation, and returns the result to the local client COPILOT; Step 2-5, if the local model library returns empty data, the local database is called to return suboptimal data; Step 2-6, the local client COPILOT returns the result to the multi-modal interface; In step 2-4, the corresponding algorithm comprises: Feature weighted average algorithm: Wherein, n refers to the number of application scenarios, f(x) represents the probability of real person detection, Xi represents the feature vector of the i-th modality, and Wi represents the multi-modal weight; Multi-modal feature representation learning algorithm: combine features from different modalities to obtain more informative and abstract feature representations; Data preparation: based on the data collected in step 1, the user feature vector is pre-set to obtain the modality feature vector Xi, wherein i is the index of the modality, i takes a value from 1 to n, and n is the number of modalities; Feature fusion and connection: connect the feature vectors of different modalities to form a new feature vector [X1, X2,..., Xn], wherein Xn represents the feature vector of the n-th modality; Multi-layer perception model: the weight matrix and bias vector of the multi-layer perception model are used, and the weight matrix and bias vector will be updated during the training process; First hidden layer: the connected feature vector [x1, x2,... xn] is input into the first hidden layer; the activation function of each neuron uses ReLU, and the calculation formula is: Where hj is the output of the jth hidden layer of the first hidden layer, Wji is the corresponding weight, and bj is the bias; Second hidden layer: the output of the first layer is taken as input, and the ReLU activation function is also used for transformation: Where hk is the output of the kth neuron of the second hidden layer, and Wkj is the corresponding weight; Output layer: input the output of the second hidden layer to the output layer to obtain the learned multi-modal feature z: Where Wk is the weight of the output layer, and p is the number of neurons in the second hidden layer; Weighted fusion algorithm: according to the importance of each modality, the multiple modal features are fused dynamically: Where Pi represents the probability of the target person being judged as a real person, ni represents the number of groups in which the real person is located, n represents the total number of groups, wi represents the weight of the multi-modal of the ith person, and f(Xi) represents the probability of real person detection of the ith person; Threshold setting and real person judgment: set a proper threshold, and judge whether it is a real person according to the calculation result; adjust the threshold based on the verification result of the training data to balance the accuracy and recall rate; Promise judgment: in the process of human-computer interaction, according to the calculated probability of real person detection, increase the real person promise in the interaction process as a reference basis for real person judgment; Step 3 includes: using the API of the large model platform, model fine-tuning is performed by providing the dialogue history, user input and labeled expected output, and the API call returns the generated text, which is compared with the labeled output to calculate the loss; Also includes: Step 6, privacy and security considerations: all data are stored on the user's computer side, and the user data are not saved on the large model platform side.

2. A storage medium, characterized by A computer program or instructions are stored, which, when executed, implement the method of claim 1. A computer program or instructions are stored, which, when executed, implement the method of claim 1.

Images