A system that dynamically calls on artificial intelligence memory and emotions to interact with large language models
By dynamically calling artificial intelligence memory and emotion systems, combined with heterogeneous graph neural network and facial expression recognition, the problem of insufficient emotion capture in the existing technology is solved, and accurate monitoring of user and administrator emotions and secure management of knowledge data is achieved. , ensuring data security.
Patent Information
- Application Number
- CN202411015349.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2044-07-26
AI Technical Summary
The existing emotion capture technology is mainly concentrated in large language models, used for pure language interaction, failing to effectively judge the emotional needs of users and administrators, and affecting the management and security of knowledge data.
A system that dynamically calls artificial intelligence memory and emotions is designed. Through processors, remote servers, camera modules and voice modules, combined with heterogeneous graph neural networks and facial expression recognition, it captures the emotional state of users and administrators and realizes emotional monitoring. and the security management of knowledge data.
It realizes accurate monitoring of users and administrators' emotions, promptly detects emotional abnormalities, ensures the security of knowledge data, avoids unexpected security issues, and improves the management efficiency of knowledge data.
Smart Images
Figure CN118839004B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an interactive knowledge management system, and in particular to a system for dynamically calling artificial intelligence memory and emotion to interact with a large language model, and belongs to the field of intelligent management. Background Art
[0002] Knowledge management generally analyzes the knowledge fields that users may be interested in based on the user's historical input information, and then recommends and statistically analyzes the knowledge data in the relevant knowledge fields. Administrators may need to manage each piece of knowledge data. Therefore, the analysis of the knowledge fields of the administrator's historical input is of no practical significance. The administrator's daily psychological and emotional state may be more important for data management. Therefore, capturing the administrator's emotions may be more important. Existing emotion capture is concentrated in large language models such as CHAT GPT, which are used for pure language interaction. It is not used to judge the user's emotional needs. Therefore, first of all, whether user identification can replace historical input for knowledge data interaction, and secondly, how to pay attention to the administrator's own emotions to ensure the security of knowledge data. These are technical problems that need to be solved urgently. Summary of the invention
[0003] In view of the above problems, the present invention proposes a system for dynamically calling artificial intelligence memory and emotion to interact with a large language model, including a processor, a remote server for processing the emotional state of an administrator; a first display module for users installed with a first camera module and a second display module for administrators installed with a second camera module, which are respectively used to capture the user's face and the administrator's face; and a first voice module used by the user and a second voice module used by the administrator.
[0004] The processor is capable of running the following program modules: a knowledge acquisition module, a knowledge parsing module, an input module, and an evaluation module, wherein:
[0005] The knowledge acquisition module is used to acquire knowledge from the knowledge generation source and record the spatial location and generation time of the knowledge generation. The knowledge includes knowledge related to the equipment related to the knowledge, specifically including the name of the equipment, the knowledge generation source and / or user to which it belongs, the use and technical principles related to the equipment, as well as scientific and technological resources, social and humanities resources, specifically including documents, pictures, videos, and audio. The knowledge also includes the name of the equipment, the knowledge generation source and / or user to which it belongs, the use and technical principles related to the equipment, usage records, historical and real-time indicators and environmental parameters, and equipment management information. The data related to the knowledge forms knowledge data.
[0006] It is easy to understand that the purpose includes the knowledge field and equipment functions used by the equipment, the usage records include users, usage periods, historical and real-time indicators and environmental parameters include historical and real-time technical parameters of equipment operation, and physical parameters of the equipment operating environment, equipment management information, including information on equipment purchase, operation and maintenance, update, borrowing, and relocation.
[0007] The knowledge parsing module is connected to the knowledge acquisition module, and is used to construct a visualized heterogeneous graph neural network (HGNN), and use the knowledge and spatial location input by the received knowledge generation source to construct an attention mechanism (AM) based on the heterogeneous graph neural network. After the construction is completed, the knowledge context is retrieved again based on the new input (in most cases, the new input is a search operation, that is, the relevant knowledge input is searched for any relevant knowledge information of interest, or there are other purposes of use, learning, and organization), and the analysis result of the attention mechanism is given based on the knowledge context, and the historical analysis results corresponding to the relevant faces in the block heterogeneous graph neural network are presented to the user. At the same time, the data management password of each block is set based on the block graph neural network, and the HGNN and AM are updated within a predetermined time period, and the blocks and the data management password are continuously updated accordingly.
[0008] The input module is connected to the computer network and is used to receive the newly input knowledge of the user, and input it into the knowledge parsing module through the computer network, and the knowledge is clustered in a preset target domain with a domain label, and when the user newly inputs knowledge, it is necessary to select a target domain for the newly input knowledge and select the target domain label.
[0009] The evaluation module is used to conduct a comprehensive analysis of the retrieved knowledge context and the analysis results of the attention mechanism, and give a final evaluation that is suitable for the real meaning of the user's search, specifically including comparing the predicted knowledge data classification and the domain label, and taking the comparison result as the final evaluation, wherein:
[0010] The first display module is also used to display the analysis results of the knowledge context, attention mechanism and comprehensive analysis results and operate the visualized HGNN, wherein:
[0011] The knowledge generation source periodically inputs the knowledge generated according to the confidentiality classification into the knowledge acquisition module, and transmits it to the knowledge analysis module for the update.
[0012] Optionally, the knowledge generation sources include: administrative departments, enterprises, institutions, companies, school research institutions, social groups and organizations, and individuals.
[0013] Optionally, the knowledge acquisition module parses the IP address or obtains the spatial location based on the name and address of the input knowledge generation source. The generation time is obtained by recording the time of receiving the knowledge through the knowledge acquisition module. The knowledge is reviewed manually or by computer to decide whether to adopt the knowledge and record the spatial location and generation time. The operation includes dragging the visualized HGNN, clicking on it to pop up a dialog box or search bar to search and / or view, modify, delete and / or restore the knowledge.
[0014] Among them, the knowledge analysis module is also used to use the first camera to collect the user's face, identify other faces related to the user of the collected face through the first face recognition model, and provide the new input of the history corresponding to other faces to the user of the collected face for selection, so as to retrieve the knowledge context and give the analysis result of the attention mechanism based on the knowledge context; and the knowledge analysis module is also used to use the administrator's face to identify whether the administrator's facial expression is abnormal through the second face recognition model in the remote server, so that the remote server makes a decision whether to ask the administrator online whether there is any abnormality.
[0015] Optionally, the first voice module is used to communicate with the user, thereby determining the new input based on the user's response, and the second voice module is used to communicate with the administrator, so that the knowledge analysis module comprehensively determines whether the administrator's facial expression is abnormal based on the administrator's response and the facial expression.
[0016] Optionally, the relevant faces in the block-based heterogeneous graph neural network include the analysis results obtained from the new input of any user's history to determine the block, and the face of any user is collected as the relevant face.
[0017] Optionally, the first face recognition model includes mapping corresponding profile information according to the collected face, so as to associate other related faces according to the profile information. The profile information includes but is not limited to identity information, occupation information, household registration information, marriage information, real estate information, financial activity information, and the profile information and / or interpersonal relationship information of other persons voluntarily customized by the user and agreed by other persons (such as whether they are friends, teachers and students, mentors and apprentices, superiors and subordinates, colleagues, classmates, etc.).
[0018] The second face recognition model includes a facial feature extraction module, an expression extraction module, an emotion judgment module, and an optimization module, wherein the facial feature extraction module performs finite element division on the received administrator's face, and extracts the forehead, face, eyes, lips, eyebrows, and chin regions, and completes a plurality of preset feature points of the regions; the expression extraction module uses a convolutional neural network or a generative adversarial network to perform classification training on the input plurality of preset feature points of the regions according to expression labels; the emotion judgment module calls historical classifications according to the identified classifications to determine whether the administrator's facial expressions are abnormal; the optimization module regularly collects facial expressions of users and administrators, and refines emotion classifications (such as impatience and anxiety), and retrains the convolutional neural network or the generative adversarial network to obtain an optimized network.
[0019] Optionally, the reply situation includes the delay time of receiving the administrator's voice signal, the speaking speed, the pitch, and the time interval between words. The comprehensive judgment of whether the administrator's facial expression is abnormal is made based on the administrator's reply situation and the facial expression. Specifically, a support vector machine is established to classify the emotions of the reply situation, and the emotion judgment module compares the emotions according to the classification identified. If the two emotions are within the matching threshold range (for example, the anxiety level one to two levels are within the matching threshold range, and if the two emotions are anxiety level one and anxiety level three, which is a higher degree, respectively, then they do not match), the administrator's current emotion is determined. If there is no match, the second voice module initiates a program to inquire about the administrator's mood, and the emotions are classified again according to the reply situation, and the emotions are compared again with the emotion judgment module according to the classification identified. If there is a match, the current administrator's emotion is determined. If there is still no match, the remote server is notified for manual inquiry.
[0020] The threshold range is designed to cover the fluctuation of human emotions. For example, when an administrator is actually impatient, the machine's questioning may make him even more impatient or increase his level of impatience.
[0021] It is easy to understand that the emotional judgment based on the combination of the administrator's face and voice can more accurately reflect the dual representation of the administrator's inner activities in expression and speech.
[0022] Among them, constructing a heterogeneous graph neural network and using the knowledge and spatial locations input by the received knowledge generation source to construct an attention mechanism based on the heterogeneous graph neural network specifically includes:
[0023] S1 establishes a geographical map, marks the spatial locations on the geographical map as nodes of the heterogeneous graph neural network, and constructs a heterogeneous graph network based on real spatial locations;
[0024] S2 constructs at least one target heterogeneous graph network on nodes based on the heterogeneous graph neural network;
[0025] S3 constructs multi-type meta-paths in a heterogeneous graph network based on real space locations and at least one target heterogeneous graph network, and establishes an attention mechanism based on the multi-type meta-paths, first constructing a node attention mechanism, then constructing a semantic attention mechanism, and finally constructing a subgraph attention mechanism.
[0026] The method of retrieving the knowledge context based on the newly input knowledge and providing the analysis result of the attention mechanism based on the knowledge context is as follows:
[0027] S5 inputs the knowledge into the trained AM to obtain the predicted target classification.
[0028] S6 draws the knowledge context according to the corresponding spatial location and generation time of the knowledge in the predicted target classification, and retrieves the drawn knowledge context.
[0029] S7: marking nodes and / or subgraphs corresponding to the new input knowledge in the drawn knowledge context.
[0030] It should be understood that the target can be various types of knowledge data, including knowledge itself, equipment, and knowledge operation behaviors in the system, such as viewing, searching, downloading, uploading, voice, etc. These knowledge data and operation behaviors are targets, and target heterogeneous networks are constructed on the nodes.
[0031] Optionally, the comprehensive analysis includes statistically analyzing at least one of the following types of distributions: a first distribution on the geographical map in the predicted knowledge field corresponding to the knowledge and the at least one other field,
[0032] a second distribution in at least one target heterogeneous graph network for visualization,
[0033] a heterogeneous graph network of real-space locations, a mixed distribution between at least one target heterogeneous graph network, and,
[0034] According to the knowledge generation source to which the user belongs, at least one of the first distribution, the second distribution and the third distribution corresponding to the knowledge field involved in the knowledge generation source on the geographical map is displayed. Beneficial Effects
[0035] By interacting with users and administrators through facial and voice respectively, it is possible to quickly associate the faces of other users in the same field or with the same preferences, and efficiently and accurately provide the knowledge data that users are concerned about. It also realizes the emotional monitoring of administrators and can promptly detect abnormal emotions of managers, thereby making the managed knowledge data more able to avoid unexpected security issues. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 A schematic diagram of a system configuration for dynamically invoking artificial intelligence memory and emotion to interact with a large language model according to Embodiment 1 of the present invention.
[0037] Figure 2 A flow chart of knowledge data analysis of the voice interaction between the system and the user in Embodiment 1 of the present invention,
[0038] Figure 3 A flowchart of efficient and accurate analysis of user's face recognition knowledge data according to Embodiment 2 of the present invention,
[0039] Figure 4 Flow chart of comprehensive judgment of face recognition and voice interaction emotions for management in Example 2 of the present invention,
[0040] Figure 5 A specific flow chart for comprehensively judging administrator emotions. DETAILED DESCRIPTION Example 1
[0041] This embodiment describes the system configuration and its constituent functions for dynamically calling artificial intelligence memory and emotion to interact with a large language model. Figure 1 As shown, a processor is included, a remote server for processing the emotional state of an administrator; a first display module for users with a first camera module installed and a second display module for administrators with a second camera module installed, which are respectively used to capture the user's face and the administrator's face; and a first voice module used by the user, and a second voice module used by the administrator.
[0042] The processor is capable of running the following program modules: a knowledge acquisition module, a knowledge parsing module, an input module, and an evaluation module, wherein:
[0043] The knowledge acquisition module is used to acquire knowledge from the knowledge generation source and record the spatial location and generation time of the knowledge generation. The knowledge includes knowledge related to the equipment related to the knowledge, specifically including the name of the equipment, the knowledge generation source and / or user, the use related to the equipment, technical principles, and scientific and technological resources, social and humanities resources, specifically including literature, pictures, videos, and audio. The knowledge also includes the name of the equipment, the knowledge generation source and user, the use related to the equipment, technical principles, usage records, historical and real-time indicators and environmental parameters, and equipment management information. The data related to the knowledge forms knowledge data.
[0044] The knowledge parsing module is connected to the knowledge acquisition module to construct a visualized HGNN, and uses the knowledge and spatial locations input by the received knowledge generation source to construct an attention mechanism AM based on a heterogeneous graph neural network. After the construction is completed,
[0045] like Figure 2 As shown, the first voice module talks with the user, collects new input information based on the user's reply (voice reply), retrieves the knowledge context based on the new input knowledge, and gives the analysis result of the attention mechanism based on the knowledge context. The knowledge parsing module presents the historical analysis results corresponding to the relevant faces in the block-based heterogeneous graph neural network to the user.
[0046] For the relevant faces, the block is determined based on the analysis result obtained from the new input of any user's history, and the face of any user is collected.
[0047] At the same time, the knowledge parsing module sets the data management password for each block based on the block-based graph neural network, and updates the HGNN and AM within a predetermined time period, and continuously updates the block and the data management password accordingly.
[0048] The input module is connected to the computer network and is used to receive the newly input knowledge of the user, and input it into the knowledge parsing module through the computer network, and the knowledge is clustered in a preset target domain with a domain label, and when the user newly inputs knowledge, it is necessary to select a target domain for the newly input knowledge and select the target domain label.
[0049] The evaluation module is used to conduct a comprehensive analysis of the retrieved knowledge context and the analysis results of the attention mechanism, and give a final evaluation that is suitable for the real meaning of the user's search, specifically including comparing the predicted knowledge data classification and the domain label, and taking the comparison result as the final evaluation, wherein:
[0050] The first display module is also used to display the analysis results of the knowledge context, attention mechanism and comprehensive analysis results and operate the visualized HGNN, wherein:
[0051] The knowledge generation source periodically inputs the knowledge generated according to the confidentiality classification into the knowledge acquisition module, and transmits it to the knowledge analysis module for the update.
[0052] Among them, constructing a heterogeneous graph neural network and using the knowledge and spatial locations input by the received knowledge generation source to construct an attention mechanism based on the heterogeneous graph neural network specifically includes:
[0053] S1 establishes a geographical map, marks the spatial locations on the geographical map as nodes of the heterogeneous graph neural network, and constructs a heterogeneous graph network based on real spatial locations;
[0054] S2 constructs at least one target heterogeneous graph network on nodes based on the heterogeneous graph neural network;
[0055] S3 constructs multi-type meta-paths in a heterogeneous graph network based on real space locations and at least one target heterogeneous graph network, and establishes an attention mechanism based on the multi-type meta-paths, first constructing a node attention mechanism, then constructing a semantic attention mechanism, and finally constructing a subgraph attention mechanism.
[0056] The method of retrieving the knowledge context based on the newly input knowledge and providing the analysis result of the attention mechanism based on the knowledge context is as follows:
[0057] S5 inputs the knowledge into the trained AM to obtain the predicted target classification.
[0058] S6 draws the knowledge context according to the corresponding spatial location and generation time of the knowledge in the predicted target classification, and retrieves the drawn knowledge context.
[0059] S7 marks the nodes and subgraphs corresponding to the new input knowledge in the drawn knowledge context.
[0060] The comprehensive analysis includes statistically analyzing at least one of the following types of distributions: a first distribution on the geographic map, that is, a spatial distribution of users on the geographic map, in the predicted knowledge field corresponding to the knowledge and the at least one other field;
[0061] A second distribution in the visualized at least one target heterogeneous graph network, namely, a distribution of targets,
[0062] a mixed distribution between a heterogeneous graph network of real-world locations, a heterogeneous graph network of at least one target (i.e., a visualized distribution of interactions between users and targets), and,
[0063] According to the knowledge generation source to which the user belongs, at least one of the first distribution, the second distribution and the third distribution corresponding to the knowledge field involved in the knowledge generation source on the geographical map is displayed.
[0064] Example 2
[0065] This embodiment will explain face recognition and speech emotion recognition, and further illustrate the functions of the knowledge analysis module for users and administrators, in addition to the language interaction with users in embodiment 1, language interaction with administrators and emotion recognition.
[0066] like Figure 3As shown, for the user, the first camera in Example 1 collects the user's face, identifies other faces related to the user of the collected face through the first face recognition model, and provides the new input of the history corresponding to the other faces to the user of the collected face for selection, thereby retrieving the knowledge context and giving the analysis result of the attention mechanism based on the knowledge context.
[0067] The first face recognition model maps the collected face to the corresponding file information, thereby associating it with other related faces based on the file information.
[0068] like Figure 4 As shown, for the administrator, the second camera module of Example 1 captures the administrator's face, and recognizes the administrator's facial expression (such as Figure 4 The remote server can then determine whether the administrator is abnormal (the upper left is impatient, the upper right is happy, the lower left is normal, and the lower right is anxious) and whether the administrator is abnormal. This is one implementation method.
[0069] In a second implementation mode, the second voice module is used to communicate with the administrator, so that the knowledge analysis module comprehensively judges whether the administrator's facial expression is abnormal based on the administrator's reply and the facial expression. The reply includes the delay time of receiving the administrator's voice signal, the speech speed, the pitch, and the time interval between words. The comprehensive judgment of whether the administrator's facial expression is abnormal based on the administrator's reply and the facial expression specifically includes: Figure 5 As shown, a support vector machine is established to classify the emotions of the reply and compare them with the emotion judgment module based on the identified classification. If the two emotions are within the matching threshold range, the current emotion of the administrator is determined. If they do not match, the second voice module initiates a program to inquire about the administrator's mood, and classifies the emotions again based on the reply, and compares them with the emotion judgment module based on the identified classification. If they match, the current administrator's emotion is determined. If they still do not match, the remote server is notified for manual inquiry.
[0070] This enables the user to quickly and conveniently input the new input, and provides other user history new input records associated with the user, so that the user can quickly select the input. The administrator's emotions are analyzed through voice interaction and face collection of the administrator, so as to detect abnormalities in time and ensure the operation of the knowledge data's security strategy to prevent accidents.
Claims
1. A system that dynamically calls artificial intelligence memory and emotions to interact with a large language model, characterized by: A remote server including a processor for processing the emotional state of an administrator; a first display module for a user equipped with a first camera module and a second display module for an administrator equipped with a second camera module, for capturing the face of the user and the face of the administrator respectively; and a first voice module for the user and a second voice module for the administrator; The processor is capable of running the following program modules: a knowledge acquisition module, a knowledge parsing module, an input module, and an evaluation module, wherein: The knowledge acquisition module is used to acquire knowledge from a knowledge generation source and record the spatial location and generation time of the knowledge generation. The knowledge includes knowledge related to the equipment related to the knowledge, the knowledge generation source and / or user to which the knowledge belongs, the purpose and technical principle related to the equipment, as well as scientific and technological resources and social humanities resources. The knowledge also includes the name of the equipment, usage records, historical and real-time indicators and environmental parameters, equipment management information, and social humanities resources, specifically including documents, pictures, videos, and audios. The data related to the knowledge forms knowledge data; The knowledge parsing module is connected to the knowledge acquisition module, and is used to construct a visualized heterogeneous graph neural network HGNN, and use the knowledge and spatial locations input by the received knowledge generation source to construct an attention mechanism AM based on the heterogeneous graph neural network. After the construction is completed, the knowledge context is retrieved again based on the newly input knowledge, and the analysis results of the attention mechanism are given based on the knowledge context. The historical analysis results corresponding to the relevant faces in the block-based heterogeneous graph neural network are presented to the user. At the same time, the data management password of each block is set based on the block-based graph neural network, and the HGNN and AM are updated within a predetermined time period, and the blocks and the data management password are continuously updated accordingly. The input module is connected to the computer network and is used to receive the newly input knowledge of the user, and input it into the knowledge parsing module through the computer network, and the knowledge is clustered in a preset target domain with a domain label, and when the user newly inputs knowledge, it is necessary to select a target domain for the newly input knowledge and select the target domain label. The evaluation module is used to conduct a comprehensive analysis of the retrieved knowledge context and the analysis results of the attention mechanism, and give a final evaluation that is suitable for the real meaning of the user's search, specifically including comparing the predicted knowledge data classification and the domain label, and taking the comparison result as the final evaluation, wherein: The first display module is also used to display the analysis results of the knowledge context, attention mechanism and comprehensive analysis results and operate the visualized HGNN, wherein: The knowledge generation source periodically inputs the knowledge generated according to the confidentiality classification into the knowledge acquisition module, and transmits the knowledge analysis module for the update; Among them, constructing a heterogeneous graph neural network and using the knowledge and spatial locations input by the received knowledge generation source to construct an attention mechanism based on the heterogeneous graph neural network specifically includes: S1 establishes a geographical map, marks the spatial locations on the geographical map as nodes of the heterogeneous graph neural network, and constructs a heterogeneous graph network based on real spatial locations; S2 constructs at least one target heterogeneous graph network on nodes based on the heterogeneous graph neural network; S3 constructs a multi-type meta-path in a heterogeneous graph network based on a real space location and at least one target heterogeneous graph network, and establishes an attention mechanism based on the multi-type meta-path, first constructing a node attention mechanism, then constructing a semantic attention mechanism, and finally constructing a subgraph attention mechanism; The method of retrieving the knowledge context based on the newly input knowledge and providing the analysis result of the attention mechanism based on the knowledge context is as follows: S5 inputs the knowledge into the trained AM to obtain the predicted target classification. S6 draws the knowledge context according to the corresponding spatial location and generation time of the knowledge in the predicted target classification, and retrieves the drawn knowledge context. S7 marks the nodes and / or subgraphs corresponding to the new input knowledge in the drawn knowledge context; The comprehensive analysis includes statistically analyzing at least one of the following types of distributions: a first distribution on the geographic map in the predicted knowledge field and at least one other field corresponding to the knowledge, a second distribution in at least one target heterogeneous graph network for visualization, a heterogeneous graph network of real-space locations, a mixed distribution between at least one target heterogeneous graph network, and, According to the knowledge generation source to which the user belongs, at least one of the first distribution, the second distribution and the third distribution corresponding to the knowledge field involved in the knowledge generation source on the geographical map is displayed.
2. The system according to claim 1, characterized in that The sources of knowledge generation include: administrative departments, enterprises, institutions, companies, school research institutions, social groups and organizations, and individuals.
3. The system according to claim 2, characterized in that The knowledge acquisition module parses the IP address or obtains the spatial location based on the name and address of the input knowledge generation source. The generation time is obtained by recording the time of receiving the knowledge through the knowledge acquisition module. The knowledge is reviewed manually or by computer to decide whether to adopt the knowledge and record the spatial location and generation time. The operation includes dragging the visualized HGNN, clicking on it to pop up a dialog box or search bar to search and / or view, modify, delete and / or restore the knowledge.
4. The system according to claim 3, characterized in that The knowledge parsing module is also used to use the first camera module to capture the user's face, identify other faces related to the user of the captured face through the first face recognition model, and provide the new input of the history corresponding to the other faces to the user of the captured face for selection, thereby retrieving the knowledge context and providing the analysis result of the attention mechanism based on the knowledge context; and the knowledge parsing module is also used to use the administrator's face to identify whether the administrator's facial expression is abnormal through the second face recognition model in the remote server, so that the remote server can decide whether to ask the administrator online whether there is any abnormality.
5. The system according to claim 4, characterized in that The first voice module is used to communicate with the user, thereby determining the new input based on the user's response, and the second voice module is used to communicate with the administrator, so that the knowledge analysis module comprehensively determines whether the administrator's facial expression is abnormal based on the administrator's response and the facial expression.
6. The system according to claim 5, characterized in that The relevant faces in the block-based heterogeneous graph neural network include the analysis results obtained from the new input of any user's history to determine the block, and collect the face of any user as the relevant face.
7. The system according to claim 6, characterized in that The first face recognition model includes mapping the collected faces to corresponding archive information, thereby associating with other related faces according to the archive information.
8. The system according to claim 7, characterized in that The second face recognition model includes a facial feature extraction module, an expression extraction module, an emotion judgment module, and an optimization module, wherein the facial feature extraction module performs finite element division on the received administrator's face, and extracts the forehead, face, eyes, lips, eyebrows, and chin areas, and completes multiple preset feature points of the areas; the expression extraction module uses a convolutional neural network or a generative adversarial network to perform classification training on the multiple preset feature points of the input area according to expression labels; the emotion judgment module calls the historical classification based on the identified classification to determine whether the administrator's facial expression is abnormal; the optimization module regularly collects the facial expressions of users and administrators, refines the emotion classification, and retrains the convolutional neural network or the generative adversarial network to obtain an optimized network.
9. The system according to claim 8, characterized in that The reply situation includes the delay time of receiving the administrator's voice signal, the speaking speed, the pitch, and the time interval between words. Whether the administrator's facial expression is abnormal is comprehensively judged based on the administrator's reply situation and the facial expression. Specifically, a support vector machine is established to classify the emotions of the reply situation, and the emotion judgment module compares the emotions based on the identified classification. If the two emotions are within the matching threshold range, the administrator's current emotion is determined. If they do not match, the second voice module initiates a procedure to inquire about the administrator's mood, and the emotions are classified again according to the reply situation, and the emotion judgment module compares the emotions based on the identified classification again. If they match, the current administrator's emotion is determined. If they still do not match, the remote server is notified for manual inquiry.