Intelligent evaluation method and device based on real-time session and multi-modal large model

Abstract

The invention discloses an intelligent evaluation method and device based on a real-time session and a multi-modal large model, belongs to the technical field of government affair services, and aims to solve the technical problem of how to ensure the authenticity and objectivity of government affair service evaluation and effectively suppress malicious evaluation and false evaluation. According to the technical scheme, the method comprises the following steps: data acquisition: acquiring an audio file through a PC terminal, a recording device or a window terminal; reading a text file: processing the audio file through a natural language processing technology of a large model, and reading the text file; dialogue semantic analysis: after reading the text file, analyzing semantics of both dialogue parties through a large model speech recognition technology; and service evaluation: based on a semantic analysis result, performing multi-dimensional scoring on the performance of the service personnel. The device comprises an application layer, a data layer, a capability layer, a service layer and a system layer.

Description

Technical Field

[0001] The present invention relates to the field of government service technology, and in particular to an intelligent evaluation method and device based on real-time conversation and multimodal large models. Background Art

[0002] The current government service evaluation system explicitly upholds the principle of voluntary and independent evaluation by service providers, aiming to ensure the fairness of the evaluation process and the reliability of the results. It explicitly prohibits any form of coercion or interference that could influence the service provider's willingness to evaluate, striving to obtain more authentic service feedback by protecting the evaluator's autonomy.

[0003] However, in actual operations, after the business is completed, due to various reasons, such as busy affairs, memory loss, or neglect of the evaluation system, the people who do the business often forget to evaluate or do not take the initiative to initiate the evaluation. The prevalence of this phenomenon undoubtedly weakens the role of the evaluation system in reflecting the quality of service. In order to address this problem, the current government evaluation system has a one-click evaluation function, which aims to simplify the evaluation process and encourage people to actively participate in the evaluation, but the actual effect is not satisfactory. In actual applications, people who do the business click on the evaluation casually, and even staff members evaluate themselves on their behalf. These irregular evaluation behaviors undoubtedly further weaken the objectivity and accuracy of the evaluation results.

[0004] These irregular evaluation practices not only make it difficult for government service centers to obtain genuine and objective evaluations from users, but also prevent them from accurately and truthfully reflecting the service quality and work efficiency of the service halls. In the long run, this will have a negative impact on improving the quality of government services and will also reduce users' trust in the government service system.

[0005] Therefore, how to ensure the authenticity and objectivity of government service evaluations and effectively curb the occurrence of malicious and false evaluations is a technical problem that needs to be solved urgently. Summary of the Invention

[0006] The technical task of the present invention is to provide an intelligent evaluation method and device based on real-time conversation and multimodal large model to solve the problem of how to ensure the authenticity and objectivity of government service evaluation and effectively curb the occurrence of malicious evaluation and false evaluation.

[0007] The technical task of the present invention is achieved in the following manner: an intelligent evaluation method based on real-time conversation and multimodal large model, the method is specifically as follows:

[0008] Data collection: Collect audio files through PC, recording equipment or window terminal;

[0009] Text file reading: Process audio files using natural language processing technology based on large models to read text files;

[0010] Conversational semantic analysis: After reading the text file, the semantics of the two parties in the conversation are analyzed using large-scale speech recognition technology;

[0011] Service evaluation: Based on the results of semantic analysis, the service staff's performance is scored in multiple dimensions.

[0012] As a preference, the text file is read as follows:

[0013] Get .Txt files through large model text reading technology;

[0014] Using word segmentation and vectorization techniques, the conversation text is segmented into words or subword units to generate a .JSON file. Using word embedding technology, the data information in the .JSON file is mapped into a high-dimensional continuous vector to capture the semantic relationship between words.

[0015] At the same time, position information is added to the word vector through position encoding to obtain the .npy file, ensuring that the large model can understand the temporal nature of the sequence;

[0016] It uses self-attention mechanism and hierarchical feature extraction technology to dynamically focus on key information and abstract text features layer by layer, gradually deepening the understanding of the conversation content from surface grammar to deep semantics.

[0017] As a preference, the conversation semantic analysis is specifically as follows:

[0018] Through large-scale pre-training, the model learns language patterns and common sense knowledge;

[0019] Combined with downstream tasks, supervised fine-tuning is performed to optimize the model's performance in specific scenarios (such as this conversation service scenario);

[0020] At the same time, context-aware technology is used to dynamically adjust context weights to solve the problem of polysemous words; and combined with external knowledge bases and logical constraints, the model's ability to recognize implicit semantics is improved to ensure accurate understanding of the conversation content.

[0021] More optimally, we leverage the self-attention mechanism and hierarchical feature extraction technology to dynamically focus on key information and gradually abstract text features layer by layer, gradually deepening our understanding of the conversation from surface syntax to deep semantics. Specifically, we:

[0022] Perform speaker segmentation on audio files to accurately distinguish the speech segments of different speakers and verify the speaker's identity through voiceprint recognition technology;

[0023] The speech recognition algorithm Conformer combines CNN and Transformer to convert audio content into text form;

[0024] Using syntactic dependency analysis, we can deeply analyze sentence structures to generate JSON files, identify the core components of subject, predicate, and object, clarify "who did what to whom", eliminate ambiguity, and ensure accurate understanding of semantics;

[0025] Extract and annotate key entity information in the text through regular expressions and sensitive word libraries using real-name entity recognition technology;

[0026] Detailed identification of emotions contained in texts, such as complaints, anger, and swear words, based on their content;

[0027] Using Word2Vec and GloVe technologies, we map words into high-dimensional vectors, capturing semantic similarities to further explore the deeper meaning of text and provide rich context for subsequent processing.

[0028] Produce concise and clear reports or summaries through text generation technology.

[0029] An intelligent evaluation device based on real-time conversation and multimodal large model, the device comprising an application layer, a data layer, a capability layer, a service layer, and a system layer;

[0030] The application layer is used to deploy a queuing system for audio collection. The queuing system sends precise instructions through the WebSocket protocol to the recording terminals of the audio pickup, window terminals, and PC-side audio collection devices to start or stop recording. After the recording is completed, the system is evaluated when the recording file is seamlessly transferred.

[0031] The data layer uses MySQL data for data storage, retrieval, synchronization and security, providing an atomic data operation interface for the upper layer;

[0032] The capability layer and service layer use the pickup capability of the microphone to achieve recording, and use Javaa to encapsulate reusable basic capability AI algorithms and AI rule engines to provide standardized capability output;

[0033] The system layer uses Java and Vue2 technologies to implement the functions presented by the final system, including system settings, data processing, data analysis and data retrieval.

[0034] Preferably, the evaluation system includes:

[0035] Data acquisition module, used to collect audio files through PC, recording device or window terminal;

[0036] The text file reading module is used to process audio files and read text files through the natural language processing technology of the large model;

[0037] The conversation semantics analysis module is used to read text files and analyze the semantics of the two parties in the conversation using large-scale speech recognition technology;

[0038] The service evaluation module is used to provide multi-dimensional scoring of service personnel's performance based on semantic analysis results.

[0039] More preferably, the text file reading module includes:

[0040] The file acquisition submodule is used to obtain .Txt files through large model text reading technology;

[0041] The semantic capture submodule uses word segmentation and vectorization techniques to segment the conversation text into words or subword units to obtain a .JSON file. It then uses word embedding technology to map the data information in the .JSON file into high-dimensional continuous vectors to capture the semantic relationships between words.

[0042] The position information acquisition submodule is used to add position information to word vectors through position encoding to obtain .npy files, ensuring that the large model can understand the temporal nature of the sequence;

[0043] The key information focusing submodule is used to dynamically focus on key information using the self-attention mechanism and hierarchical feature extraction technology, and abstract text features layer by layer, gradually deepening the understanding of the conversation content from surface grammar to deep semantics.

[0044] Preferably, the conversation semantic analysis includes:

[0045] The model learning submodule is used to enable the model to learn language regularities and common sense knowledge through large-scale pre-training;

[0046] The fine-tuning submodule is used to perform supervised fine-tuning in conjunction with downstream tasks to optimize the model's performance in specific scenarios (such as this conversation service scenario);

[0047] The weight adjustment submodule is used to dynamically adjust context weights using context-aware technology to resolve the ambiguity problem of polysemous words; and combines external knowledge bases and logical constraints to improve the model's ability to recognize implicit semantics and ensure accurate understanding of the conversation content.

[0048] An electronic device comprising: a memory and at least one processor;

[0049] Wherein, the memory stores a computer program;

[0050] The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the intelligent evaluation method based on real-time conversation and multimodal large model as described above.

[0051] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the above-mentioned intelligent evaluation method based on real-time conversation and multimodal large model.

[0052] The intelligent evaluation method and device based on real-time conversation and multimodal large model of the present invention have the following advantages:

[0053] (1) The present invention uses large-scale model technology to intervene in the entire process of offline hall processing. Through speaker segmentation, voiceprint recognition, speech recognition, syntactic dependency analysis, real-name entity recognition, sentiment classification, semantic understanding, text generation and other technologies, it realizes semantic analysis and multi-dimensional evaluation of service providers and staff, builds a more objective, real and scientific business environment evaluation system, realizes the monitoring of the service process and extracorporeal circulation matters, ensures the authenticity and objectivity of the evaluation, effectively prevents the emergence of malicious and false evaluations, and provides strong support for government decision-making;

[0054] (2) The present invention further improves the evaluation system, ensures the authenticity and objectivity of the evaluation, effectively curbs the occurrence of malicious and false evaluations, and establishes a more scientific and reasonable evaluation mechanism; at the same time, it expands the original single five-level evaluation system (very dissatisfied, dissatisfied, satisfied, basically satisfied, and very satisfied) to more detailed evaluation dimensions (such as service etiquette, satisfaction, handling basis, proactive service, service quality, etc.), analyzes all aspects of the service process on demand, and enables targeted improvement of government services;

[0055] (3) The present invention's intelligent evaluation solution, based on real-time conversations and a multimodal large model, integrates advanced semantic analysis, sentiment analysis, entity recognition, and other technologies to conduct in-depth mining and analysis of massive amounts of language data generated by interactions at government service windows, thereby forming a technical solution model that can be widely applied and implemented in government products.

[0056] (4) The present invention uses technologies such as speaker segmentation, voiceprint recognition, speech recognition, syntactic dependency analysis, real-name entity recognition, sentiment classification, semantic understanding, and text generation to build a seamless evaluation system, which records the entire process of the applicant from consultation, application, handling, and completion, and implements semantic analysis and multi-dimensional evaluation of applicants and staff. This system builds a more objective, authentic, and scientific business environment evaluation system, monitors the process of handling affairs and extracorporeal circulation matters, ensures the authenticity and objectivity of the evaluation, and effectively prevents the emergence of malicious and false evaluations.

[0057] (5) The present invention uses multimodal deep fusion to explore the joint modeling of text, voice, and video to achieve more comprehensive semantic understanding and ensure accuracy;

[0058] (6) The recording scoring process of the present invention does not require human intervention, thus maintaining objectivity;

[0059] (7) This invention uses multi-dimensional testing and scoring, such as dialogue compliance, business capabilities, and service attitude, to provide more detailed evaluations than traditional ones. It can help service personnel more clearly improve their missing service capabilities and increase public satisfaction.

[0060] (8) The present invention can quickly process large amounts of data and perform semantic analysis from multiple dimensions, including sentiment analysis and entity recognition, to provide more comprehensive conclusions and greatly improve evaluation efficiency. It can also provide timely feedback on evaluation results to provide the government with more comprehensive and in-depth decision-making support.

[0061] (9) The present invention records the window service process in real time and uses speech-to-text technology and large-scale text reading and semantic recognition technology to perform multi-dimensional service scoring, such as service satisfaction, service etiquette, and handling basis;

[0062] (10) The present invention uses microphone recording, combines speech-to-text technology with large-model text reading and semantic recognition technology, and applies it to the evaluation of government services in a more detailed and objective manner without the need for human intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The present invention will be further described below with reference to the accompanying drawings.

[0064] Attachment Figure 1 Schematic diagram of the structure of an intelligent evaluation device based on real-time conversation and multimodal large model;

[0065] Attachment Figure 2 is a schematic diagram of the evaluation system;

[0066] Attachment Figure 3 Schematic diagram of the working process of the intelligent evaluation device based on real-time conversation and multimodal large model. DETAILED DESCRIPTION

[0067] The intelligent evaluation method and device based on real-time conversation and multimodal large model of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments.

[0068] Example 1:

[0069] This embodiment provides an intelligent evaluation method based on real-time conversation and a multimodal large model. The method is specifically as follows:

[0070] S1. Data collection: Collect audio files through PC, recording device or window terminal;

[0071] S2. Text file reading: The audio file is processed using natural language processing technology of a large model to read the text file;

[0072] S3, Conversational Semantic Analysis: After reading the text file, the semantics of the two parties in the conversation are analyzed using large-scale speech recognition technology;

[0073] S4. Service evaluation: Based on the results of semantic analysis, the service staff’s performance is scored in multiple dimensions.

[0074] The text file reading in step S2 of this embodiment is specifically as follows:

[0075] S201, obtaining a .txt file through a large model text reading technology;

[0076] S202. Segment the conversation text into words or subwords using word segmentation and vectorization techniques to obtain a .JSON file. Use word embedding technology to map the data in the .JSON file into high-dimensional continuous vectors to capture the semantic relationships between words.

[0077] S203. Add position information to the word vector through position encoding to obtain a .npy file to ensure that the large model can understand the temporal nature of the sequence;

[0078] S204. Use the self-attention mechanism and hierarchical feature extraction technology to dynamically focus on key information and abstract text features layer by layer, gradually deepening the understanding of the conversation content from surface grammar to deep semantics.

[0079] The specific details of the conversation semantic analysis in step S3 of this embodiment are as follows:

[0080] S301. Use large-scale pre-training to enable the model to learn language patterns and common sense knowledge;

[0081] S302: Perform supervised fine-tuning in conjunction with downstream tasks to optimize the model's performance in specific scenarios (such as this conversation service scenario);

[0082] S303. Use context-aware technology to dynamically adjust context weights to resolve the ambiguity problem of polysemous words; and combine external knowledge bases and logical constraints to improve the model's ability to recognize implicit semantics and ensure accurate understanding of the conversation content.

[0083] In step S204 of this embodiment, the self-attention mechanism and hierarchical feature extraction technology are used to dynamically focus on key information, abstract text features layer by layer, and gradually deepen the understanding of the conversation content from surface syntax to deep semantics. The details are as follows:

[0084] S20401. Segment the audio file to accurately distinguish the speech segments of different speakers and verify the speaker's identity through voiceprint recognition technology;

[0085] S20402, using the speech recognition algorithm Conformer to fuse CNN and Transformer, convert the audio content into text form;

[0086] S20403. Utilize syntactic dependency analysis to deeply analyze sentence structures and generate JSON files, identifying the core components of subject, predicate, and object, clarifying "who did what to whom," eliminating ambiguity and ensuring accurate understanding of semantics.

[0087] S20404. Extract and annotate key entity information in the text using regular expressions and sensitive word libraries using real-name entity recognition technology;

[0088] S20405. Detailed identification of emotions contained in texts, such as complaints, anger, and swear words, based on their content.

[0089] S20406, using Word2Vec and GloVe technologies to map words into high-dimensional vectors, capturing semantic similarities to further explore the deeper meaning of texts and provide rich context for subsequent processing;

[0090] S20407. Generate concise and clear reports or summaries through text generation technology.

[0091] Artificial intelligence (AI) evaluation technology plays a central role in this process, providing in-depth analysis and intelligent mining of evaluation data. Furthermore, the system highlights keywords in different colors based on their characteristics and importance, helping managers quickly locate key information and efficiently complete review and processing tasks.

[0092] Example 2:

[0093] As attached Figure 1 As shown, this embodiment provides an intelligent evaluation device based on real-time conversation and multimodal large model, which includes an application layer, a data layer, a capability layer, a service layer and a system layer;

[0094] The application layer is used to deploy a queuing system for audio collection. The queuing system sends precise instructions through the WebSocket protocol to the recording terminals of the audio pickup, window terminals, and PC-side audio collection devices to start or stop recording. After the recording is completed, the system is evaluated when the recording file is seamlessly transferred.

[0095] The data layer uses MySQL data for data storage, retrieval, synchronization and security, providing an atomic data operation interface for the upper layer;

[0096] The capability layer and service layer use the pickup capability of the microphone to achieve recording, and use Javaa to encapsulate reusable basic capability AI algorithms and AI rule engines to provide standardized capability output;

[0097] The system layer uses Java and Vue2 technologies to implement the functions presented by the final system, including system settings, data processing, data analysis and data retrieval.

[0098] As attached Figure 2 As shown, the evaluation system in this embodiment includes:

[0099] Data acquisition module, used to collect audio files through PC, recording device or window terminal;

[0100] The text file reading module is used to process audio files and read text files through the natural language processing technology of the large model;

[0101] The conversation semantics analysis module is used to read text files and analyze the semantics of the two parties in the conversation using large-scale speech recognition technology;

[0102] The service evaluation module is used to provide multi-dimensional scoring of service personnel's performance based on semantic analysis results.

[0103] The text file reading module in this embodiment includes:

[0104] The file acquisition submodule is used to obtain .Txt files through large model text reading technology;

[0105] The semantic capture submodule uses word segmentation and vectorization techniques to segment the conversation text into words or subword units to obtain a .JSON file. It then uses word embedding technology to map the data information in the .JSON file into high-dimensional continuous vectors to capture the semantic relationships between words.

[0106] The position information acquisition submodule is used to add position information to word vectors through position encoding to obtain .npy files, ensuring that the large model can understand the temporal nature of the sequence;

[0107] The key information focusing submodule is used to dynamically focus on key information using the self-attention mechanism and hierarchical feature extraction technology, and abstract text features layer by layer, gradually deepening the understanding of the conversation content from surface grammar to deep semantics.

[0108] The conversation semantic analysis in this embodiment includes:

[0109] The model learning submodule is used to enable the model to learn language regularities and common sense knowledge through large-scale pre-training;

[0110] The fine-tuning submodule is used to perform supervised fine-tuning in conjunction with downstream tasks to optimize the model's performance in specific scenarios (such as this conversation service scenario);

[0111] The weight adjustment submodule is used to dynamically adjust context weights using context-aware technology to resolve the ambiguity problem of polysemous words; and combines external knowledge bases and logical constraints to improve the model's ability to recognize implicit semantics and ensure accurate understanding of the conversation content.

[0112] As attached Figure 3 As shown in the figure, the device operates as follows: First, a high-performance voice recording device is installed at the service window. This device is seamlessly integrated with the queuing system to ensure accurate queue status. When a customer begins queuing, the system automatically triggers the voice recording device to start recording in real time. Once the customer has finished queuing, the system automatically triggers the voice recording device to start recording in real time. Once the recording is complete, the file is automatically uploaded to a designated program. This program not only converts speech to text but also uses a large-scale model to score the recording, enabling efficient service quality monitoring and analysis.

[0113] Example 3:

[0114] This embodiment also provides an electronic device, including: a memory and at least one processor;

[0115] wherein the memory stores computer-executable instructions;

[0116] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the intelligent evaluation method based on real-time conversation and multimodal large model in any embodiment of the present invention.

[0117] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0118] The memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, the memory can also include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state memory devices.

[0119] Example 4:

[0120] This embodiment also provides a computer-readable storage medium storing a plurality of instructions, which are loaded by a processor to cause the processor to execute the intelligent evaluation method based on real-time conversation and a multimodal large model according to any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, wherein the storage medium stores software program code that implements the functions of any of the above-described embodiments, and a computer (or CPU or MPU) of the system or device can be configured to read and execute the program code stored in the storage medium.

[0121] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0122] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RYMs, DVD-RWs, DVD+RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer via a communications network.

[0123] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0124] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU installed on the expansion board or expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent evaluation method based on real-time conversation and multimodal large model, characterized by: The method is as follows: Data collection: Collect audio files through PC, recording equipment or window terminal; Text file reading: Process audio files using natural language processing technology based on large models to read text files; Conversational semantic analysis: After reading the text file, the semantics of the two parties in the conversation are analyzed using large-scale speech recognition technology; Service evaluation: Based on the results of semantic analysis, the service staff's performance is scored in multiple dimensions.

2. The intelligent evaluation method based on real-time conversation and multimodal large model according to claim 1 is characterized in that: The text file reading is as follows: Get .Txt files through large model text reading technology; Using word segmentation and vectorization techniques, the conversation text is segmented into words or subword units to generate a .JSON file. Using word embedding technology, the data information in the .JSON file is mapped into a high-dimensional continuous vector to capture the semantic relationship between words. At the same time, position information is added to the word vector through position encoding to obtain the .npy file, ensuring that the large model can understand the temporal nature of the sequence; It uses self-attention mechanism and hierarchical feature extraction technology to dynamically focus on key information and abstract text features layer by layer, gradually deepening the understanding of the conversation content from surface grammar to deep semantics.

3. The intelligent evaluation method based on real-time conversation and multimodal large model according to claim 1 is characterized in that: The specific semantic analysis of the conversation is as follows: Through large-scale pre-training, the model learns language patterns and common sense knowledge; Combined with downstream tasks for supervised fine-tuning to optimize the model's performance in specific scenarios; At the same time, context-aware technology is used to dynamically adjust context weights to solve the problem of polysemous words; It also combines external knowledge bases and logical constraints to improve the model's ability to recognize implicit semantics and ensure accurate understanding of the conversation content.

4. The intelligent evaluation method based on real-time conversation and multimodal large model according to any one of claims 1 to 3, characterized in that: Utilizing the self-attention mechanism and hierarchical feature extraction technology, we dynamically focus on key information and gradually abstract text features layer by layer, gradually deepening our understanding of the conversation content from surface syntax to deep semantics. The details are as follows: Perform speaker segmentation on audio files to accurately distinguish the speech segments of different speakers and verify the speaker's identity through voiceprint recognition technology; The speech recognition algorithm Conformer combines CNN and Transformer to convert audio content into text form; Using syntactic dependency analysis, we can deeply analyze the sentence structure to generate JSON files, identify the core components of subject, predicate, and object, eliminate ambiguity, and ensure accurate understanding of semantics. Extract and annotate key entity information in the text through regular expressions and sensitive word libraries using real-name entity recognition technology; Make a detailed judgment on the emotions contained in the text through the content of the text; Use Word2Vec and GloVe technologies to map words into high-dimensional vectors, capturing semantic similarities and further exploring the deeper meaning of texts; Produce concise and clear reports or summaries through text generation technology.

5. An intelligent evaluation device based on real-time conversation and multimodal large model, characterized in that: The device includes an application layer, a data layer, a capability layer, a service layer, and a system layer; The application layer is used to deploy a queuing system for audio collection. The queuing system sends precise instructions through the WebSocket protocol to the recording terminals of the audio pickup, window terminals, and PC-side audio collection devices to start or stop recording. After the recording is completed, the system is evaluated when the recording file is seamlessly transferred. The data layer uses MySQL data for data storage, retrieval, synchronization and security, providing an atomic data operation interface for the upper layer; The capability layer and service layer use the pickup capability of the microphone to achieve recording, and use Javaa to encapsulate reusable basic capability AI algorithms and AI rule engines to provide standardized capability output; The system layer uses Java and Vue2 technologies to implement the functions presented by the final system, including system settings, data processing, data analysis and data retrieval.

6. The intelligent evaluation device based on real-time conversation and multimodal large model according to claim 5 is characterized in that: The evaluation system includes: Data acquisition module, used to collect audio files through PC, recording device or window terminal; The text file reading module is used to process audio files and read text files through the natural language processing technology of the large model; The conversation semantics analysis module is used to read text files and analyze the semantics of the two parties in the conversation using large-scale speech recognition technology; The service evaluation module is used to provide multi-dimensional scoring of service personnel's performance based on semantic analysis results.

7. The intelligent evaluation device based on real-time conversation and multimodal large model according to claim 6 is characterized in that: The text file reading module includes: The file acquisition submodule is used to obtain .Txt files through large model text reading technology; The semantic capture submodule uses word segmentation and vectorization techniques to segment the conversation text into words or subword units to obtain a .JSON file. It then uses word embedding technology to map the data information in the .JSON file into high-dimensional continuous vectors to capture the semantic relationships between words. The position information acquisition submodule is used to add position information to word vectors through position encoding to obtain .npy files, ensuring that the large model can understand the temporal nature of the sequence; The key information focusing submodule is used to dynamically focus on key information using the self-attention mechanism and hierarchical feature extraction technology, and abstract text features layer by layer, gradually deepening the understanding of the conversation content from surface grammar to deep semantics.

8. The intelligent evaluation device based on real-time conversation and multimodal large model according to claim 6 is characterized in that: Conversation semantic analysis includes: The model learning submodule is used to enable the model to learn language regularities and common sense knowledge through large-scale pre-training; The fine-tuning submodule is used to perform supervised fine-tuning in conjunction with downstream tasks to optimize the performance of the model in specific scenarios; The weight adjustment submodule is used to dynamically adjust context weights using context-aware technology to resolve the ambiguity problem of polysemous words; and combines external knowledge bases and logical constraints to improve the model's ability to recognize implicit semantics and ensure accurate understanding of the conversation content.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the intelligent evaluation method based on real-time conversation and multimodal large model as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the intelligent evaluation method based on real-time conversation and multimodal large model as described in any one of claims 1 to 4.

Images