Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Understanding-assisted dialogue system artificial evaluation method and device and storage medium

A dialogue system and artificial technology, applied in the fields of electrical digital data processing, natural language data processing, instruments, etc., can solve problems such as the lack of systematic consideration and understanding process, and the quality of evaluation results reliability evaluation data needs to be improved, so as to improve reliability. , the effect of enhancing understanding

Active Publication Date: 2022-05-24
UNIV OF SCI & TECH OF CHINA
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, the existing evaluation schemes such as changing wording and adding references only focus on one or two research points, and do not systematically consider the comprehension process involved in the entire dialogue evaluation task. In summary, the reliability of the evaluation results and the evaluation Data quality needs to be improved

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Understanding-assisted dialogue system artificial evaluation method and device and storage medium
  • Understanding-assisted dialogue system artificial evaluation method and device and storage medium
  • Understanding-assisted dialogue system artificial evaluation method and device and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0032] like figure 1 As shown in the figure, a manual evaluation method for a dialogue system to assist understanding mainly includes the following steps:

[0033] Step 11: Screen out several dialog evaluation criteria from the existing evaluation criteria, construct an evaluation criteria framework, and generate a basic evaluation template.

[0034] Step 12: Design reading questions with reference to the reading comprehension assessment method, embed the reading questions into the dialogue content to be assessed on the basic assessment template, generate an assessment template containing the reading questions, and provide them to workers participating in the manual assessment of the dialogue system.

[0035] In the embodiment of the present invention, a missing sentence selection strategy and a dialogue content sorting strategy are designed. These two strategies correspond to different reading problems. A corresponding strategy can be selected according to the dialogue conten...

Embodiment 2

[0082] An embodiment of the present invention provides a device for artificial evaluation of a dialogue system to assist understanding, which is mainly implemented through the solution provided in the first embodiment, such as Figure 5 As shown, it mainly includes:

[0083] Evaluation criteria screening and basic evaluation template generation unit, used to filter out several dialogue evaluation criteria from the existing evaluation criteria, construct an evaluation criteria framework, and generate a basic evaluation template;

[0084] The reading question embedding unit is used to design reading questions with reference to the reading comprehension assessment method. On the basic assessment template, the reading questions are embedded into the dialogue content to be assessed, and an assessment template containing the reading questions is generated and provided to each participating dialogue system. Manually assessed workers;

[0085] The evaluation result screening unit is ...

Embodiment 3

[0089] The present invention also provides a processing device, such as Image 6 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the One or more processors implement the method provided in the first embodiment.

[0090] Further, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, the memory, the input device, and the output device are connected through a bus.

[0091] In this embodiment of the present invention, the specific types of the memory, the input device, and the output device are not limited; for example:

[0092] The input device can be a touch screen, an image capture device, a physical button or a mouse, etc.;

[0093] The output device can be a display terminal;

[0094] The memory may be random access memory (Random Access Memory, RAM), or may be...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses an understanding-assisted dialogue system artificial evaluation method and device and a storage medium, and the method comprises the steps: carrying out the grouping and sorting of common standards, selecting a to-be-evaluated dialogue standard, constructing a basic evaluation template, and referring to the common reading understanding test and reading strategy design in the Chinese and English examination. Reading tasks of single-choice completion dialogues and content dragging and sorting are embedded in the part, presenting the dialogue history, of the basic evaluation template, the time for evaluating the dialogue history by a worker is recorded and collected, the consistency index of answering by the worker is calculated according to a consistency algorithm, and improvement of the evaluation result by the dialogue history is illustrated and understood. According to the scheme, from the perspective of task understanding, the conversation evaluation scheme is perfected by enhancing the task understanding degree of workers, the reliability of worker evaluation is improved, and therefore high-quality evaluation data is obtained.

Description

technical field [0001] The present invention relates to the technical field of natural language processing and human-computer interaction, and in particular, to a method, device and storage medium for artificial evaluation of a dialogue system for assisting understanding. Background technique [0002] The research of chat-type dialogue is a research topic that cannot be ignored in the field of natural language processing, and the current immature dialogue evaluation technology greatly limits the further research and application of dialogue systems. According to the different evaluation subjects, dialogue system evaluation is generally divided into two methods: manual evaluation and automatic evaluation. Automated evaluation mainly uses some evaluation indicators and evaluation models, and uses statistical dialogue features and even deep learning models to mine the semantic relationship in the dialogue context, so as to carry out automatic evaluation; however, for chats in th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F40/205G06F40/186
CPCG06F40/205G06F40/186
Inventor 李华庆何向南向元新
Owner UNIV OF SCI & TECH OF CHINA
Features
  • Generate Ideas
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com