Sample library construction method, diversified examination system, equipment and storage medium

Abstract

The invention provides a sample library construction method, a diversified examination system, equipment and a storage medium, a question bank generation module obtains various types of question materials, generates examination questions and standard answers thereof based on the various types of question materials, and generates different examination question banks based on the examination questions and the standard answers; the user examination module responds to the question selection instruction, extracts a plurality of target questions indicated by the question selection instruction from each examination question bank, and displays each target question to a target user; for the questions without the standard answers, the standard answer generation module generates the standard answers of the questions based on the answers of the users, and the sample library construction module constructs a sample library based on the question materials of various types, the standard answers of the questions and the answers submitted by the target users for the target questions. By adopting the method, the answers in the sample library can be ensured to be closer to real scenes and user requirements, and the quality, effectiveness and accuracy of the sample library are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a sample library construction method, a diversified examination system, a device and a storage medium. Background Art

[0002] Samples play a vital role in the training of artificial intelligence models. The lack of samples has a huge impact on the capabilities of artificial intelligence models. Currently, in most industries, medical care, education and other fields, there is a serious lack of correctly labeled and clearly labeled samples, which makes it difficult to promote the application of artificial intelligence in the industry.

[0003] There are currently two main ways to build a sample library. One is to collect images, videos, audios, and texts based on annotation software, and assign annotators to perform manual annotations before construction; the other is to develop automatic algorithms for automatic annotation based on manual annotations and then build. However, in the study, it was found that in order to obtain suitable samples, the annotation personnel need to have corresponding industry knowledge. However, the current samples are mainly annotated by a third-party annotation team, or several professionals are assigned to cooperate in annotation. There are many annotation errors and limited quality, which cannot meet the subsequent artificial intelligence model training requirements. The automatic algorithm also needs to manually perform a certain number of sample library construction work, and then the algorithm learns these annotation rules for automatic annotation. It can only increase the number of samples, but cannot add new knowledge autonomously. Its annotation quality is also affected by manual annotation. For some rules with incorrect annotations and poor quality, the algorithm may amplify the adverse effects, resulting in further decline in sample quality. Therefore, no matter which of the above methods is used to build a sample library, the quality, effectiveness and accuracy of the sample library will be reduced. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a sample library construction method, a diversified examination system, a device and a storage medium to ensure that the answers in the sample library are closer to real scenarios and user needs, and to improve the quality, effectiveness and accuracy of the sample library.

[0005] In a first aspect, an embodiment of the present application provides a sample library construction method, which is applied to a diversified examination system, wherein the system includes a question bank generation module, a user examination module, a standard answer generation module and a sample library construction module, and the method includes:

[0006] The question bank generation module obtains various types of question materials, generates a number of test questions and standard answers for each test question based on the various types of question materials, generates a first test question bank based on each test question, or generates a second test question bank based on each test question and its standard answer;

[0007] When generating a first test question bank based on each test question, the user test module extracts a number of first target questions from the first test question bank in response to the question selection instruction, and displays each first target question to the target user; the standard answer generation module generates a first standard answer for each first target question based on the first answer submitted by the target user for each first target question; the sample library construction module constructs a first sample library based on various types of question materials, each first standard answer and each first answer;

[0008] When a second test question bank is generated based on each test question and each standard answer, the user test module responds to the question selection instruction, extracts a number of second target questions from the second test question bank, and displays each second target question to the target user; the sample library construction module constructs a second sample library based on various types of question materials, standard answers to each test question and the second answer submitted by the target user to each second target question.

[0009] Optionally, the system further includes a score calculation module and a score display module, and the method further includes:

[0010] The score calculation module calculates the test score of the target user based on the answers submitted by the target user to each target question and the standard answers to each target question;

[0011] The score display module displays the test scores.

[0012] Optionally, a first sample library is constructed based on various types of question materials, first standard answers and first answer answers, including:

[0013] Constructing a first basic sample library based on various types of question materials and first standard answers;

[0014] Build an enhanced sample library based on various types of question materials and first answers;

[0015] The constructing of the second sample library based on various types of question materials, standard answers to each test question and second answers submitted by the target user for each second target question includes:

[0016] Constructing a second basic sample library based on various types of question materials and standard answers to each test question;

[0017] A second enhanced sample library is constructed based on various types of question materials and second answer answers.

[0018] Optionally, the system further comprises a question bank disclosure module and a question bank verification module, and the method further comprises:

[0019] The question bank disclosure module discloses each test question and its standard answer in the second test question bank;

[0020] The question bank checking module checks each test question and its standard answer.

[0021] Optionally, the display interface of the diversified examination system includes a question display area, a question number display area, a tool display area, a material display area and an answer input area, wherein the question display area is used to display each target question, the question number display area is used to display the question number of each target question, the material display area is used to display the material of each question, the tool display area is used to display the tools corresponding to the material of each question, and the answer input area is used to input the answer to each target question.

[0022] In a second aspect, an embodiment of the present application provides a diversified examination system, the system comprising a question bank generation module, a user examination module, a standard answer generation module and a sample library construction module;

[0023] The question bank generating module is used to obtain various types of question materials, generate a number of test questions and standard answers for each test question based on the various types of question materials, generate a first test question bank based on each test question, or generate a second test question bank based on each test question and its standard answer;

[0024] When generating a first test question bank based on each test question, the user test module is used to extract a number of first target questions from the first test question bank in response to a question selection instruction, and display each first target question to a target user; the standard answer generation module is used to generate a first standard answer to each first target question based on the first answer submitted by the target user to each first target question; the sample library construction module is used to construct a first sample library based on various types of question materials, each first standard answer and each first answer;

[0025] When a second test question bank is generated based on each test question and each standard answer, the user test module is used to extract a number of second target questions from the second test question bank in response to a question selection instruction, and display each second target question to the target user; the sample library construction module is used to construct a second sample library based on various types of question materials, standard answers to each test question and second answers submitted by the target user for each second target question.

[0026] Optionally, the system further includes a score calculation module and a score display module;

[0027] The score calculation module is used to calculate the test score of the target user based on the answers submitted by the target user to each target question and the standard answers to each target question;

[0028] The score display module is used to display the test scores.

[0029] Optionally, a first sample library is constructed based on various types of question materials, first standard answers and first answer answers, including:

[0030] Constructing a first basic sample library based on various types of question materials and first standard answers;

[0031] Build an enhanced sample library based on various types of question materials and first answers;

[0032] The constructing of the second sample library based on various types of question materials, standard answers to each test question and second answers submitted by the target user for each second target question includes:

[0033] Constructing a second basic sample library based on various types of question materials and standard answers to each test question;

[0034] A second enhanced sample library is constructed based on various types of question materials and second answer answers.

[0035] Optionally, the system further comprises a question bank disclosure module and a question bank verification module;

[0036] The question bank disclosure module is used to disclose each test question and its standard answer in the second test question bank;

[0037] The question bank checking module is used to check each test question and its standard answer.

[0038] Optionally, the display interface of the diversified examination system includes a question display area, a question number display area, a tool display area, a material display area and an answer input area, wherein the question display area is used to display each target question, the question number display area is used to display the question number of each target question, the material display area is used to display the material of each question, the tool display area is used to display the tools corresponding to the material of each question, and the answer input area is used to input the answer to each target question.

[0039] In a third aspect, an embodiment of the present application provides a computer device, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the sample library construction method described in any optional implementation of the first aspect are performed.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the sample library construction method described in any optional implementation of the first aspect are executed.

[0041] The technical solution provided by this application includes but is not limited to the following beneficial effects:

[0042] This application first widely collects and integrates materials of various types of questions through the question bank construction module, generates test questions and standard answers based on the question materials, and then generates a first test question bank based on the test questions alone; or combines the test questions with their corresponding standard answers to generate a second test question bank. The user interactive test module responds to the user's question selection instruction, determines the target question from the first test question bank or the second test question bank, and displays the target question to the user, so that the user can answer each target question and provide answers, providing a construction basis for building a sample library.

[0043] Then, for the target questions extracted from the first test question bank that do not have corresponding standard answers, the standard answer generation module will dynamically generate standard answers for each question based on the user's answers as the basis for building the sample library. For the target questions extracted from the second test question bank that have corresponding standard answers, the standard answers in the question bank are directly used as the basis for building the sample library. This step dynamically generates standard answers so that the sample library can be flexibly adjusted according to the actual user's answers, thereby better adapting to the needs and levels of different user groups. And because the standard answers are generated based on the user's actual answers, this helps ensure that the answers in the sample library are closer to real scenarios and user needs.

[0044] Finally, the sample library construction module constructs a sample library based on question materials, standard answers and user answers. It can combine each test with sample annotation to involve each user in the sample annotation process, solving the current problem of insufficient patterns and knowledge learned in sample annotation. At the same time, sample annotation is completed through a diversified examination system, which greatly improves the efficiency of sample annotation and improves the quality, effectiveness and accuracy of the sample library.

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 A flow chart of a sample library construction method provided in the first embodiment of the present invention is shown;

[0048] Figure 2 A flowchart of a diversified examination method provided by the first embodiment of the present invention is shown;

[0049] Figure 3 A flow chart of a question bank checking method provided by the first embodiment of the present invention is shown;

[0050] Figure 4 A schematic diagram showing a display interface of a diversified examination system provided in the first embodiment of the present invention is shown;

[0051] Figure 5 A schematic diagram showing a display interface of a second diversified examination system provided in the first embodiment of the present invention is shown;

[0052] Figure 6 A schematic diagram showing the implementation of a diversified examination system function provided by the first embodiment of the present invention;

[0053] Figure 7 A schematic diagram showing a diversified examination process provided by the first embodiment of the present invention;

[0054] Figure 8 A schematic diagram showing a second diversified examination process provided by the first embodiment of the present invention;

[0055] Fig. 9 A schematic diagram of the structure of a diversified examination system provided by the second embodiment of the present invention is shown;

[0056] Fig.10 A schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention is shown. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0058] Embodiment 1

[0059] To facilitate understanding of this application, Figure 1 The flowchart of a sample library construction method provided in the first embodiment of the present invention is shown to describe the content of the first embodiment of the present application in detail.

[0060] See also Figure 1 As shown, Figure 1 A flow chart of a sample library construction method provided by the first embodiment of the present invention is shown, wherein the method is applied to a diversified examination system, the system includes a question bank generation module, a user examination module, a standard answer generation module and a sample library construction module, and the method includes steps S101 to S103:

[0061] S101: The question bank generation module obtains various types of question materials, generates a number of test questions and standard answers for each test question based on various types of question materials, generates a first test question bank based on each test question, or generates a second test question bank based on each test question and its standard answer.

[0062] Specifically, you first need to upload the test material through the upload submodule of the question bank generation module in the diversified examination system, and set the question description. The question description is set by entering text to explain the test content and answer method. The question material can be four types: image, video, text, and audio; according to different types of materials, the diversified examination system provides different question creation tools.

[0063] Generating a first test question bank based on each test question includes: integrating the question material and the question description into a database to finally obtain the first test question bank.

[0064] The generating of the second test question bank based on each test question and its standard answer includes: integrating the question material, the question description and the standard answer into a database to finally obtain the second test question bank.

[0065] When integrating the title materials and title descriptions, various shapes of box selection marks, closed polyline tools, and text annotation tools are provided for image materials. The box selection mark tool uses a quadrilateral or ellipse to confirm the starting point by holding down the left mouse button, and release the left mouse button to complete the end point, forming a quadrilateral or ellipse frame at the specified position. The closed polyline tool determines the starting and end points of the line segment by clicking with the left mouse button, and the end point is used as the starting point of the next line segment to form a polygon of any shape and any number of sides. In addition, a multi-level list setting tool is provided. The multi-level list setting tool is to pre-set multiple alternative text annotations and classify them. When setting text annotations for a large number of similar materials, you can select the corresponding text annotation by clicking in the list, which is convenient and quick to add text description answers to the box selection part or the entire image. The box selection and the corresponding text annotation or multi-level list selection item annotation are in order, which also applies to the following materials. After making the question, the answer is saved in the examination system as the coordinate data of the starting point and end point of the box selection, the coordinate data of all endpoints of the closed line segment, and the text description data or the set multi-level list data and the multi-level list selection data (i.e. the standard answer). The question will be saved as the image itself, the question description and the answer part. Multiple box selections and closed polylines can be added to one image material, and each can have a text description or multi-level list data added.

[0066] For video materials, we provide frame selection marks, text annotation tools, and multi-level list setting tools for each frame. The format is consistent with the image material tools, and an additional video range marking tool is provided to mark a section of video from the start of a certain frame to the end of a certain frame in the video, and provide text annotations. After making the questions, the answers in the examination system are saved as the time of the marked frame or a section of video in the video, and the corresponding frame selection start and end coordinates data, as well as text description data or set multi-level list data and multi-level list selection item data (i.e. standard answers). The questions will be saved as the video material itself, the question description, and the answer part. For each frame of video material, it is allowed to add multiple frame selections and closed polylines, each of which can be added with text descriptions or multi-level list data.

[0067] For text materials, the diversified examination system provides a string marking tool for selecting any length of position in the text material and an annotation tool for the selected string. In addition, it also has a tool for creating multi-level list selection annotations. The answer is saved as the marked string position data and the annotation content text data or the set multi-level list data and the multi-level list selection item data (i.e. the standard answer). The question will be saved as the text material itself, the question description and the answer part. Each text material allows multiple strings to be marked, and each marked string can be added with a text description or a multi-level list data.

[0068] For audio materials, text annotation tools for any audio time point and time period are provided, and multi-level list setting tools are provided. In the examination system, the answers are saved as all the marked time points and time period data and the corresponding text annotation data or the set multi-level list data and multi-level list selection item data (i.e., standard answers). The questions will be saved as the text material itself, the question description, and the answer part. Through the above method, based on the operation of the four materials by the tools provided by the diversified examination system, it is possible to achieve the examination needs of graphics, videos, texts, and audios for various industries, set knowledge points such as equipment failures and anomalies in images, correct operations and illegal operations in videos, understanding of key information in texts, and assessments of clear expressions and bad attitudes in audio, etc.

[0069] Finally, the integration result is stored in the database alone or the integration result and the standard answer are integrated again and stored in the database to obtain the first test question bank and the second test question bank.

[0070] S102: When a first test question bank is generated based on each test question, the user test module extracts a number of first target questions from the first test question bank in response to a question selection instruction, and displays each first target question to a target user; the standard answer generation module generates a first standard answer for each first target question based on the first answer submitted by the target user for each first target question; the sample library construction module constructs a first sample library based on various types of question materials, each first standard answer and each first answer.

[0071] Specifically, before the exam, the diversified examination system is used to extract examination questions, determine the list of candidates, and determine six parameters: the number of examination questions, examination time, passing score, score calculation threshold, and whether to shuffle the order of questions. The examination questions are then distributed, and the candidates begin to answer the questions based on the client or web page of the diversified examination system.

[0072] The user test module of the diversified test system in this application displays questions in sequence according to the order of the question bank in sequence or randomly shuffled. The questions are displayed as saved images, videos, texts, audio materials and corresponding question descriptions. During the answering process, tools corresponding to all types of materials in step one and multi-level list option selection tools when a multi-level list is set are provided for answering. After answering a question, click the "Next Question" button to move on to the next question, and answer the questions in a cycle. After answering all the questions, you can submit the answers manually or the system will automatically collect the answers to all questions at the set test time.

[0073] After the answer is submitted or collected by the system, according to different question categories, the image category saves the corresponding question number and the starting and ending coordinates of the answer box, the coordinate data of all endpoints of the closed line segment, as well as text description data and multi-level list selection item data; the video category saves the corresponding question number and the time of the marked frame or a video segment in the video, the starting and ending coordinates of the box, the coordinate data of the end point, the coordinate data of all endpoints of the closed line segment, as well as text description data and multi-level list selection item data; the text category saves the corresponding question number and the marked string position data of the answer, the annotation content text data, and multi-level list selection item data; the audio category saves the corresponding question number and all marked time points and time period data of the answer, the corresponding text annotation data, and multi-level list selection item data.

[0074] When generating the first test question bank based on each test question, since the first test question bank only contains test questions but not standard answers, it is necessary to generate the first standard answer for each first target question based on the first answer submitted by the target user for each first target question through the standard answer generation module. Specifically, according to the answer results of all target users, a variety of standard answer generation methods are provided, coordinates, time points, string position data, truncated mean, Z score method and other robust statistical methods are provided, and the answers agreed by most users are calculated as standard answers. For text data and multi-level list selection item data, after the coordinates, time points, and string position data are calculated, the majority of the answer results are selected as the standard answer.

[0075] The sample library construction module constructs the first sample library required for artificial intelligence algorithm training based on various types of question materials, standard answers and answer types.

[0076] S103: When a second test question bank is generated based on each test question and each standard answer, the user test module responds to the question selection instruction, extracts a number of second target questions from the second test question bank, and displays each second target question to the target user; the sample library construction module constructs a second sample library based on various types of question materials, standard answers to each test question and second answers submitted by the target user to each second target question.

[0077] Specifically, in this step, since the second test question bank generated based on each test question and each standard answer includes not only the test questions but also their standard answers, the standard answers of each test question can be directly retrieved from the second test question bank. The workflow of the user test module and the sample library construction module can refer to step S102.

[0078] In an alternative embodiment, see Figure 2 As shown, Figure 2A flowchart of a diversified examination method provided by the first embodiment of the present invention is shown, wherein the system further includes a score calculation module and a score display module, and the method further includes steps S201 to S202:

[0079] S201: The score calculation module calculates the test score of the target user based on the answers submitted by the target user to each target question and the standard answers to each target question.

[0080] Specifically, after the test is over and the corresponding answers are saved, the score calculation module is used to calculate the score. First, the score calculation threshold parameter is used for judgment. The threshold contains two sub-parameters. One is the percentage value L, which is used to judge the difference between the candidate's answer and the standard answer of the question. When the image overlap is greater than L, it will be recorded as 100%. The audio is i=(60-c) / 60, where c is the absolute value of the difference between the audio mark point time of the candidate's answer and the standard answer mark point time. When i is greater than L, c is recorded as 0. One is the character count threshold; the character count threshold is used to judge the text description data in the answer. Once the number of character strings in the text description exceeds the threshold, the text description is converted to manual scoring and recorded as R0; in other cases, the system automatically calculates the score. For image types, when the answers are all drawn with the box selection tool, the percentage of overlap between the candidate's answer and the standard answer graphic will be calculated for each question, recorded as the graphic score T1, and then the text description or list options will be automatically compared to see if they are consistent with the standard answer. If they are consistent, it will be recorded as "yes", and if they are inconsistent, it will be recorded as "no". The result is recorded as T2, and finally T1 and T2 are retained as is the evaluation result of this question; for the video type, for each single frame, the time of the frame where the candidate's answer is marked with the standard answer in the video is automatically compared, and the time difference S0 is calculated; for each question, the percentage of the degree of overlap between the candidate's answer and the standard answer graphic is calculated, recorded as the graphic score S1, and then the text description or list options are automatically compared to see if they are consistent with the standard answer. If they are consistent, it is recorded as "yes", and if they are inconsistent, it is recorded as "no", and the result is recorded as S2. Finally, S0, S1 and S2 are retained as the evaluation result of this question; the starting frame and ending frame of a non-single-frame video are regarded as two single frames and calculated twice according to the above single-frame calculation method. For the text category, first calculate the number of the starting position, length, number, text description or list options of the string marked in the candidate's answer that are all consistent with the standard answer, record it as W1, then for the text description or list options that are completely consistent with the standard answer, but the starting position, length, and number of the marked string are not completely consistent with the standard answer, calculate the number of string overlaps divided by the number of corresponding standard answer strings, take two decimal places, add up the decimals calculated in all such cases, record it as W2, W1+W2 is recorded as W0, and finally retain W0 as the evaluation result of this question; for the audio category, select the text description or list options of the audio marked in the candidate's answer that are completely consistent with the standard answer, then calculate the difference between the starting position and end position time of the marked audio point or audio segment and the corresponding standard answer, each of which is recorded as Y0, and finally retain Y0 as the evaluation result of this question. The final test score table for one candidate is as follows:

[0081]

[0082] Among them, Tn (n is a natural number 1, 2, 3, 4, etc.) represents the question number. S represents the graphic score, S0 is the degree of overlap of the time points unique to the video type, S1 is the degree of overlap of the mark boxes, and S2 is the degree of text mark matching. -n (n is a natural number 1, 2, 3, 4, etc.) represents the different mark boxes in a question. If there are three candidate answer marks or three standard answer marks in a question (whichever is more), then there will be T1S1-1 (the graphic overlap of the first box of question 1), T1S1-2 (the graphic overlap of the second box of question 1), and T1S1-3 (the graphic overlap of the third box of question 1). -nn represents different marked frames in different frames of video questions. If there are three frames with answers or standard answer marks in a question, and there are two answers or standard answer marks in the second frame, then there will be T1S0-1 (time matching of the first frame of the first question), T1S1-1 (graphic overlap of the first frame of the first question), T1S2-1 (text answer matching of the first frame of the first question), T1S0-2 (time matching of the second frame of the first question), T2S1-21 (graphic overlap of the first frame of the second frame of the first question), T2S1-22 (graphic overlap of the second frame of the second frame of the first question), etc. K0 is manual scoring. "T" represents the number of questions, "-" represents the number of test points in this question, S, W, and Y represent different parts of the score, S is for graphics and video, W is for text, and Y is for voice.

[0083] The test score table can be exported to Excel, and also supports freely setting the passing boundary values ​​for each type of test questions and the weight of each column in the system to calculate the final score.

[0084] S202: The score display module displays the test scores.

[0085] Specifically, the results are announced through the results display module and the score query function for each question is provided. If the examinees have objections to the answers, they can correct the questions through the question correction module of the diversified examination system. After the error correction function is used, the standard answer will be edited again and submitted in step S101 according to the different types of materials. After submission, the two answers can be compared and selected. After the selection is confirmed, the examination question bank of step S101 will be updated. The purpose of this step is to further ensure the examination effect and ensure the correctness of the examination question bank.

[0086] Currently, there are three ways to test and evaluate employees in various industries. One is an online exam with multiple-choice, multiple-select, and true / false questions. Another is an offline exam that includes short-answer and drawing questions in addition to multiple-choice, multiple-select, and true / false questions. The third is a practical exam using a special simulation exam system or simulation equipment that is consistent with industry applications. For the first two methods, the cost is low and the implementation is simple. However, since many actual scenarios are translated into written descriptions, the exam results are not satisfactory, and it is easy for employees to memorize the questions and interfere with the answers, failing to achieve the purpose of improving personnel skills and evaluating industry experience. For the third method, the cost is high, and it does not meet the conditions for large-scale evaluation. It can only be tested in batches and on a small scale. There is an urgent need for an effective system in the exam evaluation of various industries to improve the quality of exam evaluation and truly reflect personnel skills and industry experience while achieving low-cost large-scale exams. The diverse exam system of this application can intuitively and fully evaluate the industry knowledge and experience of the examinees through the above steps, which is beneficial to improving the quality of exam testing for enterprises in various industries.

[0087] In an alternative embodiment, constructing the first sample library based on various types of question materials, various first standard answers, and various first response answers includes: constructing a first basic sample library based on various types of question materials and various first standard answers; constructing an enhanced sample library based on various types of question materials and various first response answers.

[0088] Constructing the second sample library based on various types of question materials, the standard answers of each exam question, and the second response answers submitted by the target user for each second target question includes: constructing a second basic sample library based on various types of question materials and the standard answers of each exam question; constructing a second enhanced sample library based on various types of question materials and various second response answers.

[0089] Specifically, the basic sample library construction module constructs the first basic sample library / second basic sample library required for artificial intelligence algorithm training based on various types of question materials and each standard answer.

[0090] Furthermore, for image types, the saved image materials are used as unlabeled samples, the starting coordinates and ending coordinates of the selected area, and all the endpoint coordinates of the closed line segments are used as the boundaries for image annotation such as object detection and image segmentation, and the text description data or selected list data are used as the labels for image annotation such as classification, object detection, and image segmentation.

[0091] The video category uses saved video materials as unlabeled samples. For single-frame annotation, the time location of the marked frame in the video material is used for extraction. The starting point coordinates and end point coordinate data of the box selection and the coordinate data of all endpoints of the closed line segment are used as the boundaries of image annotation such as target detection and image segmentation. The text description data or the selected list data are used as labels for video annotation such as video classification and target detection. For the construction of sample libraries such as continuous target tracking, target detection, and behavior annotation, the entire video is extracted based on the start and end frames annotated by the video range marking tool, and then annotated using the automatic annotation algorithm.

[0092] The text category uses the saved text material as the unlabeled sample, and the marked string position data and annotation content text data are used as the basis for determining the annotated text and labels in the text annotation. The audio category uses the saved audio material as the unlabeled sample, and the marked time point, time period data and corresponding text annotation data are used as the basis for segmentation and labeling of audio classification, recognition and other annotations.

[0093] It should be noted that for the marking of video segments, after the marking of the start frame and the end frame is completed, there is currently a method of using artificial intelligence algorithms to automatically mark all the intermediate frames. In addition, there is also a method of using artificial intelligence algorithms to perform natural language analysis and classification of title descriptions to further make sample classification and collection more efficient, but the premise of these two methods is also based on the present invention, and considering that these two points are not very original, the two methods are not included. This application should also include protection for derivative schemes based on such methods.

[0094] The basic sample library construction module constructs a first enhanced sample library / a second enhanced sample library based on various types of question materials and various answers.

[0095] Furthermore, the enhanced sample library converts the qualified answers into sample files. It is worth noting that even if the overall score of the reference candidate is unqualified, each qualified question is also converted. Finally, an enhanced sample library is formed. Except for the manually scored questions, most of the other sample buckets in the enhanced sample library are similar to the features contained in the corresponding basic samples, but this is still of great value for the training of artificial intelligence models.

[0096] In an alternative embodiment, see Figure 3 As shown, Figure 3 A flowchart of a question bank checking method provided in the first embodiment of the present invention is shown, wherein the system further includes a question bank disclosure module and a question bank checking module, and the method further includes steps S301 to S302;

[0097] S301: The question bank disclosure module discloses each test question and its standard answer in the second test question bank.

[0098] S302: The question bank checking module checks each test question and its standard answer.

[0099] Specifically, the question bank disclosure module will make the prepared questions and answers public, and all internal personnel of the organization for which the questions are intended will study and check the questions and answers. If errors are found, the questions can be corrected through the question correction module of the diversified examination system. After the error correction function is used, it will return to step S101 according to the different types of materials, and the answers will be edited and submitted for the second time. After submission, the two answers can be compared and selected. After the selection is confirmed, the question bank of step S101 will be updated, and the basic sample library will be updated synchronously. The purpose of this step is to ensure that the answers to the questions are correct and the sample library is constructed accurately. After the test questions are made public, all candidates entered by the administrator can log in to the examination system to study and practice. The learning process displays all selected question banks and answers. The basic process of the practice process is the same as the subsequent steps. After the test, the calculation list and score are directly given, and the correct enhanced samples are still collected. Feedback on wrong answers is supported.

[0100] In an alternative embodiment, see Figure 4 As shown, Figure 4 A schematic diagram of a display interface of a diversified examination system provided in Example 1 of the present invention is shown, wherein the display interface of the diversified examination system includes a question display area 01, a question number display area 02, a tool display area 03, a material display area 04 and an answer input area 05, wherein the question display area is used to display each target question, the question number display area is used to display the question number of each target question, the material display area is used to display the material of each question, the tool display area is used to display the tool corresponding to the material of each question, and the answer input area is used to input the answer to each target question.

[0101] Specifically, the question display area 01 displays the content of each target question. For example, the question is: Please select all the products with problems in the picture and explain (select) what the problem is in the text description). The question number display area 02 displays the question number of each target question. The completed questions are displayed in green, the unanswered questions are displayed in gray, the correct answers are displayed in green during practice, and the incorrect answers are displayed in red. For example, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16. The tool display area 03 displays the tools corresponding to each question material, and the tools that cannot be used for the materials are displayed in gray. For example, the box selection tool, the polyline tool, set as the starting point, set as the end point, and mark the text. The material display area 04 displays the materials of each question, and the audio is not displayed. The answer input area 05 includes a text description adding area and a hierarchical list option area. Any number of characters can be added to the text description adding area, and the hierarchical list option area can be edited and added when making a question bank. For example, the first level classification, the first level classification and the second level classification, the first level classification includes the first option and the second option, and the second level classification includes the first option and the second option. In addition, see Figure 5 As shown, Figure 5 A schematic diagram of a display interface of a second diversified examination system provided in the first embodiment of the present invention is shown, and the display interface also includes a time display area 06 and an examination operation area 07. The time display area 06 displays the remaining time of the examination. For example, the remaining time of the examination is: 60 minutes. The examination operation area 7 displays the operation controls for the examination. For example, the previous question, the next question and the paper submission controls.

[0102] Here we explain the functional implementation of the diversified examination system provided by this application. Figure 6 As shown, Figure 6A schematic diagram of a diversified examination system function implementation provided by the first embodiment of the present invention is shown, wherein the diversified examination system function implementation is divided into a login part, a question making and examination part, a database and a sample part. The login part is for the question making personnel to enter the personnel information for personnel identity verification. The question making and examination part is provided by the client or web page end with a question description tool (only provided to the question making personnel), including picture tools, video tools, text tools and audio tools. Based on the above tools and database, a question bank is made or, based on the above tools, the examination questions are selected for the examinee to take the examination. The database includes picture information, video information, text information, audio information, multi-level lists and personnel information. Picture information includes picture material (picture format), answer part (JOSN, xml and other formats) and question description (text format). Video information includes video material (video format), answer part (JOSN, xml and other formats) and question description (text format). Text information includes text material (TXT format), answer part (JOSN, xml and other formats) and question description (text format). Audio information includes audio material (audio format), answer part (JOSN, xml, etc.) and test description (text format). Multi-level lists contain text and hierarchical data. Personnel information is imported by the system administrator, mainly dictionary data, including test number, name, ID number, affiliated unit and organization, etc. Samples are basic samples and enhanced samples after removing the test description, which are used for artificial intelligence algorithm training.

[0103] Based on the functional realization of the above diversified examination system, see Figure 7 As shown, Figure 7 A schematic diagram of a diversified examination process provided by the first embodiment of the present invention is shown, wherein the system administrator adds the account information and permissions of the examiner and the question bank manager, and imports the list of candidates. The examiner or the question bank manager uses the diversified examination system to create questions. The uploaded materials include image question materials, video question materials, audio question materials, and text question materials. For image question materials, set the question description, multi-level list, set the box selection range answer and the text description or multi-level list selection answer corresponding to each box selection. For video question materials, set the question description, multi-level list, set the video frame time point or time period answer, each frame box selection range answer and each box selection corresponding to the text description or multi-level list selection answer. For audio question materials, set the question description, multi-level list, voice time point, time period and the text description or multi-level list selection answer corresponding to each time point or time period. For text question materials, set the question description, multi-level list, string starting position, length answer and the text description or multi-level list selection answer corresponding to each string. A question bank is formed based on the above material information.

[0104] After the question bank is formed, the examination is conducted using a diversified examination system. The examination process is that the examiner logs in and initiates the examination, extracts questions from the question bank, determines the list of candidates, and determines five parameters: the number of examination questions, examination duration, passing score, score calculation threshold, and whether to shuffle the order of questions. Each candidate logs in and starts answering.

[0105] During the exam or practice, for image questions, candidates observe the image according to the requirements of the question, set the selection range at the part that they think meets the requirements, enter the corresponding text description or select the answer they think is correct from the multi-level list. For video questions, candidates browse the video according to the requirements of the question, pause at the position that they think meets the requirements, set whether it is the starting point or end point of the video segment, set the selection range, enter the corresponding text description or select the answer they think is correct from the multi-level list. For audio questions, candidates listen to the audio according to the requirements of the question, pause at the position that they think meets the requirements, set whether it is the starting point or end point of the audio segment, enter the corresponding text description or select the answer they think is correct from the multi-level list. For text questions, candidates read the text according to the requirements of the question, start marking a free-length string at the position that they think meets the requirements, enter the corresponding text description or select the answer they think is correct from the multi-level list.

[0106] See also Figure 8 As shown, Figure 8 The schematic diagram of the second diversified examination process provided by the first embodiment of the present invention is shown. After the examinee submits the answer through the above method, it is determined whether the text description part of the answer exceeds the threshold. If so, the system prompts the examiner to manually judge the score, and the final score is calculated according to the weights and rules set by the examiner and the score is announced; otherwise, the standard answer is obtained according to the answer of the examinee. After the standard answer is determined, it is divided into two branches. The first branch is to make the standard answer public as part of the question bank, realize the public question bank and calibrate it internally, and then collect the question materials and answer parts to form a basic sample library. The second branch is that the system automatically calculates the score, gives a detailed list, and then determines whether each test point in each question is qualified. If so, the material and answer parts are collected to form a reinforced sample library. After the system automatically calculates the score and gives a detailed list, it can also calculate the final score according to the weights and rules set by the examiner to announce the score, and then determine whether the answer to the test question is wrong. If so, return to the first branch process to correct the answer and re-form the basic sample library.

[0107] The diversified examination methods provided by this application can meet the examination needs of the industrial industry and other industries. In the power, petroleum, smelting and other industries, the picture type can examine the ability of practitioners to identify the operating status and defect problems of the equipment they operate and maintain, and examine the ability to identify and deal with various phenomena in the production process; it can also examine the ability to correctly operate equipment, tools, and systems. The audio type can examine the identification and judgment of personnel on abnormal sounds of equipment and production processes. For example, in the trade industry, the purchase and quality inspection personnel of products in the industry need to be able to distinguish the quality of products. The usual assessment method is to use a text test to describe quality problems and contract order problems or print relevant pictures, and the reference personnel give written answers. The picture test based on the diversified examination system can further examine the personnel's recognition ability, requiring personnel to select the problem product, or point out the location of specific problems in the contract or order, and then describe and answer; in addition, audio questions can also be used to examine business personnel's ability to judge customer needs and emotions, requiring personnel to find and mark the key needs of customers and the key time points and time periods of emotional fluctuations in an audio clip. For example, in the medical industry, based on a diversified examination system, the image category can examine the doctor's ability to analyze and diagnose various medical images. It can also place pictures of medical devices and instruments, and require personnel to circle the correct device to be selected and the corresponding button to be operated for a certain operation step, so as to examine whether the personnel know how to operate. In addition, because in step one, the box selection and text annotation of the description are in order, it is possible to examine whether a series of operations of the personnel are correct in one picture. The video category can examine the doctor's sensitivity to some violations during the operation process. The audio category can examine whether the doctor can make a correct judgment based on the patient's description of his own symptoms. In addition, the existing artificial intelligence sample annotation industry can also use this system to examine the sample annotation ability of personnel. Using this system for annotation, among a large number of test questions without answers, randomly insert questions with standard answers in the question bank to achieve the purpose of spot-checking the annotation quality of the annotation personnel.

[0108] Embodiment 2

[0109] See also Fig. 9 As shown, Fig. 9 The structure diagram of a diversified examination system provided by the second embodiment of the present invention is shown, wherein the system includes a question bank generation module 901, a user examination module 902, a standard answer generation module 903 and a sample library construction module 904;

[0110] The question bank generating module is used to obtain various types of question materials, generate a number of test questions and standard answers for each test question based on the various types of question materials, generate a first test question bank based on each test question, or generate a second test question bank based on each test question and its standard answer;

[0111] When generating a first test question bank based on each test question, the user test module is used to extract a number of first target questions from the first test question bank in response to a question selection instruction, and display each first target question to a target user; the standard answer generation module is used to generate a first standard answer to each first target question based on the first answer submitted by the target user to each first target question; the sample library construction module is used to construct a first sample library based on various types of question materials, each first standard answer and each first answer;

[0112] When a second test question bank is generated based on each test question and each standard answer, the user test module is used to extract a number of second target questions from the second test question bank in response to a question selection instruction, and display each second target question to the target user; the sample library construction module is used to construct a second sample library based on various types of question materials, standard answers to each test question and second answers submitted by the target user for each second target question.

[0113] In an optional embodiment, the system further includes a score calculation module and a score display module;

[0114] The score calculation module is used to calculate the test score of the target user based on the answers submitted by the target user to each target question and the standard answers to each target question;

[0115] The score display module is used to display the test scores.

[0116] In an optional implementation, a first sample library is constructed based on various types of question materials, first standard answers, and first answer answers, including:

[0117] Constructing a first basic sample library based on various types of question materials and first standard answers;

[0118] Build an enhanced sample library based on various types of question materials and first answers;

[0119] The constructing of the second sample library based on various types of question materials, standard answers to each test question and second answers submitted by the target user for each second target question includes:

[0120] Constructing a second basic sample library based on various types of question materials and standard answers to each test question;

[0121] A second enhanced sample library is constructed based on various types of question materials and second answer answers.

[0122] In an optional embodiment, the system further comprises a question bank disclosure module and a question bank verification module;

[0123] The question bank disclosure module is used to disclose each test question and its standard answer in the second test question bank;

[0124] The question bank checking module is used to check each test question and its standard answer.

[0125] In an optional implementation scheme, the display interface of the diversified examination system includes a question display area, a question number display area, a tool display area, a material display area and an answer input area, wherein the question display area is used to display each target question, the question number display area is used to display the question number of each target question, the material display area is used to display the material of each question, the tool display area is used to display the tools corresponding to the material of each question, and the answer input area is used to input the answer to each target question.

[0126] Embodiment 3

[0127] Based on the same application concept, see Fig.10 As shown, Fig.10 FIG. 4 shows a schematic diagram of the structure of a computer device provided by Embodiment 3 of the present invention, wherein: Fig.10 As shown, a computer device 1000 provided in Embodiment 3 of the present application includes:

[0128] A processor 1001, a memory 1002 and a bus 1003. The memory 1002 stores machine-readable instructions executable by the processor 1001. When the computer device 1000 is running, the processor 1001 communicates with the memory 1002 via the bus 1003. When the processor 1001 is running, the machine-readable instructions execute the steps of the sample library construction method shown in the first embodiment.

[0129] Embodiment 4

[0130] Based on the same application concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the sample library construction method described in any one of the above embodiments are executed.

[0131] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0132] The computer program product for constructing a sample library provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the above method embodiment. The specific implementation can be found in the method embodiment and will not be described in detail here.

[0133] The diversified examination system provided by the embodiment of the present invention can be specific hardware on the device or software or firmware installed on the device. The system provided by the embodiment of the present invention has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the system embodiment can refer to the corresponding contents in the aforementioned method embodiment. Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0134] In the embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some communication interface, device or unit, which can be electrical, mechanical or other forms.

[0135] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] In addition, each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0137] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0138] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0139] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or perform equivalent replacements on some of the technical features thereof; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for constructing a sample library, characterized in that: Applied to a diversified examination system, the system includes a question bank generation module, a user examination module, a standard answer generation module and a sample library construction module, and the method includes: The question bank generation module obtains various types of question materials, generates a number of test questions and standard answers for each test question based on the various types of question materials, generates a first test question bank based on each test question, or generates a second test question bank based on each test question and its standard answer; When generating a first test question bank based on each test question, the user test module extracts a number of first target questions from the first test question bank in response to the question selection instruction, and displays each first target question to the target user; the standard answer generation module generates a first standard answer for each first target question based on the first answer submitted by the target user for each first target question; The sample library construction module constructs a first sample library based on various types of question materials, first standard answers and first answer answers; When a second test question bank is generated based on each test question and each standard answer, the user test module responds to the question selection instruction, extracts a number of second target questions from the second test question bank, and displays each second target question to the target user; the sample library construction module constructs a second sample library based on various types of question materials, standard answers to each test question and the second answer submitted by the target user to each second target question.

2. The method according to claim 1, characterized in that The system further includes a score calculation module and a score display module, and the method further includes: The score calculation module calculates the test score of the target user based on the answers submitted by the target user to each target question and the standard answers to each target question; The score display module displays the test scores.

3. The method according to claim 1, characterized in that The first sample library is constructed based on various types of question materials, first standard answers and first answer answers, including: Constructing a first basic sample library based on various types of question materials and first standard answers; Build an enhanced sample library based on various types of question materials and first answers; The constructing of the second sample library based on various types of question materials, standard answers to each test question and second answers submitted by the target user for each second target question includes: Constructing a second basic sample library based on various types of question materials and standard answers to each test question; A second enhanced sample library is constructed based on various types of question materials and second answer answers.

4. The method according to claim 1, characterized in that: The system further includes a question bank disclosure module and a question bank verification module, and the method further includes: The question bank disclosure module discloses each test question and its standard answer in the second test question bank; The question bank checking module checks each test question and its standard answer.

5. The method according to claim 1, characterized in that The display interface of the diversified examination system includes a question display area, a question number display area, a tool display area, a material display area and an answer input area, wherein the question display area is used to display each target question, the question number display area is used to display the question number of each target question, the material display area is used to display the material of each question, the tool display area is used to display the tools corresponding to the material of each question, and the answer input area is used to input the answer to each target question.

6. A diversified examination system, characterized in that: The system includes a question bank generation module, a user test module, a standard answer generation module and a sample library construction module; The question bank generating module is used to obtain various types of question materials, generate a number of test questions and standard answers for each test question based on the various types of question materials, generate a first test question bank based on each test question, or generate a second test question bank based on each test question and its standard answer; When a first test question bank is generated based on each test question, the user test module is used to extract a number of first target questions from the first test question bank in response to a question selection instruction, and display each first target question to a target user; The standard answer generation module is used to generate a first standard answer for each first target question based on the first answer submitted by the target user for each first target question; The sample library construction module is used to construct a first sample library based on various types of question materials, first standard answers and first answer answers; When a second test question bank is generated based on each test question and each standard answer, the user test module is used to extract a number of second target questions from the second test question bank in response to a question selection instruction, and display each second target question to the target user; the sample library construction module is used to construct a second sample library based on various types of question materials, standard answers to each test question and second answers submitted by the target user for each second target question.

7. The system according to claim 6, characterized in that The system also includes a score calculation module and a score display module; The score calculation module is used to calculate the test score of the target user based on the answers submitted by the target user to each target question and the standard answers to each target question; The score display module is used to display the test scores.

8. The system according to claim 6, characterized in that A first sample library is constructed based on various types of question materials, first standard answers and first answer answers, including: Constructing a first basic sample library based on various types of question materials and first standard answers; Build an enhanced sample library based on various types of question materials and first answers; The constructing of the second sample library based on various types of question materials, standard answers to each test question and second answers submitted by the target user for each second target question includes: Constructing a second basic sample library based on various types of question materials and standard answers to each test question; A second enhanced sample library is constructed based on various types of question materials and second answer answers.

9. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the sample library construction method according to any one of claims 1 to 5 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the sample library construction method according to any one of claims 1 to 5 are executed.

Images