A problem text processing method, device, equipment and medium
By introducing preset templates and the Bart model for problem text processing, the problems of low processing efficiency and insufficient accuracy in existing technologies are solved, and standardized processing and efficient classification of problem text are achieved, which is applicable to civil aviation security inspections.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CIVIL AVIATION FLIGHT UNIV OF CHINA
- Filing Date
- 2026-01-14
- Publication Date
- 2026-06-05
AI Technical Summary
Existing problem text processing methods are inefficient and inaccurate, and fail to form a standardized feature-category mapping relationship, resulting in low efficiency in identifying safety hazards during civil aviation security inspections.
By introducing a preset template for text preprocessing, the sentence probability value and text distribution value of the word sequence are calculated. The Bart model is used for classification. The problem category is determined by combining the first and second probability values, and a threshold rule is set to ensure accuracy.
It achieves the standardization and digitization of problem texts, improves the accuracy and efficiency of classification, and is suitable for high-load civil aviation inspection scenarios.
Smart Images
Figure CN122153630A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of language processing technology, and in particular to a method, apparatus, device and medium for processing problem text. Background Technology
[0002] Civil aviation safety is the lifeline of civil aviation development. It is necessary to regularly collect existing problems in order to conduct safety inspections. The specific inspection content includes multiple core areas such as safety management, rules and regulations, facilities and equipment, information management, personnel training, and special inspections. After collecting the problem texts, the problems need to be classified in order to further determine the categories of current major safety hazards, so as to make timely adjustments.
[0003] Existing problem text processing methods generally classify the acquired problem text directly, relying on pre-defined classification rules such as keyword matching, regular expressions, and fixed writing rules. The classification labels are simply associated with the original text and do not form a standardized feature-category mapping relationship.
[0004] However, existing problem text processing methods suffer from low processing efficiency and inaccurate results. Summary of the Invention
[0005] This application provides a method, apparatus, device, and medium for processing problematic text, in order to solve the problems of low processing efficiency and inaccurate processing results in existing problematic text processing methods.
[0006] Firstly, this application provides a method for processing problem text, the method comprising: Based on the preset template, the initial question text is determined and preprocessed to obtain the first probability value and word sequence corresponding to each question category; Calculate the sentence probability value and text distribution value of each word in the word sequence, and map each word into a feature vector based on the sentence probability value and text distribution value; The feature vector is input into the target Bart model to obtain the second probability value corresponding to each problem category output by the target Bart model, and the category probability value is determined based on the first probability value and the second probability value. The target probability value with the largest value among the probability values of each category is compared with the preset threshold, and the target question category corresponding to the initial question text is determined based on the comparison results.
[0007] In some embodiments of this application, the initial question text is determined according to a preset template, including: Obtain the original question text, and extract the text from the original question text based on multiple preset templates to obtain the template question text; Determine if there is any missing text in the template question text; If so, determine the missing template corresponding to the missing text in the preset template, and output a prompt message to the user based on the missing template; Based on user feedback, the missing template is completed to obtain the complete template issue text, and the complete template issue text is determined as the initial issue text. If not, then the template question text is determined to be the initial question text.
[0008] In some embodiments of this application, the initial question text is preprocessed to obtain the first probability value and word sequence corresponding to each question category, including: The initial question text is cleaned, and the semantic similarity threshold between the cleaned initial question text and each question category is calculated to obtain multiple first probability values; Determine the word sequence corresponding to the initial question text.
[0009] In some embodiments of this application, determining the lexical sequence corresponding to the initial question text includes: The initial question text is segmented using the word segmentation component to obtain a word sequence.
[0010] In some embodiments of this application, calculating the sentence probability value and text distribution value of each word in the word sequence includes: Determine all the question statements included in the initial question text, and the total number of words in each question statement; Based on the number of times each word is repeated in each question statement and the total number of words corresponding to each question statement, determine the statement probability value corresponding to each word in each question statement; Based on the number of times each word is repeated in all question statements and the total number of question statements in the initial question text, determine the text distribution value corresponding to each word in the initial question text.
[0011] In some embodiments of this application, before inputting the feature vector into the target Bart model to obtain the second probability value corresponding to each problem category output by the target Bart model, the method further includes: Determine the standard question text and its corresponding standard category probability value, and input the standard question text into the initial Bart model to obtain the model category probability value output by the initial Bart model; Based on the standard question category and model category probability values, the model parameters of the initial Bart model are adjusted to obtain the target Bart model, so that the target Bart model can output the corresponding standard category probability value based on the input standard question text.
[0012] In some embodiments of this application, the target probability value with the largest value among the probability values of each category is compared with a preset threshold, and the target question category corresponding to the initial question text is determined based on the comparison result, including: The target probability value is determined by identifying the category with the largest probability value among all categories, and then compared with a preset threshold to obtain the comparison result. If the comparison result shows that the target probability value is greater than the preset threshold, then the problem category corresponding to the target probability value is determined as the target problem category; If the comparison result shows that the target probability value is not greater than the preset threshold, a prompt message will be output to inform the user that the current calculation is incorrect and the probability value needs to be recalculated.
[0013] Secondly, this application provides a problem text processing apparatus, the apparatus comprising: The preprocessing module is used to determine the initial question text according to the preset template, and to preprocess the initial question text to obtain the first probability value and word sequence corresponding to each question category; The calculation module is used to calculate the sentence probability value and text distribution value of each word in the word sequence, and map each word into a feature vector based on the sentence probability value and text distribution value; The input module is used to input the feature vector into the target Bart model, obtain the second probability value corresponding to each problem category output by the target Bart model, and determine the category probability value based on the first probability value and the second probability value. The comparison module is used to compare the target probability value with the highest value among the probability values of each category with a preset threshold, and to determine the target question category corresponding to the initial question text based on the comparison results.
[0014] Thirdly, this application provides an apparatus, including: a processor, and a memory communicatively connected to the processor; The memory stores the instructions that the computer executes; The processor executes computer execution instructions stored in memory to implement the method of this application.
[0015] Fourthly, this application provides a computer-readable storage medium storing program code, which, when executed by a processor, is used to implement the method of this application.
[0016] This application provides a method, apparatus, device, and medium for processing question text. The method involves: determining initial question text based on a preset template; preprocessing the initial question text to obtain a first probability value and a word sequence corresponding to each question category; calculating the sentence probability value and text distribution value of each word in the word sequence; mapping each word to a feature vector based on the sentence probability value and text distribution value; inputting the feature vector into a target Bart model to obtain a second probability value corresponding to each question category output by the target Bart model; determining the category probability value based on the first and second probability values; comparing the target probability value with the largest value among the category probability values with a preset threshold; and determining the target question category corresponding to the initial question text based on the comparison result.
[0017] In this way, by introducing structured templates and preprocessing rules, the uniformity and digitization of collected question texts can be ensured, thereby establishing a standardized processing framework, ensuring that question descriptions use a unified template, and eliminating ambiguity caused by free text. Furthermore, the Bart model is used to classify the text. The intelligent classification algorithm based on the Bart model can achieve efficient inference in a GPU-accelerated environment. Through a self-attention mechanism, it automatically extracts semantic features from the standardized question texts and sets threshold rules. By assigning first and second probability values, the final question category corresponding to the initial question text is determined comprehensively, improving the accuracy and efficiency of classification, and making it suitable for high-load civil aviation inspection scenarios. Attached Figure Description
[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0019] Figure 1 A flowchart illustrating a method for processing problematic text provided in an embodiment of this application; Figure 2 A flowchart illustrating another method for processing problematic text provided in this application embodiment; Figure 3 A schematic diagram of the structure of a problem text processing device provided in an embodiment of this application; Figure 4 This is a structural block diagram of an apparatus for performing a problem text processing method according to an embodiment of this application. Detailed Implementation
[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0021] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0022] Figure 1 This is a flowchart illustrating a method for processing problematic text provided in an embodiment of this application. Figure 1 As shown, this method for processing problem text may include the following steps: S110. Based on the preset template, determine the initial question text and preprocess the initial question text to obtain the first probability value and word sequence corresponding to each question category.
[0023] The preset template refers to a pre-defined template used to standardize the problem text. By deploying predefined field templates on the system front end, which can include four fixed fields: "problem type", "scope of impact", "key entity" and "severity level", key sentences can be extracted from the collected problem text. For example, when the collected problem text is "equipment aging caused downtime", the preset template determines that the text corresponds to "problem type: 01", "scope of impact: operation interruption", "key entity: equipment number X" and "severity level: high", thus obtaining the initial problem text, so as to achieve the standardization of problem text for subsequent problem classification.
[0024] The initial question text refers to the standardized question text obtained after processing with a preset template.
[0025] Preprocessing can be understood as transforming the initial question text into a standard format that the model can process. This can include text processing operations such as text cleaning and vector transformation, thereby converting the initial question text into a standard format that can be used for model prediction.
[0026] Problem categories refer to the types of problems that may exist in the present, such as safety management, regulations and systems, facilities and equipment, information management, personnel training and special inspections. Civil aviation safety requires frequent safety inspections. By regularly collecting relevant safety issues from staff and classifying them, we can determine the problem categories with the most potential safety hazards, so as to carry out timely safety inspections and hazard detection, thereby eliminating safety hazards.
[0027] The first probability value refers to the probability value between the initial question text and each question category. It can be obtained by calculating the semantic similarity between the initial question text and each question category, thereby obtaining multiple corresponding first probability values.
[0028] A lexical sequence refers to an ordered list formed by breaking down a continuous initial question text into a series of smallest semantic units. These lexical units can be individual words, phrases, technical terms, or symbols. For example, after word segmentation, the initial question text "Employees did not participate in the emergency drill, resulting in operational errors" generates a lexical sequence of ["employees", "not", "participate", "emergency drill", "resulting in", "operational errors"]. Each element here is a lexical unit, arranged according to the semantic order of the original text, together constituting a structured breakdown of the original text.
[0029] Based on this, a structured initial question text is obtained by using a preset template. Further preprocessing is then performed to calculate the first probability value between the initial question text and each question category, and to determine the corresponding word sequence. This allows for subsequent vector mapping of the text based on the word sequence to obtain the corresponding feature vector, thereby further determining the second probability value. Finally, based on the first and second probability values, the question category corresponding to the initial question text is determined, thus achieving the classification of the initial question text.
[0030] S120. Calculate the sentence probability value and text distribution value of each word in the word sequence, and map each word into a feature vector based on the sentence probability value and text distribution value.
[0031] Among them, the statement probability value can be understood as the TF value corresponding to the word, and the text distribution value can be understood as the IDF value corresponding to the word. The TF value (Term Frequency) refers to the frequency of a certain word in the current question description text. The calculation formula is: TF value = number of times a certain word appears in the text / total number of all words in the text. The IDF value (Inverse Document Frequency) refers to the inverse ratio of the prevalence of a certain word in the entire question database. For example, if the civil aviation question database has a total of 10,000 records (documents), and there are 100 records containing "airworthiness", then the IDF value of "airworthiness" is log(10,000 / 100) = log(100) = 2.
[0032] Feature vectors convert words in text into numerical representations that can be processed by computers, thereby preserving the semantic information of the text while adapting to the mathematical operation requirements of machine learning models.
[0033] Based on this, in practical applications, existing technologies only perform simple word segmentation and vector conversion on the text, without combining the importance of a word in a single sentence (sentence probability value) and its distribution characteristics in the overall text (text distribution value). This results in the feature vectors failing to accurately represent the core semantics of the text, thus affecting classification accuracy. The sentence probability value reflects the importance of a word in a single question description; the more times a word appears in the current text, the more representative it is of the core features of the question. The text distribution value reflects the uniqueness of a word in the entire question database; the fewer times a word appears in the text, the higher its IDF value, indicating its greater role in distinguishing different questions. By calculating the sentence probability value and text distribution value of each word in the initial question text, the segmented text is converted into a numerical vector for subsequent category prediction. Furthermore, based on the sentence probability value and text distribution value, the accuracy of the feature vectors is improved.
[0034] S130. Input the feature vector into the target Bart model to obtain the second probability value corresponding to each problem category output by the target Bart model, and determine the category probability value based on the first probability value and the second probability value.
[0035] The target Bart model refers to the Bart model obtained after specific model training, which can be used to output the probability values between the text and each question category based on the feature vector of the text. The Bart model is based on the Transformer architecture and integrates the design of "bidirectional encoder" and "autoregressive decoder". It can perform bidirectional semantic encoding on the input text, capture complete contextual information, and use an autoregressive Transformer structure to generate output results in sequence based on the semantic features output by the encoder.
[0036] The second probability value refers to the probability value between the feature vector of the question text output by the Bart model and each question category. The first probability value is determined by calculating semantic similarity, while the second probability value is the probability value output by the target Bart model after processing the feature vector corresponding to the initial question text. By combining the first and second probability values, the accuracy of question text classification is improved.
[0037] The category probability value is the complete probability value between the initial question text and each question category, obtained by combining the first probability value and the second probability value. For example, it can be determined based on the sum of the first probability value and the second probability value, or it can be a weighted probability value determined by assigning different weight values to the first probability value and the second probability value respectively.
[0038] Based on this, the target Bart model is obtained by training the corresponding Bart model. The feature vector corresponding to the initial question text is then processed according to the target Bart model to output the second probability value between the initial question text and each question category. The first and second probability values are then combined to determine the category probability value between the initial question text and each question category.
[0039] S140. Compare the target probability value with the largest value among the probability values of each category with the preset threshold, and determine the target question category corresponding to the initial question text based on the comparison results.
[0040] The preset threshold is a pre-defined threshold used to determine whether the category probability value of the initial question text meets the actual requirements.
[0041] The target question category is the question category corresponding to the initial question text among multiple preset question categories.
[0042] Based on this, by first comparing the numerical values of the probability values of each category, the target probability value with the largest value is determined. Then, the target probability value is compared with the preset threshold. If the target probability value is greater than the preset threshold, it indicates that the current calculation is correct. Therefore, the question category corresponding to the target probability value can be determined as the question category corresponding to the initial question text, thereby achieving text classification. If the target probability value is not greater than the preset threshold, it indicates that the current calculation may have an error. The target probability value obtained in this calculation cannot be used to determine the question category corresponding to the initial question text. In this case, a prompt message needs to be output to prompt the user to recalculate.
[0043] Based on the feasible implementation of S110 described above, this application further provides the steps for determining the initial problem text according to a preset template, including: Obtain the original question text, and extract the text from the original question text based on multiple preset templates to obtain the template question text; Determine if there is any missing text in the template question text; If so, determine the missing template corresponding to the missing text in the preset template, and output a prompt message to the user based on the missing template; Based on user feedback, the missing template is completed to obtain the complete template issue text, and the complete template issue text is determined as the initial issue text. If not, then the template question text is determined to be the initial question text.
[0044] The original question text refers to the question text that has been collected and has not undergone any secondary text processing.
[0045] Text extraction refers to information extraction operations based on structured templates, which are used to accurately extract key information that matches preset template fields from unstructured raw text, thereby achieving standardized transformation of problem descriptions.
[0046] Template question text is the question text obtained after standardization through a preset template.
[0047] Missing text refers to text in which no corresponding information was extracted. In practical applications, when collecting questions from relevant staff, the staff may not provide sufficient descriptions of the questions, resulting in incomplete information. For example, if the preset template requires extracting whether the current question has a serious impact, but the staff's description does not contain the corresponding description, then the template will contain blank missing text in the question text extracted from the template.
[0048] The missing template is the template information corresponding to the missing text.
[0049] The prompt message is used to guide the user to complete the text.
[0050] Therefore, in practical applications, there may be situations where the key semantic information of the text is incomplete due to inaccurate descriptions by staff or free-flowing language. If the missing text information is not supplemented, the subsequent model will be unable to correctly classify the text as a problem. By further judging the template problem text obtained after template extraction, it is possible to determine whether there is missing text information in the template problem text. If so, it is necessary to further output prompt information to the user so that the user can operate according to the user's feedback. For example, the user can describe the problem again according to the missing template corresponding to the missing text information, thereby obtaining the complete template problem text, so that the template problem text can be identified as the initial problem text.
[0051] Based on the feasible implementation of S110 described above, this application further provides a method for preprocessing the initial question text to obtain the first probability value and word sequence corresponding to each question category, including the following steps: The initial question text is cleaned, and the semantic similarity threshold between the cleaned initial question text and each question category is calculated to obtain multiple first probability values; Determine the word sequence corresponding to the initial question text.
[0052] Text cleaning is used to remove irrelevant, redundant, and interfering information from the initial problem text, thereby correcting formatting errors, standardizing expression, and transforming messy original text into clean and consistent intermediate data. For example, it can remove stop words (such as "of" and "and") and special characters from the initial problem text.
[0053] Based on this, text cleaning is performed on the initial question text to correct the text format and remove redundancy, so as to further calculate the semantic similarity between the initial question text and each question category, thereby obtaining the first probability value.
[0054] Based on the feasible implementation of S110 described above, this application further provides steps for determining the lexical sequence corresponding to the initial question text, including: The initial question text is segmented using the word segmentation component to obtain a word sequence.
[0055] The word segmentation component is a tool used to split the complete initial question text into the smallest independent semantic units according to semantic logic and output an ordered sequence of word units. For example, it can be the Jieba word segmentation tool. Jieba has a built-in Chinese common dictionary (containing tens of thousands of common words and phrases) by default, and also supports users to add custom dictionaries. Therefore, professional terms related to civil aviation safety can be added to the custom dictionary in advance to ensure that professional terms are not split.
[0056] Based on this, by determining the word segmentation component, the initial question text can be segmented according to the word segmentation component in order to obtain the corresponding word sequence.
[0057] Based on the feasible implementation of S120 described above, this application further provides steps for calculating the sentence probability value and text distribution value of each word in the word sequence, including: Determine all the question statements included in the initial question text, and the total number of words in each question statement; Based on the number of times each word is repeated in each question statement and the total number of words corresponding to each question statement, determine the statement probability value corresponding to each word in each question statement; Based on the number of times each word is repeated in all question statements and the total number of question statements in the initial question text, determine the text distribution value corresponding to each word in the initial question text.
[0058] Among them, the problem statement refers to the individual statements included in the initial problem text. It is a sentence or continuous expression that carries specific problem information in the initial problem text. It can be understood as the smallest text fragment that can independently express a complete security problem. For example, if the initial problem text is "Equipment aging often leads to downtime, which is highly dangerous", then the problem statements included are "Equipment aging often leads to downtime" and "highly dangerous".
[0059] Lexical totals refer to the total number of lexical units included in each question statement.
[0060] The repetition count refers to the number of times each word appears in each question statement. For example, in the question statement "Equipment is aging and shut down and the equipment failure has not been repaired", the word "equipment" appears twice, and the corresponding repetition count is 2.
[0061] Based on this, by determining all the question statements included in the initial question text and the total number of lexical units in each question statement, the statement probability value corresponding to each lexical unit in each question statement and the text distribution value corresponding to each lexical unit in the entire initial question text can be determined according to the number of times each lexical unit is repeated in each question statement and the total number of lexical units corresponding to that question statement.
[0062] Based on the feasible implementation of S130 described above, this application further provides the following steps before inputting the feature vector into the target Bart model and obtaining the second probability value corresponding to each problem category output by the target Bart model: Determine the standard question text and its corresponding standard category probability value, and input the standard question text into the initial Bart model to obtain the model category probability value output by the initial Bart model; Based on the standard question category and model category probability values, the model parameters of the initial Bart model are adjusted to obtain the target Bart model, so that the target Bart model can output the corresponding standard category probability value based on the input standard question text.
[0063] The standard question text and its corresponding standard category probability value refer to the pre-determined question text and category probability value used for model training. The standard question text is a standardized pre-processed labeled sample text, which is the input data for model training. The standard category probability value is a probabilistic representation of the real category labels labeled by humans, which is the supervision signal for model training and is used to guide the model to learn the mapping relationship from question text to category probability value.
[0064] Based on this, standard question texts and corresponding standard category probability values are determined using historical datasets for model training. By inputting standard question texts into the initial Bart model and adjusting the model parameters according to the probability values output by the model, a target Bart model is obtained, which can then be used in practical applications to output the probability values between question texts and various question categories.
[0065] Based on the feasible implementation of S140 described above, this application further provides a target probability value and a preset threshold that are the largest among the probability values of each category, and determines the target question category corresponding to the initial question text based on the comparison results, including the following steps: The target probability value is determined by identifying the category with the largest probability value among all categories, and then compared with a preset threshold to obtain the comparison result. If the comparison result shows that the target probability value is greater than the preset threshold, then the problem category corresponding to the target probability value is determined as the target problem category; If the comparison result shows that the target probability value is not greater than the preset threshold, a prompt message will be output to inform the user that the current calculation is incorrect and the probability value needs to be recalculated.
[0066] Based on this, after determining the target category probability value with the largest value among the category probability values between the initial question text and each question category, it is necessary to further determine whether the target category probability value is greater than a preset threshold. If it is greater, it indicates that the current calculation is correct, so the question category corresponding to the target category probability value can be determined as the target question category corresponding to the initial question text, thus achieving the classification of the initial question text. If it is not greater, it indicates that there is an error in the current calculation of the category probability value, and it needs to be recalculated. Therefore, a prompt message can be output so that the user can recalculate the probability values between the initial question text and each question category according to the prompt message.
[0067] Please refer to Figure 2 , Figure 2 A flowchart illustrating another method for processing problematic text provided in this application embodiment; as follows: Figure 2 As shown, the original description is obtained and processed by the standardization module. The original description is then cleaned, segmented, and vectorized into a 256-dimensional feature vector, which is used as standardized data input to the subsequent classification module. The input vector is then forward-propagated according to the BART model to calculate the probability value of each category. The problem category (such as "facilities and equipment") is determined based on the maximum probability, and a classification report (including confidence score) is generated.
[0068] In existing technologies, inspectors lack standardized descriptions of issues discovered during inspections (e.g., arbitrary text formatting, missing keywords), leading to inaccurate issue categorization and difficulty in identifying recurring or frequently occurring problems. Specific deficiencies include a lack of structure, with issue descriptions often being free-form text without standardized templates (such as fixed fields or codes), making data digitization and comparison difficult; non-standardized descriptions hinder effective data mining, for example, the inability to automatically associate unsafe events, requiring manual secondary processing, which is time-consuming and error-prone; low processing efficiency, reliance on manual operation, and high costs; furthermore, issue classification requires manual completion, a time-consuming process prone to errors due to fatigue; the lack of advanced algorithms for intelligent summarization or classification results in slow processing speed, and the absence of efficient computational models, insufficient standardization of issue processing, and a lack of automatic classification mechanisms (such as text processing based on Transformer or BART), making it impossible to extract key features from issue descriptions.
[0069] In some embodiments of this application, the original text is extracted and completed using a pre-set template to obtain initial question text. This initial question text is then cleaned, and semantic similarity thresholds between the cleaned initial question text and each question category are calculated to obtain multiple first probability values. Further text segmentation is performed to obtain corresponding word sequences, thereby determining the sentence probability value and text distribution value corresponding to each word in the word sequence. This enables vector mapping of the word elements to obtain corresponding feature vectors. The target Bart model outputs probability values between the input feature vectors and each question category to obtain second probability values. These first and second probability values are then combined to calculate the final category probability value. The target probability value, which is the largest among these category probability values, is compared with a preset judgment standard threshold, and the final target question category corresponding to the question text is determined based on the comparison result.
[0070] Thus, by introducing structured templates and preprocessing rules, the uniformity and digitization of collected question texts are ensured, establishing a standardized processing framework to ensure that question descriptions use a unified template and eliminate ambiguity caused by free text. Furthermore, the Bart model is used for text classification. The intelligent classification algorithm based on the Bart model can achieve efficient inference in a GPU-accelerated environment. It automatically extracts semantic features from the standardized question text through a self-attention mechanism and sets threshold rules. By assigning a combined first and second probability value, the accuracy and efficiency of classification are improved, making it suitable for high-load civil aviation inspection scenarios. Figure 3 This is a schematic diagram of the structure of a problem text processing device 300 provided in an embodiment of this application. For example... Figure 3As shown, the problem text processing device 300 includes: a preprocessing module 310, a calculation module 320, an input module 330, and a comparison module 340; wherein: The preprocessing module 310 is used to determine the initial question text according to the preset template, and to preprocess the initial question text to obtain the first probability value and word sequence corresponding to each question category; The calculation module 320 is used to calculate the sentence probability value and text distribution value of each word in the word sequence, and to map each word into a feature vector based on the sentence probability value and text distribution value. The input module 330 is used to input the feature vector into the target Bart model, obtain the second probability value corresponding to each problem category output by the target Bart model, and determine the category probability value based on the first probability value and the second probability value. The comparison module 340 is used to compare the target probability value with the largest value among the probability values of each category with a preset threshold, and to determine the target question category corresponding to the initial question text based on the comparison results.
[0071] In this embodiment of the application, the preprocessing module 310 may also be specifically used for: Obtain the original question text, and extract the text from the original question text based on multiple preset templates to obtain the template question text; Determine if there is any missing text in the template question text; If so, determine the missing template corresponding to the missing text in the preset template, and output a prompt message to the user based on the missing template; Based on user feedback, the missing template is completed to obtain the complete template issue text, and the complete template issue text is determined as the initial issue text. If not, then the template question text is determined to be the initial question text.
[0072] In this embodiment of the application, the preprocessing module 310 may also be specifically used for: The initial question text is cleaned, and the semantic similarity threshold between the cleaned initial question text and each question category is calculated to obtain multiple first probability values; Determine the word sequence corresponding to the initial question text.
[0073] In this embodiment of the application, the preprocessing module 310 may also be specifically used for: The initial question text is segmented using the word segmentation component to obtain a word sequence.
[0074] In this embodiment of the application, the calculation module 320 can also be specifically used for: Determine all the question statements included in the initial question text, and the total number of words in each question statement; Based on the number of times each word is repeated in each question statement and the total number of words corresponding to each question statement, determine the statement probability value corresponding to each word in each question statement; Based on the number of times each word is repeated in all question statements and the total number of question statements in the initial question text, determine the text distribution value corresponding to each word in the initial question text.
[0075] In this embodiment of the application, the input module 330 can also be specifically used for: Determine the standard question text and its corresponding standard category probability value, and input the standard question text into the initial Bart model to obtain the model category probability value output by the initial Bart model; Based on the standard question category and model category probability values, the model parameters of the initial Bart model are adjusted to obtain the target Bart model, so that the target Bart model can output the corresponding standard category probability value based on the input standard question text.
[0076] In this embodiment of the application, the comparison module 340 can also be specifically used for: The target probability value is determined by identifying the category with the largest probability value among all categories, and then compared with a preset threshold to obtain the comparison result. If the comparison result shows that the target probability value is greater than the preset threshold, then the problem category corresponding to the target probability value is determined as the target problem category; If the comparison result shows that the target probability value is not greater than the preset threshold, a prompt message will be output to inform the user that the current calculation is incorrect and the probability value needs to be recalculated.
[0077] Figure 4 This is a schematic diagram of the structure of an apparatus for performing a problem text processing method according to an embodiment of this application. Figure 4 As shown, the device 400 includes: The device 400 may include a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a communication component 403, and other components. The processor 401, memory 402, and communication component 403 are connected via a bus 404.
[0078] In the specific implementation process, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to execute the above-described problem text processing method.
[0079] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0080] Furthermore, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0081] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0082] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0083] In some embodiments, a computer program product is also provided, comprising a computer program or instructions that, when executed by a processor, implement the steps in any of the above-described methods for processing problem text.
[0084] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.
[0085] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.
[0086] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of program codes, which can be loaded by a processor to execute the steps in any of the problem text processing methods provided in embodiments of this application.
[0087] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0088] According to one aspect of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium.
[0089] Since the instructions stored in the storage medium can execute the steps in any of the problem text processing methods provided in the embodiments of this application, the beneficial effects that any of the problem text processing methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.
[0090] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the appended claims.
[0091] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.
Claims
1. A method for processing problem text, characterized in that, The method includes: According to the preset template, the initial question text is determined, and the initial question text is preprocessed to obtain the first probability value and word sequence corresponding to each question category; Calculate the statement probability value and text distribution value of each word in the word sequence, and map each word into a feature vector based on the statement probability value and text distribution value; The feature vector is input into the target Bart model to obtain the second probability value corresponding to each of the problem categories output by the target Bart model, and the category probability value is determined based on the first probability value and the second probability value. The target probability value with the largest value among the probability values of each category is compared with a preset threshold, and the target question category corresponding to the initial question text is determined based on the comparison result.
2. The method according to claim 1, characterized in that, The step of determining the initial question text according to the preset template includes: Obtain the original question text, and extract the text from the original question text according to multiple preset templates to obtain template question text; Determine whether there is any missing text in the template question text; If so, then determine the missing template corresponding to the missing text in the preset template, and output a prompt message to the user according to the missing template; Based on the user's feedback, the missing template is completed to obtain the complete template problem text, and the complete template problem text is determined as the initial problem text; If not, then the template question text is determined to be the initial question text.
3. The method according to claim 1, characterized in that, The preprocessing of the initial question text to obtain the first probability value and word sequence corresponding to each question category includes: The initial question text is cleaned, and the semantic similarity threshold between the cleaned initial question text and each question category is calculated to obtain multiple first probability values; Determine the lexical sequence corresponding to the initial question text.
4. The method according to claim 3, characterized in that, Determining the lexical sequence corresponding to the initial question text includes: The initial question text is segmented using the word segmentation component to obtain the word sequence.
5. The method according to claim 1, characterized in that, The calculation of the sentence probability value and text distribution value of each word in the word sequence includes: Determine all the question statements included in the initial question text, and the total number of words in each question statement; Based on the number of times the word element is repeated in each question statement and the total number of words corresponding to each question statement, the probability value of each word element in each question statement is determined. Based on the number of times the term is repeated in all the question statements and the total number of question statements in the initial question text, the text distribution value corresponding to each term in the initial question text is determined.
6. The method according to claim 1, characterized in that, Before inputting the feature vector into the target Bart model to obtain the second probability value corresponding to each of the problem categories output by the target Bart model, the method further includes: Determine the standard question text and its corresponding standard category probability value, and input the standard question text into the initial Bart model to obtain the model category probability value output by the initial Bart model; Based on the standard question category and the model category probability value, the model parameters of the initial Bart model are adjusted to obtain the target Bart model, so that the target Bart model can output the corresponding standard category probability value based on the input standard question text.
7. The method according to claim 1, characterized in that, The process involves comparing the target probability value with the largest value among the probability values of each category with a preset threshold, and determining the target question category corresponding to the initial question text based on the comparison results, including: The target probability value is determined by identifying the category with the largest value among all the category probability values, and the target probability value is compared with the preset threshold to obtain the comparison result; If the comparison result shows that the target probability value is greater than the preset threshold, then the problem category corresponding to the target probability value is determined to be the target problem category; If the comparison result indicates that the target probability value is not greater than the preset threshold, a prompt message is output. The prompt message is used to inform the user that the current calculation is incorrect and the probability value needs to be recalculated.
8. A device for processing problem text, characterized in that, The device includes: The preprocessing module is used to determine the initial question text according to the preset template, and to preprocess the initial question text to obtain the first probability value and word sequence corresponding to each question category; The calculation module is used to calculate the sentence probability value and text distribution value of each word in the word sequence, and map each word into a feature vector based on the sentence probability value and text distribution value; The input module is used to input the feature vector into the target Bart model to obtain the second probability value corresponding to each of the problem categories output by the target Bart model, and to determine the category probability value based on the first probability value and the second probability value. The comparison module is used to compare the target probability value with the largest value among the probability values of each category with a preset threshold, and determine the target question category corresponding to the initial question text based on the comparison result.
9. A device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in memory and configured to be executed by one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be called by a processor to perform the method as described in any one of claims 1 to 7.