Feedback information classification method and apparatus, electronic device, and storage medium

Abstract

The application discloses a feedback information classification method and device, electronic equipment and a storage medium, relates to the technical field of computers, and particularly relates to the fields of artificial intelligence such as deep learning, large models and natural language processing. The specific implementation scheme is as follows: feedback information for a target application program is acquired; the feedback information is classified to obtain a first classification result; wherein the first classification result comprises a content summary of the feedback information; a first content label of the feedback content is determined according to the similarity between content labels in a content label library corresponding to the target application program and the content summary; and a second classification result of the feedback information is determined according to the first content label.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, especially to the field of artificial intelligence such as deep learning, large models, and natural language processing, and specifically relates to a feedback information classification method and device, an electronic device, and a storage medium. BACKGROUND

[0002] With the continuous development of computer technology and the rapid iteration of program development, application programs have become an indispensable part of people's lives. Users can give feedback to application programs during use, and the feedback of users can be various. SUMMARY

[0003] The present application provides a feedback information classification method and device, an electronic device, and a storage medium.

[0004] According to an aspect of the present application, a feedback information classification method is provided, comprising:

[0005] obtaining feedback information for a target application program;

[0006] classifying the feedback information to obtain a first classification result; wherein the first classification result includes a content summary of the feedback information;

[0007] determining a first content label of the feedback content according to the content summary and the similarity between content labels in a content label library corresponding to the target application program;

[0008] determining a second classification result of the feedback information according to the first content label.

[0009] According to another aspect of the present application, a feedback information classification device is provided, comprising:

[0010] an obtaining module configured to obtain feedback information for a target application program;

[0011] a classification module configured to classify the feedback information to obtain a first classification result; wherein the first classification result includes a content summary of the feedback information;

[0012] a first determining module configured to determine a first content label of the feedback content according to the content summary and the similarity between content labels in a content label library corresponding to the target application program;

[0013] a second determining module configured to determine a second classification result of the feedback information according to the first content label.

[0014] According to another aspect of the present application, an electronic device is provided, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein

[0017] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the above embodiments.

[0018] According to another aspect of the present application, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the method described in the above embodiments.

[0019] According to another aspect of the present application, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method described in the above embodiments.

[0020] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings are used to better understand the present application, and do not limit the present application. Among them:

[0022] Figure 1 a flowchart of a feedback information classification method provided by an embodiment of the present application;

[0023] Figure 2 a flowchart of a feedback information classification method provided by another embodiment of the present application;

[0024] Figure 3 a flowchart of a feedback information classification method provided by another embodiment of the present application;

[0025] Figure 4 a process diagram of a feedback information classification method provided by an embodiment of the present application;

[0026] Figure 5 a process diagram of similarity calculation between content summary and content label provided by an embodiment of the present application;

[0027] Figure 6 a structural diagram of a feedback information classification device provided by an embodiment of the present application;

[0028] Figure 7 a block diagram of an electronic device for implementing the feedback information classification method of the embodiments of the present application. DETAILED DESCRIPTION

[0029] The exemplary embodiments of this application are described herein with reference to the accompanying drawings, which are meant to be exemplary in nature, and include various details intended to facilitate understanding of the application. Thus, it should be apparent to those skilled in the art that various modifications and changes can be made in the embodiments described without departing from the scope and spirit of the application. Likewise, the description is not to be understood as implying that the application is limited to the particular forms described, but that the application is to be given the broadest possible interpretation in accordance with the appended claims.

[0030] The data acquisition, transmission, storage, use, processing and the like in the technical solutions of the application comply with the provisions of national laws and regulations and do not violate public order and good customs.

[0031] The feedback information classification method, device, electronic device and storage medium of the embodiments of the application are described below with reference to the accompanying drawings.

[0032] In some embodiments, keywords corresponding to content categories can be constructed in advance, and the user feedback content is retrieved and matched. If a plurality of keywords of a certain content category are matched, it can be considered that the user feedback belongs to the category. However, the selection and setting of keywords can be affected by subjective factors, and different classifiers can select different keywords, resulting in inconsistent classification results, thereby affecting the accuracy of the classification results.

[0033] Figure 1 The flowchart of the feedback information classification method provided by an embodiment of the application is shown.

[0034] The feedback information classification method of the embodiments of the application can be executed by the feedback information classification device of the embodiments of the application. The device can be configured in an electronic device to realize the classification function of feedback information.

[0035] The electronic device can be any device with computing capability, such as a personal computer, a mobile terminal, a server, etc. The mobile terminal can be a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, etc. with various operating systems, touch screens and / or display screens.

[0036] As shown in the figure, the feedback information classification method includes: Figure 1

[0037] Step 101, obtaining feedback information for a target application program.

[0038] For example, the feedback information can be given by a user for the use of a target application program, such as "Application A has multiple product comparison functions, which is quite good".

[0039] ​The feedback information can be in the form of text, voice or other forms, and is not limited in this regard. If the feedback information is in the form of voice or other forms, it can be converted into text and then processed.

[0040] The feedback information for the target application can be obtained by crawling from the network, read from a locally stored feedback-related document, or obtained by other means, and is not limited in this regard.

[0041] The feedback information for the target application can be a problem of the target application, a positive evaluation of the target application, or other types of feedback, and is not limited in this regard.

[0042] In step 102, the feedback information is classified to obtain a first classification result; wherein the first classification result includes a summary of the content of the feedback information.

[0043] In this application, the feedback information can be classified in at least one classification dimension to obtain a first classification result. For example, the feedback information can be classified in the content summary dimension to obtain a content summary of the feedback information, or other classification dimensions can also be used.

[0044] For example, the content summary of the feedback information can be understood as a summary of the feedback content in the feedback information.

[0045] For example, the feedback information includes a feedback title and a feedback content, and the feedback information can be classified according to the feedback title and the feedback content to obtain a first classification result. For example, keywords can be extracted from the feedback content, and the feedback content can be semantically recognized, and the extracted keywords can be combined with the semantic recognition result to obtain a feedback summary. The feedback title can be matched with a preset feedback scenario, and the feedback scenario of the feedback information can be determined according to the matching result, that is, the first classification result obtained includes the feedback summary and the feedback scenario.

[0046] For example, some feedback information may not distinguish which is the feedback title and which is the feedback content. The feedback title and the feedback content can be extracted from the feedback information, and the feedback information can be classified according to the feedback title and the feedback content to obtain a first classification result. For example, the feedback title of the feedback information can be obtained by a title generation model, and the similarity between the feedback title and the sentences in the feedback information can be calculated. The sentences in the feedback information other than the sentence with the highest similarity to the feedback title can be taken as the feedback content.

[0047] Therefore, the feedback title and the feedback content in the feedback information are determined, and the feedback information is classified according to the feedback title and the feedback content, so that the accuracy of the classification result is improved.

[0048] In step 103, the first content label of the feedback content is determined according to the content summary and the similarity between the content labels in the content label library corresponding to the target application program.

[0049] The content label library includes at least one content label, and one content label can be used to represent one content category. For example, the content labels in the content label library corresponding to an application program include "download failure", "download list not displayed", etc. For example, the content label can be a single-level label or a multi-level label. For example, a content label can be "play-video function-subtitle-intelligent ai subtitle", which is a four-level label.

[0050] For example, different application programs can correspond to different content label libraries.

[0051] In this application, the similarity between the content summary and each content label in the content label library corresponding to the target application program can be calculated, and the first content label of the feedback content can be determined according to the similarity corresponding to each content label. For example, the content label with the highest similarity can be used as the first content label of the feedback content.

[0052] For example, the content summary and the content label are respectively vectorized to obtain a first vector of the content summary and a second vector of the content label, and the cosine of the first vector and the second vector is calculated to obtain a cosine value, and the similarity between the content summary and the content label is determined according to the cosine value. For example, the content summary and the content label can be vectorized using the same word vector model. For example, the cosine value can be used as the similarity between the content summary and the content label.

[0053] Therefore, by calculating the cosine of the vectors of the content summary and the content label to determine the similarity, the accuracy of the calculation is improved.

[0054] In step 104, the second classification result of the feedback information is determined according to the first content label.

[0055] For example, the second classification result can include the first content label.

[0056] Exemplarily, if the classification of the feedback information is a multi-dimensional classification, the second classification result can include other classification labels, such as a scene label and an attitude label, in addition to the first content label, where the scene label represents a feedback scene category to which the feedback information belongs, and the attitude label represents an attitude category of a feedback attitude of the feedback information. In this way, the classification labels of multiple classification dimensions of the feedback information can be determined, and the fine-grained classification of the feedback information can be implemented.

[0057] In the embodiments of this application, the feedback information of the target application program is classified to obtain a first classification result, and the similarity between the content summary in the first classification result and the content labels in the content label library corresponding to the target application program is determined to determine the first content label of the feedback content, and then the second classification result of the feedback information is determined according to the first content label. In this way, after the content summary of the feedback information is classified, the content label of the feedback information is determined based on the similarity between the content summary and the content labels in the content label library, and the accuracy of the classification of the feedback information is improved.

[0058] Figure 2 The flowchart of the feedback information classification method provided by another embodiment of this application is shown.

[0059] As shown in Figure 2 The feedback information classification method includes the following steps.

[0060] In step 201, the feedback information for the target application program is obtained.

[0061] In this application, step 201 can adopt any of the implementation manners of the embodiments of this application, and therefore will not be described here.

[0062] In step 202, the feedback information is classified to obtain a first classification result, and the content summary of the feedback information is included in the first classification result.

[0063] In this application, step 202 can adopt any of the implementation manners of the embodiments of this application, and therefore will not be described here.

[0064] In step 203, if the similarity between the content summary and the second content label in the content label library is greater than a first threshold, the first content label is determined according to the second content label.

[0065] In this application, if the similarity between the content summary of the feedback information and the second content label in the content label library is greater than the first threshold, the first content label of the feedback information can be determined according to the second content label and the number of the second content label.

[0066] Exemplarily, if the second content label is one, the second content label can be directly determined as the first content label of the feedback information.

[0067] For example, if there are multiple second content tags, i.e., multiple content tags in the content tag library have similarity with the content summary greater than the first threshold, the second content tag with the highest similarity with the content summary can be determined from the multiple second content tags, and the second content tag with the highest similarity can be determined as the first content tag. In this way, the second content tag with the highest similarity is determined as the first content tag of the feedback information, which can improve the accuracy of the content tag, and thus can improve the accuracy of the final classification result of the feedback information.

[0068] Alternatively, if the similarity between the content summary of the feedback information and the content tags in the content tag library is less than the first threshold, i.e., there is no suitable content tag for the feedback information in the content tag library, the content summary of the feedback information can be determined as the first content tag of the feedback information. In this way, when there is no suitable content tag for the feedback information in the content tag library, the content summary is determined as the content tag of the feedback information, which improves the accuracy of the classification of the feedback information.

[0069] Alternatively, if the similarity between the content summary of the feedback information and the content tags in the content tag library is less than the first threshold, i.e., there is no suitable content tag for the feedback information in the content tag library, the content summary of the feedback information can be determined as the first content tag of the feedback information. In this way, when there is no suitable content tag for the feedback information in the content tag library, the content summary is determined as the content tag of the feedback information, which improves the accuracy of the classification of the feedback information.

[0070] Step 204, determining a second classification result of the feedback information according to the first content tag.

[0071] In this application, step 204 can use any of the implementation manners of the embodiments of the present application, and thus will not be described here.

[0072] In the embodiments of the present application, if the similarity between the content summary of the feedback information and the second content tag in the content tag library is greater than the first threshold, the first content tag of the feedback information can be determined according to the second content tag. In this way, the content tag of the feedback information is determined according to the second content tag in the content tag library with similarity greater than the first threshold with the content summary, which can improve the accuracy of the content tag of the feedback information, and thus can improve the accuracy of the classification result of the feedback information.

[0073] Figure 3 The flowchart of the feedback information classification method provided by another embodiment of the present application is shown.

[0074] As shown in Figure 3 the feedback information classification method comprises:

[0075] Step 301, obtaining feedback information for a target application.

[0076] In the present application, step 301 can adopt any of the implementation manners of the embodiments of the present application, and therefore will not be described here.

[0077] In step 302, the feedback information is classified in multiple classification dimensions by using a large language model to obtain a first classification result.

[0078] The multiple classification dimensions can include a content summary dimension and other classification dimensions.

[0079] In the present application, a prompt template for obtaining feedback information can be obtained, and the prompt template is modified according to the feedback information to obtain prompt information, and the prompt information is input into a large language model to classify the feedback information in multiple classification dimensions to obtain a first classification result.

[0080] For example, the prompt template has multiple slots to be filled, and the prompt template can be filled with slots according to the feedback information to obtain prompt information, and the prompt information is input into a large language model to classify the feedback information in multiple classification dimensions to obtain a first classification result.

[0081] For example, the prompt template can include multiple classification dimensions, and can also include roles, feedback information, return result formats, etc. For example, the role and return format in the prompt template can be known, and the feedback information to be classified can be filled into the prompt template to obtain prompt information. For example, the role is a feedback classification expert, and the return result format is a json format.

[0082] For example, the prompt template is "Please act as a feedback classification expert and combine knowledge reserves to classify the feedback information [] in feedback scene dimension, feedback attitude dimension and content summary dimension, output classification result, and the classification result format is json format", and the feedback information is "The video cannot be played after downloading, and keeps rotating when opened". Therefore, the prompt information is "Please act as a feedback classification expert and combine knowledge reserves to classify the feedback information [video cannot be played after downloading, and keeps rotating when opened] in feedback scene dimension, feedback attitude dimension and content summary dimension, output classification result, and the classification result format is json format".

[0083] For example, the prompt template can also include reference feedback of a target application program corresponding to a content label in a content label library. For example, the reference feedback can be historical feedback information of the target application program, or a content summary of the historical feedback information of the target application program. For example, the reference feedback corresponding to one content label can be one or more, which is not limited.

[0084] For example, the content label "smart ai subtitle" corresponds to reference feedback [subtitle is not clear, subtitle is not displayed, subtitle does not follow], and the label corresponds to three reference feedbacks.

[0085] In the present application, the large language model can classify the feedback information based on the reference feedback corresponding to each content label in the prompt information. Thus, using the reference feedback to classify the feedback information can improve the classification accuracy and efficiency.

[0086] For example, the feedback information can include a feedback title and feedback content, and the prompt template includes a feedback title slot and a feedback content slot. The prompt template is modified according to the feedback title and the feedback content to generate the prompt information. For example, the prompt template can be filled with the feedback title slot and the feedback content slot according to the feedback title and the feedback content of the feedback information to generate the prompt information. For example, the feedback title slot in the prompt template can be filled according to the feedback title of the feedback information, and the feedback content slot in the prompt template can be filled according to the feedback content.

[0087] For example, the prompt information is "Please provide feedback as a feedback classification expert, based on the knowledge reserve, classify the feedback information with a feedback title of [video download and play abnormally] and feedback content of [video download and play abnormally, open and keep rotating] in three dimensions of feedback scene dimension, feedback attitude dimension, and content summary dimension, output the classification result, and the format of the classification result is json format".

[0088] For example, some feedback information may not distinguish which is the feedback title and which is the feedback content. The feedback title and the feedback content can be extracted from the feedback information, and the prompt template is modified according to the feedback title and the feedback content to generate the prompt information.

[0089] Thus, the large language model classifies the feedback information based on the feedback title and the feedback content of the feedback information, which can improve the accuracy and efficiency of the classification.

[0090] For example, the multiple classification dimensions can also include a feedback scene dimension and a feedback attitude dimension. The scene label library and the attitude label library corresponding to the target application program can be obtained in advance, and a large language model can be used to classify the feedback information in the feedback scene dimension and the feedback attitude dimension based on the scene label library and the attitude label library by processing the prompt information to obtain a first classification result. The first classification result can also include the feedback scene and the feedback attitude of the feedback information.

[0091] The scene label library can include one or more scene labels, wherein a scene label can represent a feedback scene category. The attitude label library can include one or more attitude labels, wherein an attitude label represents a feedback attitude category.

[0092] For example, the scene tags in the scene tag library corresponding to an application scenario include "download", "member", "activity", "backup", and the like, and the attitude tags in the attitude tag library corresponding to the application scenario include "criticism", "praise", "promotion", "consultation", and the like.

[0093] For example, the large language model can determine the feedback scene to which the feedback information belongs from the scene tags in the scene tag library, and determine the feedback attitude of the feedback information from the attitude tags in the attitude tag library.

[0094] Therefore, the feedback information is classified in multiple classification dimensions based on the scene tag library and the attitude tag library, which can improve the accuracy and efficiency of the classification result, and can also achieve fine-grained separation of the feedback information.

[0095] In step 303, a first content tag of the feedback content is determined according to the content summary and the similarity between the content tags in the content tag library corresponding to the target application program.

[0096] In this application, step 302 can adopt any of the implementation manners of the embodiments of the present application, and therefore will not be described here.

[0097] In step 304, a second classification result of the feedback information is determined according to the first content tag.

[0098] In this application, step 304 can adopt any of the implementation manners of the embodiments of the present application, and therefore will not be described here.

[0099] For example, the first classification result can also include the feedback scene corresponding to the feedback information. If the feedback scene of the feedback information is a first preset category, the scene tag of the feedback information can be generated by using the large language model, and then the second classification result can be determined according to the scene tag and the first content tag.

[0100] The second classification result can include the first content tag and the scene tag, and can also include other category tags.

[0101] For example, the scene tag library can include the first preset category. If the feedback information does not match the scene tags in the scene tag library, the feedback scene of the feedback information can be determined as the first preset category. For example, the first preset category is "other", and if the feedback scene of the feedback information is "other", the scene tag of the feedback information can be generated by using the large language model.

[0102] Exemplarily, the prompt information can be generated according to the feedback information and the first preset category, and the prompt information is input into the large language model to enable the large language model to generate a scene label for the feedback information based on context information. For example, the prompt information is “the feedback scene of the feedback information [xxxx] obtained through classification is “other”, please generate a scene label for the feedback information in combination with the knowledge reserve, and the output format is json format”.

[0103] Therefore, if the feedback scene of the feedback information is the first preset category, the large language model can be used to generate a scene label for the feedback information, thereby improving the accuracy of scene classification.

[0104] Optionally, if the feedback scene of the feedback information is the first preset category, the large language model can be used to generate a scene label for the feedback information, and the scene label of the feedback information is added to the scene label library to update the scene label library, thereby enriching the scene label library.

[0105] Exemplarily, the first classification result can further include a first feedback attitude of the feedback information, which can be determined based on an attitude label library corresponding to the target application program. If the first feedback attitude is a second preset category, a second feedback attitude of the feedback information is determined based on the attitude label library. If the second feedback attitude matches a target attitude label in the attitude label library, a second classification result of the feedback information can be determined according to the first content label and the target attitude label.

[0106] The second classification result can include the first content label and the target attitude label, and can further include other category labels, such as a scene label of the feedback information.

[0107] Exemplarily, if the first feedback attitude does not match any attitude label in the attitude label library, the first feedback attitude of the feedback information can be determined as the second preset category. In the case where the first feedback attitude is the second preset category, a second feedback attitude of the feedback information can be determined based on the attitude label library.

[0108] For example, the second preset category is “other”, and if the first feedback attitude of the feedback information is “other”, the feedback attitude of the feedback information can be determined based on the attitude label library.

[0109] Exemplarily, the first feedback attitude can be obtained by classifying the feedback information in the feedback attitude dimension based on the attitude label library corresponding to the target application program. If the first feedback attitude is the second preset category, a second feedback attitude of the feedback information can be determined from the attitude label library using the large language model. If the second feedback attitude matches a target attitude label in the attitude label library, a second classification result can be determined according to the target attitude label and the first content label.

[0110] For example, the second feedback attitude "praise" of certain feedback information matches the target attitude label "praise" in the attitude label library, and the second classification result of the feedback information can be determined according to the attitude label.

[0111] Therefore, if the first feedback attitude of the feedback information is the second preset category, the second feedback attitude of the feedback information can be determined based on the attitude label library, and when the second feedback attitude matches the target attitude label in the attitude label library, the attitude label of the feedback information is obtained, thereby improving the accuracy of feedback attitude classification and further improving the accuracy of the classification result.

[0112] In the embodiments of the present application, the feedback information is classified in multiple classification dimensions by using a large language model to obtain a first classification result, and the first content label of the feedback information is retrieved in the content label library according to the content summary. Therefore, the feedback information is classified by combining the large language model and the retrieval method, thereby improving the classification efficiency and accuracy.

[0113] The feedback information classification method of the embodiments of the present application can include the following four steps: feedback category library building, prompt information generation, feedback classification, and similarity calculation.

[0114] Feedback category library building: different databases are established for the multi-dimensional classification of user feedback content, mainly including feedback attitude, feedback problem scenario, and specific problem content label, which will be the main basis for subsequent classification

[0115] Prompt information generation: the prompt information required by the large language model can include roles, feedback content information (including title, specific content, etc.), required data format (such as json format), and classification (such as feedback attitude, feedback problem scenario, and problem summary).

[0116] Feedback classification: the large language model is called through the prompt information, and the large language model classifies the feedback attitude, feedback scenario, and content summary.

[0117] Similarity calculation: since the large language model has randomness each time, it is easy to cause one feedback information and one content label, so the content summary obtained through classification cannot be directly used as a content label. The cosine value can be calculated by vector. If the cosine value of a certain label is greater than the first threshold value, it is considered to be highly similar, and the content label with the largest cosine value in the content label library is used. If there are multiple content labels with cosine values greater than the first threshold value, the content label with the largest cosine value is used. If there is none, the content summary is inserted into the content label library as a new label.

[0118] In order to facilitate understanding, the following will be described in combination with Figure 4 and Figure 5 Figure 4 ​This is a schematic diagram illustrating a feedback information classification process provided in an embodiment of this application. Figure 5 This is a schematic diagram illustrating the process of calculating the similarity between content summaries and content tags, provided as an embodiment of this application.

[0119] like Figure 4 As shown, three databases—scene tag library, content tag library, and attitude tag library—can be pre-built for an application that provides cloud storage services. For example, the feedback information might be "Video playback is abnormal after download; the video cannot be played after download, it keeps spinning." This feedback information includes a feedback title and feedback content. Based on the feedback title and content, the feedback title and feedback content slots in the prompt template can be filled to obtain the prompt information. The prompt information used includes slot fields such as role, feedback title, feedback content, and return format. Additionally, the output fields include feedback scenario, feedback attitude, and content summary.

[0120] After constructing the prompt message, input the prompt message into the large language model for processing, and obtain the output result of the large language model {feedback scenario: video; feedback attitude: 1 (criticism); content summary: the downloaded video keeps spinning when played}.

[0121] Next, the similarity between the summary of the feedback information and the content tags in the content tag library is calculated. For details of the calculation process, please refer to [link / reference needed]. Figure 5 . Figure 5 In this process, the content summary and tags in the content tag library can be vectorized using a word vector model. The cosine of the vector of the content summary and the vector of each content tag is calculated. If the cosine value is greater than a first threshold of 0.7, the old tag is used; otherwise, the content summary is inserted as a new tag into the content tag library. Using an old tag can be understood as using the content tag with the highest similarity among content tags in the content tag library with a similarity greater than 0.7 as the content tag for feedback information.

[0122] like Figure 5 As shown, it can also determine whether there is a feedback scene with feedback information in the scene tag library. If it does not exist, a new scene tag is generated and added to the scene tag library.

[0123] To achieve the above embodiments, this application also proposes a feedback information classification device. Figure 6 This is a schematic diagram of the structure of a feedback information classification device provided in an embodiment of this application.

[0124] like Figure 6 As shown, the feedback information classification device 600 includes:

[0125] The acquisition module 610 is used to acquire feedback information for the target application;

[0126] The classification module 620 is configured to classify the feedback information to obtain a first classification result; wherein the first classification result includes a content summary of the feedback information.

[0127] The first determination module 630 is configured to determine a first content label of the feedback content according to the content summary and similarity between content labels in a content label library corresponding to the target application program.

[0128] The second determination module 640 is configured to determine a second classification result of the feedback information according to the first content label.

[0129] Optionally, the first determination module 630 is configured to:

[0130] In a case where the similarity between the content summary and the second content label in the content label library is greater than a first threshold, the first content label is determined according to the second content label.

[0131] Optionally, the second content label is a plurality of, and the first determination module 630 is configured to:

[0132] determine a second content label with the highest similarity to the content summary from the plurality of second content labels;

[0133] determine the second content label with the highest similarity as the first content label.

[0134] Optionally, the first determination module 630 is configured to:

[0135] In a case where the similarity between the content summary and the content label in the content label library is less than the first threshold, the content summary is determined as the first content label.

[0136] Optionally, the apparatus can further include:

[0137] The first adding module is configured to add the content summary to the content label library in a case where the similarity between the content summary and the content label in the content label library is less than the first threshold.

[0138] Optionally, the classification module 620 is configured to:

[0139] obtain a prompt template; wherein the prompt template includes a plurality of classification dimensions, and the plurality of classification dimensions include a content summary dimension;

[0140] modify the prompt template according to the feedback information to generate prompt information;

[0141] classify the feedback information in a plurality of classification dimensions based on the prompt information by using a large language model to obtain the first classification result.

[0142] Optionally, the classification module 620 is configured to:

[0143] extract a feedback title and a feedback content from the feedback information;

[0144] modify the prompt template according to the feedback title and the feedback content, and generate prompt information.

[0145] Optionally, the prompt template further includes reference feedback of a target application program corresponding to a content label in the content label library.

[0146] Optionally, the multiple classification dimensions include a feedback scene dimension and a feedback attitude dimension, and the classification module 620 is configured to:

[0147] obtain a scene label library and an attitude label library corresponding to the target application program;

[0148] adopt a large language model, classify the feedback information in the feedback scene dimension and the feedback attitude dimension based on the scene label library and the attitude label library by processing the prompt information, and obtain a first classification result.

[0149] Optionally, the first classification result further includes a feedback scene corresponding to the feedback information, and the second determination module 640 is configured to:

[0150] in a case where the feedback scene is a first preset category, generate a scene label of the feedback information by using the large language model;

[0151] determine a second classification result according to the scene label and the first content label.

[0152] Optionally, the apparatus can further include:

[0153] the second adding module is configured to add the scene label to the scene label library corresponding to the target application program.

[0154] Optionally, the first classification result further includes a first feedback attitude of the feedback information, and the second determination module 640 is configured to:

[0155] in a case where the first feedback attitude is a second preset category, continue to determine a second feedback attitude of the feedback information based on the attitude label library;

[0156] in a case where the second feedback attitude matches a target attitude label in the attitude label library, determine the second classification result according to the target attitude label and the first content label.

[0157] Optionally, the classification module 620 is configured to:

[0158] extract a feedback title and a feedback content from the feedback information;

[0159] classify the feedback information according to the feedback title and the feedback content, and obtain a first classification result.

[0160] Optionally, the apparatus can further include:

[0161] a vectorization module configured to vectorize the content summary and the content label respectively to obtain a first vector of the content summary and a second vector of the content label;

[0162] a calculation module configured to calculate a cosine value by performing cosine calculation on the first vector and the second vector;

[0163] a third determination module configured to determine a similarity between the content summary and the content label according to the cosine value.

[0164] It should be noted that the explanation of the foregoing feedback information classification method embodiment is also applicable to the feedback information classification apparatus of the embodiment, and thus will not be described here again.

[0165] In the embodiment of the application, the feedback information of the target application program is classified to obtain a first classification result, and a first content label of the feedback content is determined according to the similarity between the content summary in the first classification result and the content label in the content label library corresponding to the target application program, and a second classification result of the feedback information is determined according to the first content label. Thus, after the content summary of the feedback information is classified, the content label of the feedback information is determined based on the similarity between the content summary and the content label in the content label library, and the accuracy of the feedback information classification is improved.

[0166] According to the embodiments of the application, the application further provides an electronic device, a readable storage medium and a computer program product.

[0167] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the application described and / or claimed in this document.

[0168] As Figure 7As shown, the device 700 includes a computing unit 701 that can perform various appropriate actions and processes in accordance with a computer program stored in a ROM (Read-Only Memory) 702 or a computer program loaded into a RAM (Random Access Memory) 703 from the storage unit 708. Various programs and data required for the operation of the device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An I / O (Input / Output) interface 705 is also connected to the bus 704.

[0169] A plurality of components in the device 700 are connected to the I / O interface 705, including an input unit 706 such as a keyboard, a mouse, and the like; an output unit 707 such as various types of displays, speakers, and the like; a storage unit 708 such as a magnetic disk, an optical disk, and the like; and a communication unit 709 such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0170] The computing unit 701 can be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, and the like. The computing unit 701 performs various methods and processes described above, such as the feedback information classification method. For example, in some embodiments, the feedback information classification method can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the feedback information classification method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the feedback information classification method by any other appropriate means, such as by means of firmware.

[0171] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a Field Programmable Gate Array (FPGA), an Application-Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a System on a Chip (SOC), a Complex Programmable Logic Device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0172] Program code for carrying out methods of the present application can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0173] In the context of this application, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include a linearly-programmed electronic storage, a portable computer diskette, a hard disk, a RAM, a ROM, an EPROM (Electrically Programmable Read-Only-Memory), or a flash memory, an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0174] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0175] The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, and a blockchain network.

[0176] The computer system can include clients and servers. This relationship can be between two computers, or between computers and servers located throughout the network, depending on the context in which the term is used. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are a host product in the cloud computing service system to solve the defects of large management difficulty and weak business scalability in traditional physical hosts and VPS services (Virtual Private Server, Virtual Private Server). The server can also be a server of a distributed system, or a server combined with a blockchain.

[0177] According to the embodiments of the present application, the present application also provides a computer program product, when the instruction processor in the computer program product executes, executes the feedback information classification method proposed in the above embodiments of the present application.

[0178] It should be understood that the steps shown above can be reordered, added, or deleted using various forms of flow. For example, the steps described in the present application can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present application can be achieved, which is not limited herein.

[0179] The above detailed description does not constitute a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for classifying feedback information, characterized in that, The method includes: Obtain feedback information about the target application from the web through web scraping; The feedback title and content are extracted from the feedback information; the prompt template is modified according to the feedback title and the feedback content to generate prompt information; using a large language model, the feedback information is classified into three dimensions based on the prompt information: content summary dimension, feedback scenario dimension, and feedback attitude dimension, to obtain a first classification result; wherein, the first classification result includes the content summary, first feedback attitude, and feedback scenario of the feedback information, and the first feedback attitude is determined based on the attitude tag library corresponding to the target application; Based on the similarity between the content summary and the content tags in the content tag library corresponding to the target application, a first content tag for the feedback content is determined; wherein: if the similarity between the content summary and the second content tag in the content tag library is greater than a first threshold, the first content tag is determined based on the second content tag and the number of the second content tags; if all the similarities are less than the first threshold, the content summary is used as the first content tag; Based on the first content tag, a second classification result of the feedback information is determined; wherein, when the feedback scenario is a first preset category, a scenario tag for the feedback information is generated using a large language model, and the second classification result is determined based on the scenario tag and the first content tag; when the first feedback attitude is a second preset category, the second feedback attitude of the feedback information is further determined based on the attitude tag library; when the second feedback attitude matches a target attitude tag in the attitude tag library, the second classification result is determined based on the target attitude tag and the first content tag.

2. The method as described in claim 1, wherein, Determining the first content tag based on the second content tag and the number of the second content tags includes: If the number of the second content tags is one, then the second content tag is determined as the first content tag.

3. The method as described in claim 1, wherein, Determining the first content tag based on the second content tag and the number of the second content tags includes: If there are multiple second content tags, then the second content tag with the highest similarity is determined as the first content tag.

4. The method of claim 1, further comprising: If all the similarities are less than the first threshold, then the content summary will be added to the content tag library.

5. The method of claim 1, wherein, The prompt template also includes reference feedback for the target application corresponding to the content tags in the content tag library.

6. The method of claim 1, wherein, The method employs a large language model to classify the feedback information based on the prompt information across three dimensions: content summary, feedback scenario, and feedback attitude, to obtain the first classification result, including: Obtain the scene tag library and attitude tag library corresponding to the target application; Using the large language model, based on the scene tag library and the attitude tag library, the prompt information is processed to classify the feedback information according to the feedback scene dimension and the feedback attitude dimension, thus obtaining the first classification result.

7. The method of claim 1, further comprising: Add the scene tag to the scene tag library corresponding to the target application.

8. The method according to any one of claims 1-7, further comprising: The content summary and the content tags are vectorized respectively to obtain a first vector of the content summary and a second vector of the content tags; Perform cosine calculations on the first vector and the second vector to obtain cosine values; The similarity between the content summary and the content tag is determined based on the cosine value.

9. A feedback information classification device, characterized in that, The device includes: The acquisition module is used to obtain feedback information for the target application from the network through crawling. A classification module is used to extract the feedback title and feedback content from the feedback information; modify the prompt template according to the feedback title and feedback content to generate prompt information; and use a large language model to classify the feedback information according to three dimensions: content summary dimension, feedback scenario dimension, and feedback attitude dimension, to obtain a first classification result; wherein, the first classification result includes the content summary, first feedback attitude, and feedback scenario of the feedback information, and the first feedback attitude is determined based on the attitude tag library corresponding to the target application; The first determining module is used to determine a first content tag of the feedback content based on the similarity between the content summary and the content tags in the content tag library corresponding to the target application; wherein: if the similarity between the content summary and the second content tag in the content tag library is greater than a first threshold, then the first content tag is determined based on the second content tag and the number of the second content tags; if all the similarities are less than the first threshold, then the content summary is used as the first content tag; The second determining module is used to determine a second classification result of the feedback information based on the first content tag; wherein, when the feedback scenario is a first preset category, a scenario tag for the feedback information is generated using a large language model, and the second classification result is determined based on the scenario tag and the first content tag; when the first feedback attitude is a second preset category, the second feedback attitude of the feedback information is further determined based on the attitude tag library; when the second feedback attitude matches a target attitude tag in the attitude tag library, the second classification result is determined based on the target attitude tag and the first content tag.

10. The apparatus of claim 9, wherein, Determining the first content tag based on the second content tag and the number of the second content tags includes: If the number of the second content tags is one, then the second content tag is determined as the first content tag.

11. The apparatus of claim 9, wherein, Determining the first content tag based on the second content tag and the number of the second content tags includes: If there are multiple second content tags, then the second content tag with the highest similarity is determined as the first content tag.

12. The apparatus of claim 9, further comprising: The first adding module is used to add the content summary to the content tag library if all the similarities are less than the first threshold.

13. The apparatus of claim 9, wherein, The prompt template also includes reference feedback for the target application corresponding to the content tags in the content tag library.

14. The apparatus of claim 9, wherein, The method employs a large language model to classify the feedback information based on the prompt information across three dimensions: content summary, feedback scenario, and feedback attitude, to obtain the first classification result, including: Obtain the scene tag library and attitude tag library corresponding to the target application; Using the large language model, based on the scene tag library and the attitude tag library, the prompt information is processed to classify the feedback information according to the feedback scene dimension and the feedback attitude dimension, thus obtaining the first classification result.

15. The apparatus of claim 9, further comprising: The second adding module is used to add the scene tag to the scene tag library corresponding to the target application.

16. The apparatus of any one of claims 9-15, further comprising: A vectorization module is used to vectorize the content summary and the content tags respectively, to obtain a first vector of the content summary and a second vector of the content tags; The calculation module is used to perform cosine calculation on the first vector and the second vector to obtain the cosine value; The third determining module is used to determine the similarity between the content summary and the content tag based on the cosine value.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

19. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-8.

Images