Operation information recommendation method and device and computer equipment

Abstract

The invention discloses an operation information recommendation method and device and computer equipment. The method comprises the steps that a sample data set is obtained, the sample data set comprises a pre-labeled positive sample set and a pre-labeled negative sample set, positive samples are selected webpage data, and negative samples are unselected webpage data; the samples in the sample data set are processed, notional words of fine texts and positive samples used for representing sample key information are obtained, and the notional words are at least used for representing user preference content; and determining to-be-recommended target operation information based on the conceptual words and the vectors corresponding to the refined text. The technical problem that a model training process is relatively tedious due to the fact that a large amount of artificial feature engineering needs to be carried out in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the technical field of text recommendation, and in particular to an operation information recommendation method, device and computer equipment. Background Art

[0002] The current text recommendation system is mainly based on the following technologies: Collaborative filtering recommendation can make recommendations based on user historical behavior and similar user preferences, but it is difficult to handle the cold start problem. Content-based recommendation achieves recommendations by matching product content features with user portraits, which requires a lot of manual feature engineering. Association rule recommendation can discover association rules based on product co-occurrence relationships, but the rules are prone to being fixed and not updated in a timely manner. Deep learning recommendation uses multi-layer neural networks to extract product features, which requires a lot of training data to support model training. With the development of the Internet, personalized recommendation systems have become increasingly important. Traditional recommendation systems often use collaborative filtering, content filtering, or hybrid methods to provide users with personalized recommendations. However, these methods are usually limited in terms of data size, distribution, quality, professional feature engineering, scalability, and accuracy. Summary of the invention

[0003] The embodiments of the present application provide an operation information recommendation method, apparatus, and computer device to at least solve the technical problem in the related art that the model training process is relatively cumbersome due to the need for a large amount of manual feature engineering.

[0004] According to one aspect of an embodiment of the present application, a method for recommending operational information is provided, comprising: obtaining a sample data set, the sample data set comprising: a pre-labeled positive sample set and a negative sample set, wherein the positive sample is selected web page data, and the negative sample is unselected web page data; processing samples in the sample data set to obtain concise text for characterizing key information of the sample and conceptual words for the positive sample, wherein the conceptual words are at least used to characterize content preferred by the user; and determining target operational information to be recommended based on vectors corresponding to the conceptual words and the concise text.

[0005] Optionally, the samples in the sample data set are processed to obtain a concise text for characterizing the key information of the samples, including: obtaining the number of text words corresponding to the samples in the sample data set; when the number of text words is less than a first number, directly terminating the processing of the samples; when the number of text words is greater than the first number and less than a second number, determining the samples in the sample data set as the concise text; when the number of text words is greater than the second number, using a pre-trained language model to process the samples in the sample data set according to a preset first prompt engineering template to obtain the initial concise text; replacing the line break symbols in the initial concise text with spaces to obtain the concise text.

[0006] Optionally, the samples in the sample data set are processed to obtain the concept words of the positive samples, including: using the pre-trained language model to process the essential text corresponding to the positive samples according to a preset second prompt engineering template to obtain multiple concept word data, wherein, when there are spaces and line breaks before and after each concept word data, the spaces and line breaks are deleted, and when there are multiple consecutive line breaks before and after each concept word data, the multiple consecutive line breaks are replaced with spaces; when the multiple concept word data are a line of data, using preset symbols to segment the data corresponding to the multiple concept words to obtain multiple concept words; when the multiple concept word data are in multiple lines, the concept word data in each line are respectively determined as a concept word.

[0007] Optionally, the target operation information recommended to the user is determined based on the vectors corresponding to the concept words and the essential text, including: eliminating non-phrase parts of multiple concept words, and determining the frequency-inverse document scores of the multiple concept words in the positive sample; selecting multiple concept words with the highest frequency-inverse document scores from the multiple concept words, and determining them as target concept words; converting the positive sample text set and the negative sample text set in the essential text into a positive sample vector set and a negative sample vector set, respectively, and performing L2 normalization on the vectors in the positive sample vector set and the vectors in the negative sample vector set; determining the target operation information based on the target concept words, the positive sample vector set and the negative sample vector set.

[0008] Optionally, the target operation information is determined based on the target concept word, the positive sample vector set and the negative sample vector set, including: obtaining multiple web pages containing operation information according to the target concept word; filtering the multiple web pages, eliminating web pages containing preset sensitive words and advertising information from the multiple web pages, and obtaining filtered web pages; processing the filtered web pages to obtain essential texts corresponding to the filtered web pages; converting the essential texts corresponding to the filtered web pages into text vectors; comparing the text vectors with the positive sample vector set and the negative sample vector set to obtain target scores for the filtered web pages, selecting multiple web pages with the highest target scores from the filtered web pages as multiple web pages corresponding to the target operation information; and determining the multiple web pages corresponding to the target operation information as recommended web pages.

[0009] Optionally, the text vector is compared with the positive sample vector set and the negative sample vector set to obtain the score of the filtered web page, including: obtaining the text vector and comparing it with the positive sample vector set and the negative sample vector set, selecting a preset number of target positive sample vectors and a preset number of target negative sample vectors with the highest similarity from the positive sample vector set and the negative sample vector set, respectively, wherein, when the similarity between the text vector and a vector in the positive sample vector set or a vector in the negative sample vector set is greater than a preset similarity threshold, determining that the web page corresponding to the text vector is not the recommended web page; and determining the difference between the similarity score between the text vector and the preset number of target positive sample vectors and the similarity score between the text vector and the preset number of target negative sample vectors as the target score of the filtered web page.

[0010] Optionally, the method also includes: upon receiving a new positive sample and a new negative sample, determining a first similarity between the new positive sample and samples in the positive sample set and a second similarity between the new negative sample and samples in the negative sample set; when the first similarity is less than a first preset similarity threshold, adding the new positive sample to the positive sample set and deleting the original sample from the positive sample set; when the second similarity is greater than a second preset similarity threshold, adding the new negative sample to the negative sample set and deleting the original sample from the negative sample set.

[0011] According to another aspect of an embodiment of the present application, an operation information recommendation device is also provided, including: an acquisition module, used to acquire a sample data set, the sample data set including: a pre-labeled positive sample set and a negative sample set, wherein the positive sample is selected web page data, and the negative sample is unselected web page data; a processing module, used to process samples in the sample data set to obtain a concise text for characterizing key information of the sample and a conceptual word for the positive sample, wherein the conceptual word is at least used to characterize content preferred by the user; a recommendation module, used to determine the target operation information to be recommended based on the vectors corresponding to the conceptual word and the concise text.

[0012] According to another aspect of the embodiment of the present application, a computer device is also provided, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned operation information recommendation method.

[0013] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned operation information recommendation method by running the computer program.

[0014] According to another aspect of the embodiments of the present application, a computer program product is also provided, including computer instructions, which implement the above-mentioned operation information recommendation method when executed by a processor.

[0015] In an embodiment of the present application, a sample data set is obtained, wherein the sample data set includes: a pre-labeled positive sample set and a negative sample set, wherein the positive sample is selected web page data, and the negative sample is unselected web page data; samples in the sample data set are processed to obtain concise text for characterizing key information of the sample and conceptual words for the positive sample, wherein the conceptual words are at least used to characterize content preferred by the user; the target operation information to be recommended is determined based on vectors corresponding to the conceptual words and the concise text, thereby achieving the purpose of using the data of interest selected by the user for recommendation and avoiding the use of a large amount of training data, thereby achieving the technical effect of improving and simplifying the recommendation model training process, and further solving the technical problem in the related technology that the model training process is relatively cumbersome due to the need for a large amount of manual feature engineering. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 is a hardware structure block diagram of a computer terminal for implementing an operation information recommendation method according to an embodiment of the present application;

[0018] Figure 2 is a flow chart of an operation information recommendation method according to an embodiment of the present application;

[0019] Figure 3 is a flow chart of a method for cold starting an operation information recommendation system according to an embodiment of the present application;

[0020] Figure 4 is a flow chart of another operation information recommendation method according to an embodiment of the present application;

[0021] Figure 5 is a flow chart of another operation information recommendation method according to an embodiment of the present application;

[0022] Figure 6 It is a structural diagram of an operation information recommendation device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] The information collected in the embodiments of the present application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or reject automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0026] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0027] LLM: Language Language Model, language representation model or pre-trained language model. It refers to a neural network model that obtains general language representation through self-supervised learning on a large-scale text corpus. The main features of LLM include: Pre-training: Pre-training on a large amount of unlabeled text data through self-supervised objectives to learn language representation. Common pre-training objectives include Masked Language Modeling (Masked LM) and Next Sentence Prediction (NextSentence Prediction). Language Representation: During the pre-training process, LLM learns the distributed vector representation of the language, which can represent the semantic information of words, sentences and even paragraphs. These language representations are very helpful for downstream NLP tasks. Fine-tuning: Based on pre-training, LLM can be transferred to downstream NLP tasks through simple fine-tuning to achieve the most advanced results. The larger the model size, the better the effect: With the improvement of computing power, LLM models are getting larger and larger, from millions of parameters to tens of billions of parameters. The larger the model size, the richer the learned language representation and the better the effect. Typical LLM models include GPT, BERT, RoBERTa, etc. They are all pre-trained on a large amount of text to obtain language representation, and then applied to downstream NLP tasks. LLM has become an important basic technology in the field of natural language processing.

[0028] Milvus: One of the vector libraries that supports fast retrieval of large numbers of vectors and scalars.

[0029] Assistant: This management software integrates functions such as crawlers, public account management, and group robot management.

[0030] Prompt Engineering: refers to the process of optimizing the prompt design when using large language models (such as GPT-3, etc.).

[0031] Cosin: Cosine similarity.

[0032] Context: The text content that needs to be processed.

[0033] Positive samples: content that users are very interested in, including network security, network failures, and cutting-edge technology information.

[0034] Negative samples: content that users are not interested in.

[0035] Collaborative Filtering: It is a recommendation algorithm based on collaboration. It predicts the content or products that the target users may like by analyzing the preferences and behavior patterns of different users.

[0036] Recall Ranking: refers to the process of sorting the list after the recommendation system generates a candidate recommendation list.

[0037] Pipeline: Usually used to represent the entire workflow of generating recommendation results. It breaks down complex recommendation tasks into multiple modular subtasks for pipeline processing.

[0038] Cold Start: refers to the situation where the recommendation system is unable to make accurate recommendations when there is insufficient user or product data.

[0039] Feature Engineering: A critical step in machine learning and recommendation systems. It refers to extracting, transforming and constructing more valuable features from raw data to improve the model's expressiveness and prediction accuracy.

[0040] Association rule recommendation: a recommendation technology based on user behavior association analysis. Its core idea is to use the co-occurrence rules between different products to recommend potentially related products based on the user's historical behavior.

[0041] Deep learning recommendation: a method for modeling recommendation systems using deep neural networks.

[0042] Robustness: refers to the ability of a system or algorithm to maintain stability and performance in a complex and changing environment.

[0043] TF-IDF (Term Frequency-Inverse Document Frequency): A commonly used text feature extraction technology, mainly used in information retrieval and text mining to evaluate the importance of a word to a document in a document set or corpus.

[0044] In order to solve the problems existing in the related art, the present application embodiment provides an operation information recommendation method, which can be run on Figure 1 In the computer terminal shown, the computer terminal is explained below.

[0045] The operation information recommendation method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing an operation information recommendation method. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0046] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0047] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the operation information recommendation method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned operation information recommendation method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0048] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0049] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0050] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0051] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of an operation information recommendation method. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0052] Figure 2 is a flow chart of an operation information recommendation method according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0053] Step S202, obtaining a sample data set, wherein the sample data set includes: a pre-labeled positive sample set and a negative sample set, wherein the positive sample is selected web page data, and the negative sample is unselected web page data;

[0054] In step S202, in the case of the first recommendation, the positive sample may be a historical network security report or a historical network failure report.

[0055] Step S204, processing the samples in the sample data set to obtain the concise text used to characterize the key information of the samples and the concept words of the positive samples, wherein the concept words are at least used to characterize the content preferred by the user;

[0056] In step S204, the samples in the sample data set may be processed using an LLM model.

[0057] Step S206: determining the target operation information to be recommended based on the vectors corresponding to the concept words and the essential text.

[0058] Through the above steps S202 to S206, a sample data set is obtained, wherein the sample data set includes: a pre-labeled positive sample set and a negative sample set, wherein the positive sample is selected web page data, and the negative sample is unselected web page data; the samples in the sample data set are processed to obtain a concise text for characterizing the key information of the sample and a conceptual word for the positive sample, wherein the conceptual word is at least used to characterize the content preferred by the user; the target operation information to be recommended is determined based on the vectors corresponding to the conceptual word and the concise text, thereby achieving the purpose of using the user-selected data of interest for recommendation and avoiding the use of a large amount of training data, thereby achieving the technical effect of improving and simplifying the training process of the recommendation model, and further solving the technical problem in the related technology that the model training process is relatively cumbersome due to the need for a large amount of manual feature engineering. The following is a detailed description.

[0059] In some embodiments of the present application, the recommendation of operation information can be completed through the LLM recommendation system, wherein the LLM recommendation system includes: LLM essential processing module, LLM filtering processing module, LLM concept word processing module, LLM model vectorization processing module, and LLM recommendation processing module.

[0060] In the technical solution provided in step S204 of the above-mentioned operation information recommendation method, the specific process of processing the samples in the sample data set to obtain the concise text for characterizing the key information of the samples is as follows: obtaining the number of text words corresponding to the samples in the sample data set; when the number of text words is less than a first number, directly terminating the sample processing process; when the number of text words is greater than the first number and less than the second number, determining the samples in the sample data set as the concise text; when the number of text words is greater than the second number, using a pre-trained language model to process the samples in the sample data set according to a preset first prompt engineering template to obtain the initial concise text; replacing the line break symbols in the initial concise text with spaces to obtain the concise text.

[0061] For example: the LLM essentials processing module is used to process the samples in the sample data set to obtain essential texts.

[0062] Specifically, taking the first number as 50 and the second number as 250 as an example, if the number of words in the text corresponding to the sample is less than 50, the processing of the sample is directly terminated, because a text that is too short may not contain enough information for concise extraction. In the case where the text is between 50 and 250 words, no processing is done and the original text is used directly. In the case where the text exceeds 250 words, the text is processed. Taking the preset first prompt project template as "The content of the article is as follows: {context}, please answer the question based on the above article content: Output a 300-word summary of the article" as an example, the concise text is output.

[0063] It can be understood that {context} is used to represent the text corresponding to the sample.

[0064] Optionally, the carriage return and line feed characters in the essential text may be replaced with spaces.

[0065] After obtaining the essential text, the samples in the sample data set are processed to obtain the specific process of the concept words of the positive sample as follows: the essential text corresponding to the positive sample is processed by the pre-trained language model according to the preset second prompt engineering template to obtain multiple concept word data, wherein, when there are spaces and line breaks before and after each concept word data, the spaces and line breaks are deleted, and when there are multiple continuous line breaks before and after each concept word data, the multiple continuous line breaks are replaced with spaces; when the multiple concept word data are a line of data, the data corresponding to the multiple concept words are segmented by a preset symbol to obtain multiple concept words; when the multiple concept word data are in multiple lines, the concept word data in each line are respectively determined as a concept word.

[0066] Take the second prompt project template as: "The content of the article is as follows: {context}, please answer the question based on the above article content: output five important keywords" as an example, and output multiple concept word data, in which the LLM model can be called to send the constructed request to the cloud network LLM model. The model reads the text content corresponding to the positive sample, understands the prompt requirements, and generates five words representing the core concepts of the text based on its understanding of the text. Clean the concept word data output by the LLM model to ensure that the format of the concept words is unified and clear. The specific operations include: removing the spaces, carriage returns and line feeds that may exist before and after the concept words to keep the concept words clean and tidy. Replace two or more consecutive carriage returns and line feeds with a space to avoid processing problems caused by inconsistent formats. Format the concept words: If the output concept word data has only one line of data, use a specific delimiter (such as a comma, semicolon, etc.) to separate the concept words into independent entries. The choice of delimiters should take into account the common separation habits of Chinese texts, such as using delimiters. If the concept word data is multiple lines, each line is regarded as an independent concept word. When processing, it is necessary to remove non-concept word information such as row number and sequence number from each line to ensure that each line contains only one concept word. Finally, the concept word information is organized into an array format. Each element in the array corresponds to a concept word. This format is convenient for subsequent calculations and processing, such as storage in a database, vectorization, or matching with a user interest model.

[0067] Through these steps, the recommendation method provided in the embodiment of the present application can accurately and efficiently extract key words (concept words) representing the core themes and concepts from the text. These words will then be used to build a user preference model, construct a vector space, and perform personalized recommendation calculations of the recommendation system, thereby improving the recommendation quality and user experience.

[0068] In some embodiments of the present application, the target operation information recommended to the user is determined based on the vectors corresponding to the concept words and the essential text, including: eliminating non-phrase parts of multiple concept words, and determining the word frequency-inverse document score of the multiple concept words in the positive sample set; selecting the multiple concept words with the highest scores from the multiple concept words, and determining them as target concept words; converting the positive sample text set and the negative sample text set in the essential text into a positive sample vector set and a negative sample vector set, respectively, and performing L2 normalization on the vectors in the positive sample vector set and the vectors in the negative sample vector set; determining the target operation information based on the target concept words, the positive sample vector set and the negative sample vector set.

[0069] It should be noted that the target operation information can be cloud network operation information that users are interested in. Among them, cloud network operation information refers to relevant data and messages in the fields of cloud computing and network operation, including system status, performance indicators, security events, fault reports, operation and maintenance operations, technology updates, user activities, etc.

[0070] For example: The LLM concept word processing module is used to perform collaborative filtering on multiple concept words, including: non-phrase filtering and TF-IDF filtering. Specifically, non-phrase filtering includes: Step 1: Concept word acquisition. First, based on the cloud network LLM model, a series of concept words are extracted from positive samples. These concept words may contain single words or phrases composed of multiple words. Step 2: Phrase identification. Check the list of extracted concept words to identify possible phrases or expressions. In natural language processing, a phrase is usually a combination of two or more words closely related and having a specific meaning, such as "network security", "network failure", etc. Step 3: Independent word retention. For the identified phrases or expressions, break them down into independent words. For example, "network security" may be broken down into "network" and "security", but considering the importance of "network security" as an independent concept, the overall phrase is generally retained, and only those phrases without clear meaning or representativeness are broken down. Step 4: Filtering and retention. For the concept words that have been broken down or confirmed as independent words, retain them, and filter out those parts that belong to phrases or expressions but have no independent meaning. For example, if "network" appears in both the phrases "network security" and "network construction", and the meaning of "network" alone is not sufficient to represent a specific field or concept, the word "network" may be filtered out. Step 5: Check and correction. Check the list of retained concept words to ensure that each concept word is independent and has a clear meaning, and correct possible errors or omissions. For example, some common function words (such as "of", "and", "in", etc.) may be wrongly retained and need to be corrected manually or automatically. Step 6: Output the updated concept word list. After completing the above steps, output the updated concept word list to ensure that each concept word in the list is an independent word and can accurately reflect the core concepts and fields of the text.

[0071] TF-IDF filtering includes: Step 1: Sorting of concept word scores According to the calculated TF-IDF scores, sort the concept word list in descending order to ensure that the concept word with the highest score is at the front. Step 2: Concept word selection Determine the number of concept words K to be retained. Usually K is a small value to ensure the simplicity and effectiveness of the model. Select the first K concept words from the sorted concept word list. Step 3: Update the concept word list Delete the concept words with scores lower than the first K, and only keep the K concept words with the highest scores as the final concept word list. This final concept word list will be used to build and update the user interest model to better match user preferences in the recommendation system.

[0072] It can be understood that K is a positive integer.

[0073] In some embodiments of the present application, the specific steps for determining the target operation information based on the target concept word, the positive sample vector set, and the negative sample vector set are as follows: obtain multiple web pages containing operation information according to the target concept word; filter the multiple web pages, and remove web pages containing preset sensitive words and advertising information from the multiple web pages to obtain filtered web pages; process the filtered web pages to obtain the essential text corresponding to the filtered web pages; convert the essential text corresponding to the filtered web pages into a text vector; compare the text vector with the positive sample vector set and the negative sample vector set to obtain the target score of the filtered web page, and select multiple web pages with the highest target scores from the filtered web pages to determine as the multiple web pages corresponding to the target operation information; determine the multiple web pages corresponding to the target operation information as recommended web pages.

[0074] The specific vectorization process is as follows: the LLM model vectorization processing module is used to convert positive samples and negative samples into vector representations. Specifically, the positive samples and negative samples are converted into N-dimensional vector representations, and then the L2 norms of the positive sample vectors and the negative sample vectors are obtained so that the L2 norm of each vector is equal to 1.

[0075] The LLM filtering processing module can be used to filter multiple web pages. Through LLM filtering and sensitive word library joint filtering, some invalid web pages can be filtered, such as web pages containing advertising information, sensitive information, etc. Remove irrelevant content and improve information quality. Advertisements, irrelevant links, etc. will introduce noise and reduce the density of effective information. Avoid recommending invalid content. Web pages containing a large number of advertisements or useless information are not suitable for recommendation to users. Filter out web pages containing sensitive words.

[0076] In some embodiments of the present application, the text vector is compared with the positive sample vector set and the negative sample vector set to obtain the score of the filtered web page, including: obtaining the text vector and comparing it with the positive sample vector set and the negative sample vector set, and selecting a preset number of target positive sample vectors and a preset number of target negative sample vectors with the highest similarity from the positive sample vector set and the negative sample vector set, respectively, wherein, when the similarity between the text vector and a vector in the positive sample vector set or a vector in the negative sample vector set is greater than a preset similarity threshold, determining that the web page corresponding to the text vector is not the recommended web page; and determining the difference between the similarity score between the text vector and the preset number of target positive sample vectors and the similarity score between the text vector and the preset number of target negative sample vectors as the target score of the filtered web page.

[0077] Specifically, the normalized text vector is stored in the Milvus vector library, and together with the previously stored positive and negative sample text vectors, a vector space is formed. The result range of calculating the similarity between the text vector and the positive and negative sample vectors in the vector library is [-1, 1]. Among them, -1 means that the directions of the two vectors are completely opposite, and 1 means that the directions of the two vectors are exactly the same. In practical applications, in order to more intuitively represent the similarity, the result can be mapped to the [0, 2] interval.

[0078] Recall the top W positive samples and top W negative samples with the highest similarity scores with the new text vector from the Milvus vector library. Check whether there are samples with similarity scores with the text vector exceeding the preset similarity threshold among the recalled positive and negative samples. If there are, filter out the text directly or adjust the recommendation to avoid duplicate recommendations or exclude content that the user is not interested in.

[0079] For example: when any distance among the first W recalled positive samples is greater than the preset similarity threshold, the system will directly filter the web page. This is mainly because the user may have browsed very similar content. Similarly, when any distance among the first W recalled negative samples is greater than the preset similarity threshold, the system will directly filter the web page. The main reason for this strategy is to meet the user's needs to exclude certain content. For the remaining texts, calculate their similarity scores with the user's positive and negative feedback vectors, calculate the target scores of the texts based on the similarity scores, sort them in descending order by the target scores, and select N texts as the final recommendation results. The URLs of the N texts with the highest target scores and related information are returned to the interactive assistant, which presents them to the user.

[0080] Taking W as 10 as an example, the specific calculation method of the score is as follows:

[0081]

[0082] In the formula, score represents the target score, Represents the average similarity between the first 10 positive sample vectors and the text vector, Represents the average similarity between the first 10 negative sample vectors and the text vector.

[0083] The average of the sum of similarity scores between the webpage text (text vector) to be recommended and the first 10 positive samples (content that users have liked in the past) is calculated, and then the average of the sum of similarity scores between the first 10 negative samples (content that users have disliked in the past) is calculated, and finally the difference between the two averages is calculated. This difference is used as the recommendation score to measure the degree of match between the recommended item and the user's interests. If the score is high, it means that the recommended item is more in line with the user's interests; if the score is low, it means that the recommended item is less matched with the user's interests.

[0084] It should be noted that a positive and negative sample feedback method is also designed in an embodiment of the present application, including: when a new positive sample and a new negative sample are received, a first similarity between the new positive sample and the sample in the positive sample set and a second similarity between the new negative sample and the sample in the negative sample set are determined respectively; when the first similarity is less than a first preset similarity threshold, the new positive sample is added to the positive sample set, and the original sample is deleted from the positive sample set; when the second similarity is greater than a second preset similarity threshold, the new negative sample is added to the negative sample set, and the original sample is deleted from the negative sample set.

[0085] Specifically, the positive sample vector library: calculates the cosine similarity between the vector of the newly added positive sample and all sample vectors in the positive sample set. If the similarity is less than a first preset similarity threshold threshold_pos (eg, 0.2), the corresponding original positive sample information is deleted from the positive sample set.

[0086] Negative sample vector library: Calculate the cosine similarity between the vector of the newly added negative sample and the content_vector of all samples in the negative sample set. If the similarity is greater than the second preset similarity threshold threshold_neg (for example, 0.3), the corresponding original negative sample information is deleted from the negative sample set.

[0087] Figure 3 A method for cold starting an operation information recommendation system is shown. Figure 3 As shown, including:

[0088] Step 1, data acquisition step: establish a cloud network operation knowledge base (such as network security reports, network failure reports, etc.); use a crawler assistant to crawl Internet web pages, and let users mark positive and negative samples.

[0089] Step 2, LLM essence processing step: Use the cloud network operation knowledge base to retrain the LLM pre-trained model to obtain the cloud network LLM. Both the knowledge base and the web page should use LLM to extract the essence of the text, obtain the main information of the text, and clean up the unnecessary information fed back by LLM.

[0090] Step 3, LLM concept word processing step: The positive sample needs to use LLM to extract K concept words of the text, and then perform collaborative filtering (non-phrase filtering + TF-IDF filtering) after extracting the concept words.

[0091] Step 4, LLM model vectorization processing step: convert the generated concise text into an N-dimensional vector representation, and then obtain the L2 norm of the N-dimensional vector.

[0092] Step 5: Before the positive and negative samples are stored for the first time, two collections are created in the Milvus vector library. At the same time, indexes are created for the content_vector fields of the two collections to optimize vector search performance.

[0093] Among them, Table 1 shows a positive sample data:

[0094] Table 1

[0095] Field Name Field Explanation Data Types id Unique identifier (primary key) INT64 url A string field that stores the website URL VARCHAR content_vector 1024-dimensional essential vector generated by Cloud Network LLM FLOAT_VECTOR(1024) concepts Stores the string array of K concept words generated by Cloud Network LLM VARCHAR

[0096] Table 2 shows a negative sample data:

[0097] Table 2

[0098] Field Name Field Explanation Data Types id Unique identifier (primary key) INT64 url A string field that stores the website URL VARCHAR content_vector 1024-dimensional essential vector generated by Cloud Network LLM FLOAT_VECTOR(1024)

[0099] Step 6, sample vector and concept word storage step: the concise vector, concept word and other information obtained after positive sample processing are stored in the Milvus positive sample table space; the concise vector obtained after negative sample processing is stored in the Milvus negative sample table space.

[0100] The present application also provides an operation information recommendation method, such as Figure 4As shown, it includes: input data: receiving M web pages crawled by the "assistant" according to the concept words; LLM filtering processing module: filtering some invalid web pages through LLM filtering and sensitive word library joint filtering, such as web pages containing advertising information, sensitive information, etc. The purpose of filtering invalid web pages is to remove irrelevant content and improve information quality. Advertisements, irrelevant links, etc. will introduce noise and reduce the density of effective information. Avoid recommending invalid content. Web pages containing a large number of advertisements or useless information are not suitable for recommendation to users. Avoid legal risks. Filter out web pages containing sensitive words. Improve user experience. Users do not want to see too many advertisements or irrelevant content, and filtering invalid web pages can provide a better user experience. LLM concise processing module: see (Process 1 LLM concise processing module), where K web pages are concisely processed. Recommendation processing module: output personalized recommendation results based on the similarity between the text vector and the user's positive and negative feedback vectors.

[0101] The present application also provides another operation information recommendation method, such as Figure 5 As shown, including:

[0102] Step 1, obtain cloud network news information: The crawler assistant automatically crawls various news information from the Internet, and outputs the recommended cloud network information through the recommendation module based on the cloud network LLM, and pushes it to the user through the interactive assistant.

[0103] Step 2: Get positive and negative feedback data: Receive a set of positive and negative samples from user feedback through the interactive assistant, and perform a light cleaning operation on the set.

[0104] Step 3, LLM essential processing module: Both positive and negative samples use LLM to extract the essentials of the text, obtain the main information of the text, and clean up unnecessary feedback information from LLM.

[0105] Step 4, LLM concept word processing module: The positive sample needs to use LLM to extract K concept words from the text, and then perform collaborative filtering after the concept words are extracted.

[0106] Step 5, LLM model vectorization processing module: convert the concise text generated in step 2 into an N-dimensional vector representation and perform normalization processing.

[0107] Step 6, positive and negative sample Milvus vector library update mechanism: For the insertion of new samples: new positive samples are added to the positive sample set, and new negative samples are added to the negative sample set.

[0108] Positive sample vector library: Calculate the cosine similarity between the vector of the newly added positive sample and all sample vectors in the positive sample set. If the similarity is less than the first preset similarity threshold threshold_pos (eg, 0.2), the corresponding original positive sample information is deleted from the positive sample set.

[0109] Negative sample vector library: Calculate the cosine similarity between the vector of the newly added negative sample and the content_vector of all samples in the negative sample set. If the similarity is greater than the second preset similarity threshold threshold_neg (for example, 0.3), the corresponding original negative sample information is deleted from the negative sample set.

[0110] Figure 6 An operation information recommendation device according to an embodiment of the present application includes:

[0111] The acquisition module 60 is used to acquire a sample data set, wherein the sample data set includes: a pre-labeled positive sample set and a negative sample set, wherein the positive sample is the selected web page data, and the negative sample is the unselected web page data;

[0112] A processing module 62 is used to process the samples in the sample data set to obtain a concise text used to characterize key information of the samples and a concept word of the positive sample, wherein the concept word is used to characterize at least the content preferred by the user;

[0113] The recommendation module 64 is used to determine the target operation information to be recommended based on the vectors corresponding to the concept words and the essential text.

[0114] Through the above-mentioned operation information recommendation device, a sample data set is obtained, and the sample data set includes: a pre-labeled positive sample set and a negative sample set, wherein the positive sample is selected web page data, and the negative sample is unselected web page data; the samples in the sample data set are processed to obtain a concise text for characterizing the key information of the sample and a conceptual word for the positive sample, and the conceptual word is at least used to characterize the content preferred by the user; based on the vectors corresponding to the conceptual word and the concise text, the target operation information to be recommended is determined, thereby achieving the purpose of using the user-selected data of interest for recommendation and avoiding the use of a large amount of training data, thereby achieving the technical effect of improving and simplifying the recommendation model training process, and then solving the technical problem in the related technology that the model training process is relatively cumbersome due to the need for a large amount of manual feature engineering.

[0115] The processing module 62 includes: a processing submodule, which is used to process the samples in the sample data set to obtain a concise text for characterizing the key information of the samples, including: obtaining the number of text words corresponding to the samples in the sample data set; when the number of text words is less than a first number, directly terminating the processing process of the samples; when the number of text words is greater than the first number and less than a second number, determining the samples in the sample data set as the concise text; when the number of text words is greater than the second number, using a pre-trained language model to process the samples in the sample data set according to a preset first prompt engineering template to obtain the initial concise text; replacing the line break symbols in the initial concise text with spaces to obtain the concise text.

[0116] The processing submodule includes a processing unit, which is used to process the samples in the sample data set to obtain the concept words of the positive sample, including: using the pre-trained language model to process the essential text corresponding to the positive sample according to the preset second prompt engineering template to obtain multiple concept word data, wherein, when there are spaces and line breaks before and after each concept word data, the spaces and line breaks are deleted, and when there are multiple continuous line breaks before and after each concept word data, the multiple continuous line breaks are replaced with spaces; when the multiple concept word data are a line of data, the data corresponding to the multiple concept words are segmented by a preset symbol to obtain multiple concept words; when the multiple concept word data are in multiple lines, the concept word data in each line are respectively determined as a concept word.

[0117] The recommendation module 64 includes: a recommendation submodule, which is used to determine the target operation information recommended to the user based on the vectors corresponding to the concept words and the essential text, including: eliminating non-phrase parts in multiple concept words, and determining the word frequency-inverse document score of the multiple concept words in the positive sample set; selecting the multiple concept words with the highest scores from the multiple concept words, and determining them as target concept words; converting the positive sample text set and the negative sample text set in the essential text into a positive sample vector set and a negative sample vector set, respectively, and performing L2 normalization on the vectors in the positive sample vector set and the vectors in the negative sample vector set; determining the target operation information based on the target concept words, the positive sample vector set and the negative sample vector set.

[0118] The recommendation submodule includes: a recommendation unit, which is used to determine the target operation information based on the target concept word, the positive sample vector set and the negative sample vector set, including: obtaining multiple web pages containing operation information according to the target concept word; filtering the multiple web pages, eliminating web pages containing preset sensitive words and containing advertising information from the multiple web pages, and obtaining filtered web pages; processing the filtered web pages to obtain the essential text corresponding to the filtered web pages; converting the essential text corresponding to the filtered web pages into a text vector; comparing the text vector with the positive sample vector set and the negative sample vector set to obtain the target score of the filtered web page, selecting multiple web pages with the highest target score from the filtered web pages to determine as the multiple web pages corresponding to the target operation information; determining the multiple web pages corresponding to the target operation information as recommended web pages.

[0119] The recommendation unit includes: a scoring subunit, which is used to compare the text vector with the positive sample vector set and the negative sample vector set to obtain the score of the filtered web page, including: obtaining the text vector and comparing it with the positive sample vector set and the negative sample vector set, selecting a preset number of target positive sample vectors and a preset number of target negative sample vectors with the highest similarity from the positive sample vector set and the negative sample vector set respectively, wherein, when the similarity between the text vector and a vector in the positive sample vector set or a vector in the negative sample vector set is greater than a preset similarity threshold, determining that the web page corresponding to the text vector is not the recommended web page; and determining the difference between the similarity score between the text vector and the preset number of target positive sample vectors and the similarity score between the text vector and the preset number of target negative sample vectors as the target score of the filtered web page.

[0120] The operation information recommendation device also includes: an updating module, which is used to determine, when receiving a new positive sample and a new negative sample, a first similarity between the new positive sample and the sample in the positive sample set and a second similarity between the new negative sample and the sample in the negative sample set; when the first similarity is less than a first preset similarity threshold, the new positive sample is added to the positive sample set and the original sample is deleted from the positive sample set; when the second similarity is greater than a second preset similarity threshold, the new negative sample is added to the negative sample set and the original sample is deleted from the negative sample set.

[0121] It should be noted that Figure 6 The operation information recommendation device shown is used to perform Figure 2The operation information recommendation method shown in the figure, therefore the relevant explanations in the above operation information recommendation method are also applicable to the operation information recommendation device, and will not be repeated here.

[0122] An embodiment of the present application also provides a computer device, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor is connected to the memory and is used to execute the above-mentioned operation information recommendation method.

[0123] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned operation information recommendation method by running the computer program.

[0124] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the operation information recommendation method in the present application.

[0125] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0126] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0129] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0130] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), mobile hard disk, disk or optical disk, etc., which can store program code.

[0131] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for recommending operation information, characterized in that: include: Acquire a sample data set, wherein the sample data set includes: a pre-labeled positive sample set and a negative sample set, wherein the positive sample is selected web page data, and the negative sample is unselected web page data; Processing the samples in the sample data set to obtain concise text for characterizing key information of the samples and concept words for the positive samples, wherein the concept words are at least used to characterize content preferred by the user; The target operation information to be recommended is determined based on the vectors corresponding to the concept words and the essential text.

2. The method according to claim 1, characterized in that The samples in the sample data set are processed to obtain concise text for representing key information of the samples, including: Obtain the number of words in the text corresponding to the sample in the sample data set; When the number of characters in the text is less than the first number, directly terminating the processing of the sample; When the number of characters in the text is greater than the first number and less than the second number, determining the sample in the sample data set as the concise text; When the number of words in the text is greater than the second number, a pre-trained language model is used to process the samples in the sample data set according to a preset first prompt engineering template to obtain an initial concise text; The line break symbols in the initial concise text are replaced with spaces to obtain the concise text.

3. The method according to claim 2, characterized in that Processing the samples in the sample data set to obtain the concept words of the positive samples includes: The pre-trained language model is used to process the essential text corresponding to the positive sample according to the preset second prompt engineering template to obtain a plurality of concept word data, wherein when there are spaces and line breaks before and after each concept word data, the spaces and line breaks are deleted, and when there are multiple continuous line breaks before and after each concept word data, the multiple continuous line breaks are replaced with spaces; In the case where the plurality of concept word data is a row of data, the data corresponding to the plurality of concept words are segmented using a preset symbol to obtain a plurality of concept words; When the plurality of concept word data are in a plurality of lines respectively, the concept word data in each line is determined as a concept word respectively.

4. The method according to claim 1, characterized in that: Determining target operation information recommended to the user based on the vectors corresponding to the concept words and the essential text includes: Eliminating non-phrase parts of the plurality of concept words, and determining a word frequency-inverse document score of the plurality of concept words in the positive sample; Selecting a plurality of concept words with the highest frequency-inverse document scores from the plurality of concept words and determining them as target concept words; Converting the positive sample text set and the negative sample text set in the essential text into a positive sample vector set and a negative sample vector set respectively, and performing L2 normalization on the vectors in the positive sample vector set and the vectors in the negative sample vector set; The target operation information is determined based on the target concept word, the positive sample vector set, and the negative sample vector set.

5. The method according to claim 4, characterized in that Determining the target operation information based on the target concept word, the positive sample vector set, and the negative sample vector set includes: Acquire multiple web pages containing operation information according to the target concept word; Filtering the multiple web pages, removing web pages containing preset sensitive words and advertising information from the multiple web pages, and obtaining filtered web pages; Processing the filtered webpage to obtain the essential text corresponding to the filtered webpage; Converting the essential text corresponding to the filtered web page into a text vector; The text vector is compared with the positive sample vector set and the negative sample vector set to obtain the score of the filtered web page, and a plurality of web pages with the highest target scores are selected from the filtered web pages to be determined as the plurality of web pages corresponding to the target operation information; A plurality of web pages corresponding to the target operation information are determined as recommended web pages.

6. The method according to claim 5, characterized in that The text vector is compared with the positive sample vector set and the negative sample vector set to obtain a target score of the filtered web page, including: Obtaining the text vector and comparing it with the positive sample vector set and the negative sample vector set, selecting a preset number of target positive sample vectors and a preset number of target negative sample vectors with the highest similarity from the positive sample vector set and the negative sample vector set, respectively, wherein, when the similarity between the text vector and a vector in the positive sample vector set or a vector in the negative sample vector set is greater than a preset similarity threshold, determining that the webpage corresponding to the text vector is not the recommended webpage; The difference between the similarity score between the text vector and the preset number of target positive sample vectors and the similarity score between the text vector and the preset number of target negative sample vectors is determined as the target score of the filtered web page.

7. The method according to claim 1, characterized in that The method further comprises: When a new positive sample and a new negative sample are received, respectively determining a first similarity between the new positive sample and samples in the positive sample set and a second similarity between the new negative sample and samples in the negative sample set; When the first similarity is less than a first preset similarity threshold, adding the new positive sample to the positive sample set, and deleting the original sample from the positive sample set; When the second similarity is greater than a second preset similarity threshold, the new negative sample is added to the negative sample set, and the original sample is deleted from the negative sample set.

8. An operation information recommendation device, characterized in that: include: An acquisition module is used to acquire a sample data set, wherein the sample data set includes: a pre-labeled positive sample set and a negative sample set, wherein the positive sample is selected web page data, and the negative sample is unselected web page data; A processing module, used for processing samples in the sample data set to obtain concise texts used to characterize key information of the samples and concept words of the positive samples, wherein the concept words are used to characterize at least content preferred by the user; A recommendation module is used to determine the target operation information to be recommended based on the vectors corresponding to the concept words and the essential text.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; The processor is connected to the memory, and is used to execute the operation information recommendation method described in any one of claims 1-8.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the operation information recommendation method described in any one of claims 1 to 8 is implemented.

Images