A model updating method, device, medium, and program product

Abstract

The present application relates to the technical field of information processing, in particular to a model updating method, device, medium and program product, the model updating method is used for electronic equipment, comprising: index data acquisition step, the evaluation index data of the first text type recognition model in the online environment is acquired, whether the first text type recognition model meets the standard is determined according to the evaluation index data;Cleaning step, in the case where the first text type recognition model does not meet the standard, the first text library is cleaned;Updating step, the first corpus is updated based on the cleaned first text library, and the second corpus is obtained;Re-training step, the model training is carried out again based on the second corpus, and the target text type recognition model is obtained.The model updating method of the present application can ensure the normal operation of the online environment, and improve the accuracy of the text type recognition model for text type recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information processing, in particular to a model updating method, device, medium and program product. BACKGROUND

[0002] There are usually a large number of text-related content security risks in mobile games, such as fraud information in chat channels, person-pulling advertisements in alliance announcements, resource trader transaction advertisements, and abusive / harassment / political messages. These messages are driven by black or gray industries, i.e. black or gray production, and are obtained through improper means to make profits. If these messages are allowed, they may cause losses to game operators or other players, and also affect the experience of players during the game.

[0003] In order to balance the game ecology and the experience of players, different solutions are provided by various manufacturers. Most of the solutions use artificial intelligence technologies such as deep learning, natural language processing (NLP) technology, semantic recognition, and word vector model to identify advertising content in game chat text. However, the accuracy of these technologies in identifying specific advertisements in games, especially game jargon (such as resource traders selling fruits and marble), is not high, and they cannot quickly capture new forms of advertising, such as branch sending, survival probes, and game-related words for new activities. Furthermore, the automatic iteration, autonomous discovery, and other aspects of existing models are not perfect, and the model training, corpus construction, sample set management, and real-time workflow are disconnected, which cannot well combine the actual needs of each game and the initial design of the data link, resulting in too many manual steps, affecting the safety and efficiency of development and online. SUMMARY

[0004] In view of the above problems of the prior art, the present application aims to provide a model updating method, device, medium and program product, which can improve the accuracy of text type recognition by a text type recognition model.

[0005] To solve the above problems, the present application provides a model updating method, comprising:

[0006] An index data acquisition step acquires evaluation index data of a first text type recognition model in an online environment, and determines whether the first text type recognition model meets the standard according to the evaluation index data. The first text type recognition model is obtained by training based on a first corpus, and the first corpus is obtained by text type annotation based on at least a first text library. The first text library includes text marked as belonging to a preset type;

[0007] a cleaning step of cleaning the first text library in a case where the first text type recognition model is substandard;

[0008] an updating step of updating the first corpus based on the cleaned first text library to obtain a second corpus;

[0009] a retraining step of retraining the model based on the second corpus to obtain a target text type recognition model.

[0010] Specifically, the cleaning step comprises:

[0011] obtaining historical sending data in a first preset time period, the historical sending data comprising a plurality of historical sending corpora and role information of sending the historical sending corpora;

[0012] matching each text in the first text library with each historical sending corpus respectively, and counting a first total number of roles corresponding to each historical sending corpus that is matched successfully;

[0013] removing texts in the first text library corresponding to a first total number of roles greater than a first preset threshold to obtain a cleaned first text library.

[0014] Specifically, the updating step comprises:

[0015] updating the annotation information of each corpus in the first corpus based on the cleaned first text library to obtain a second corpus.

[0016] Specifically, the retraining step comprises:

[0017] retraining the model based on the second corpus to obtain a second text type recognition model;

[0018] model testing on the second text type recognition model, and taking the second text type recognition model as a target text type recognition model in a case where the testing is passed.

[0019] Further, the method further comprises:

[0020] a model deployment step of deploying a preset text type recognition model to an online environment in a case where the first text type recognition model is substandard.

[0021] Further, the method further comprises:

[0022] The corpus obtaining step comprises: obtaining a third corpus by updating the first corpus based on a second text corpus and a newly added corpus set in a second preset time period, the second text corpus comprising the first text corpus and the newly added corpus set in the second preset time period, the newly added corpus set comprising texts in the second preset time period that are marked as belonging to a preset type;

[0023] The optimization training step comprises: optimizing and training the first text type recognition model based on the third corpus to obtain a target text type recognition model.

[0024] Specifically, the corpus obtaining step comprises:

[0025] obtaining a second text corpus and a newly added corpus set in a second preset time period, the second text corpus comprising the first text corpus and the newly added corpus set in the second preset time period;

[0026] text type marking each corpus in the newly added corpus set based on the second text corpus;

[0027] updating the marking information of each corpus in the first corpus based on the newly added corpus set;

[0028] adding the marked newly added corpus set to the first corpus after updating the marking information to obtain the third corpus.

[0029] Optionally, the second text corpus comprises:

[0030] obtaining a newly added corpus set in a second preset time period;

[0031] cleaning the newly added corpus set to remove mislabeled texts in the newly added corpus set;

[0032] adding the cleaned newly added corpus set to the first text corpus to obtain the second text corpus.

[0033] Optionally, the corpus obtaining step further comprises:

[0034] determining a suspected preset type text corpus in the newly added corpus set by using a suspected preset type text recognition model;

[0035] text type marking each corpus in the newly added corpus set based on the suspected preset type text corpus.

[0036] Optionally, the suspected preset type text corpus in the newly added corpus set is determined by using a suspected preset type text recognition model, comprising:

[0037] respectively calculate text distances between each corpus in the new corpus set two by two;

[0038] divide the new corpus set into at least one corpus subset according to the text distances, and the text distance between any two corpora in each corpus subset is less than a preset distance;

[0039] respectively determine whether each corpus subset is a suspected preset type text collection, and obtain an abnormality coefficient of each corpus subset determined as a suspected preset type text collection;

[0040] corpora in a corpus subset with an abnormality coefficient greater than a second preset threshold are regarded as suspected preset type text corpora.

[0041] Optionally, the step of respectively determining whether each corpus subset is a suspected preset type text collection comprises:

[0042] obtain sending situation data corresponding to the new corpus set in a third preset time period, and the sending situation data comprises role information of sending each corpus in the new corpus set;

[0043] respectively for each corpus subset, determine a total sending times of sending each corpus in the corpus subset and a second total number of roles according to the sending situation data;

[0044] calculate a sending non-repetition rate and abnormal behavior data corresponding to the corpus subset according to the sending situation data;

[0045] in a case where the total sending times, the second total number of roles, the sending non-repetition rate and the abnormal behavior data satisfy a preset condition, determine that the corpus subset is a suspected preset type text collection.

[0046] Optionally, the corpus library obtaining step further comprises:

[0047] obtain expert marked corpora in the new corpus set;

[0048] perform text type annotation on each corpus in the new corpus set based on the expert marked corpora.

[0049] Further, the method further comprises:

[0050] an online testing step of performing online testing on the target text type recognition model, and deploying the target text type recognition model to an online environment in a case where the testing is passed.

[0051] Another aspect of the present application provides an electronic device comprising a memory, a processor and a computer program stored on the memory, wherein the processor executes the computer program to implement the model updating method as described above.

[0052] Another aspect of the present application provides a computer readable storage medium having stored thereon computer programs / instructions which, when executed by a processor, implement the model updating method as described above.

[0053] Another aspect of the present application provides a computer program product comprising computer programs / instructions which, when executed by a processor, implement the model updating method as described above.

[0054] Thanks to the above technical solutions, the present application has the following beneficial effects:

[0055] According to the model optimization method of the embodiment of the present application, the effect index of the text type recognition model in the online environment is automatically monitored, in the case that the model does not meet the standard, the corpus is automatically cleaned and updated, and the text type recognition model is retrained based on the updated corpus, an automatic iteration link of the model is established, and the accuracy of the model retrained for text type recognition can be improved. Moreover, in the case that the model does not meet the standard, a bottom-line model is deployed in the online environment, which can be automatically fused when risks are encountered, ensuring the accuracy of the text type recognition model interface in the online environment and ensuring the normal operation of the online environment.

[0056] In addition, according to the model optimization method of the embodiment of the present application, in the case that the model meets the standard, the corpus is periodically added and the automatic cleaning and updating of the corpus library are improved, and then the text type recognition model is optimized and trained to obtain an optimized text type recognition model, a complete automatic iteration link of the model is established, and the precision and recall of the text type recognition model for text type recognition are further improved. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0058] Figure 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0059] Figure 2 is a flowchart of a model updating method provided by an embodiment of the present application;

[0060] Figure 3 is a flowchart of a model updating method provided by another embodiment of the present application;

[0061] Figure 4is a flowchart of a model updating method provided by another embodiment of the present application;

[0062] Figure 5 is a flowchart of a model updating method provided by another embodiment of the present application;

[0063] Figure 6 is a flowchart of a model updating method provided by another embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0065] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, device, product or apparatus including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or apparatuses.

[0066] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear and obvious, the embodiments of the present application will be further described in detail below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present application, and are not used to limit the embodiments of the present application. First, the embodiments of the present application explain the following concepts:

[0067] Black word library: store common advertising text phrases in the game, that is, black words, also known as black words. Black words can be understood as words or phrases with hidden meanings in a certain field, which are not standard Chinese expressions. Black words can be divided into game general black words, such as "add V", which means adding a WeChat friend, and game specific black words, such as "out of marble", which means selling mineral props in the game; according to the type of advertisement, it can be divided into social advertising black words, promotion advertising black words, resource advertising black words, and transaction advertising black words.

[0068] Corpus: The corpus stores the corpus needed to train the text type recognition model. The difference between the corpus and the black word is that the corpus is usually a complete sentence taken from the player's speech in the actual production environment. Each corpus in the corpus can be divided into non-advertising text, social advertising text, promotion advertising text, resource advertising text, transaction advertising text, etc. according to the text type. Each corpus in the corpus carries the annotation information of the corresponding text type.

[0069] Text type recognition model: A model trained by using a machine learning or deep learning algorithm, used to identify the text type of a piece of text. The input is a piece of text, and the output is the probability that the text is each type of advertising text.

[0070] Suspected black text model: used to identify texts that cannot be identified by the text type recognition model but have a high probability of being advertising texts. The input of the suspected black text model includes the text and the behavior characteristics other than the text, such as the sent environment information, behavior data, account information, etc. The output is the probability that the text is a suspected black text.

[0071] Black production: refers to a group or individual that manipulates network information content through manual or technical means to obtain illegal profits and destroy the network ecological order.

[0072] Edit distance: the minimum number of operations required to transform string S1 into string S2 through insertion, modification, deletion, etc. For example: S1 = abc, S2 = abf, then the edit distance d = 1 (only modify c to f).

[0073] The method provided by the embodiments of the present application can be applied to the scene of identifying the content safety hazards related to text in the game, such as fraud information in the chat channel, recruitment advertising in the alliance announcement, resource transaction advertising or abusive / huang-related messages. For example, the text type recognition model can be used to identify advertising in the player's speech in the online environment to determine whether the player's speech includes advertising text, including but not limited to social advertising text, promotion advertising text, resource advertising text, transaction advertising text, etc. With the emergence of new forms of advertising and the appearance of black production, the existing text type recognition model may not meet the needs of actual business, so the model updating method provided by the embodiments of the present application can be used to update the text type recognition model to adapt to the changing business needs.

[0074] It should be noted that the above application scenarios are only exemplary. In actual applications, the method provided by the embodiments of the present application can also be applied to other scenarios requiring model updating, such as scenarios of updating a detection model for specific information (such as sensitive information or information that is not expected to appear in a specific scenario) in a webpage, and the like. The embodiments of the present application do not limit this. In different application scenarios, the text library used for training and updating the model can be different. For example, in the scenario of identifying advertisements in player speeches in a game, the model can be trained and updated based on a black word library and a corpus. In the scenario of updating a detection model for specific information (such as sensitive information) in a webpage, the model can be trained and updated based on a specific word library (sensitive word library) and a corpus. That is, the text labeled as belonging to the preset type mentioned in the embodiments of the present application can be a black word, a specific word (sensitive word), and the like, and the first text library can be a black word library, a specific word library, and the like. The embodiments of the present application do not specifically limit this.

[0075] Specifically, in the scenario of identifying advertisements in player speeches in a game, the model trained based on the black word library and the corpus can be updated. In the scenario of updating a detection model for sensitive information in a webpage, the model trained based on the sensitive word library and the corpus can be updated. That is, in different scenarios, the first text library for training the model is different.

[0076] The method provided by the embodiments of the present application will be described in detail below by taking the scenario of identifying advertisements in player speeches in a game as an example.

[0077] In some embodiments, the game provider or operator usually deploys a text type identification model provided by a third party to an online environment to identify advertisements in player speeches in a game, and the model updating is also performed by the third party. This way causes the following problems when the text type identification model is applied to actual business: The text type identification model has low recognition accuracy for game-specific advertisements, especially for game black words (such as resource merchants selling fruits and marble); the automatic iteration, autonomous discovery, and the like of the model are not perfect, and are disconnected from the model training, corpus construction, sample set management, real-time workflow, and the like, which cannot well combine the actual needs of various games and the design intention of the data link, resulting in too many human-involved links, affecting the safety and efficiency of development and online; and new forms of advertisements, such as branch sending, survival probes, and game new activity strongly related words, cannot be quickly captured.

[0078] To solve the above problems, the embodiment of the present application provides a model updating method, device, medium and program product. The model updating method can be applied to the scene of advertisement identification of player speeches in games, so as to improve the accuracy of text type identification of a text type identification model, improve the coverage of different types of advertisements in different games and the coverage of new forms of advertisements, and can capture black production countermeasures in advance, improve the safety and efficiency of development and online.

[0079] It should be understood that the method provided by the embodiment of the present application can be executed in various electronic devices, such as mobile terminals, computer terminals, servers or similar computing devices and systems. Specifically, the electronic device can include a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the model updating method provided by the embodiment of the present application.

[0080] The memory can be used to store computer programs and modules, and the processor executes the computer programs and modules stored in the memory to perform various functions and data processing. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store operating systems, computer programs required by functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device or other volatile solid-state memory device. Accordingly, the memory can also include a memory controller to provide access for the processor to the memory.

[0081] Reference is made to the accompanying drawings Figure 1 which shows a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application. As shown in FIG. 1, the electronic device can include a processor 101, a memory 102, a communication interface 103 and a power supply 104. Figure 1As shown, the electronic device can be a server 100, which can have a large difference due to different configurations or performances, and can include one or more central processing units (CPUs) 110 (e.g., one or more processors) and a memory 130, one or more storage media 120 (e.g., one or more mass storage devices) storing computer applications 123 or data 122. Among them, the memory 130 and the storage medium 120 can be temporary storage or persistent storage. The program stored in the storage medium 120 can include one or more modules, each of which can include a series of instruction operations in the server. Further, the central processing unit 110 can be configured to communicate with the storage medium 120 to execute a series of instruction operations in the storage medium 120 on the server 100. The server 100 can also include one or more power supplies 160, one or more wired or wireless network interfaces 150, one or more input / output interfaces 140, and / or one or more operating systems 121 stored in the storage medium 120, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0082] Those skilled in the art can understand that Figure 1 The structure shown is only exemplary, and the server 100 can also include more or fewer components than Figure 1 shown in the figures, or have a different configuration than Figure 1 shown while achieving the same function.

[0083] Reference is made to the accompanying drawings Figure 2 which show the flow of the model updating method provided by an embodiment of the present application, which can be applied to the processor of an electronic device. Specifically, as shown Figure 2 , the method can include the following steps:

[0084] S210: An index data acquisition step, acquiring evaluation index data of a first text type recognition model in an online environment, and determining whether the first text type recognition model meets the standard according to the evaluation index data.

[0085] Among them, the first text type recognition model can be obtained based on a first corpus, and the first corpus can be obtained after text type annotation based on at least a first black word library (as an instance of the first text library), and the first black word library includes text marked as belonging to advertising text (as an instance of the preset type of text, hereinafter referred to as black word).

[0086] In the index data obtaining step S210, a text type recognition model being used in the online environment can be taken as the first text type recognition model to be updated, and evaluation index data of the first text type recognition model can be periodically obtained. The period for obtaining the evaluation index data can be set in advance according to actual needs, for example, can be set to 1 hour, 6 hours or 12 hours, etc., and the present embodiment does not make a specific limitation in this regard.

[0087] Specifically, the first text type recognition model can be obtained by training based on a first corpus, and the first corpus can include a plurality of corpus of annotated text types. The text type corresponding to the corpus in the first corpus can be determined according to actual needs, for example, can include non-advertising text and advertising text including but not limited to one or more of social advertising text, promotion advertising text, resource advertising text and transaction advertising text, and the present embodiment does not make a specific limitation in this regard.

[0088] In actual application, the plurality of corpus of annotated text types in the first corpus can be taken as training samples, the corpus in the first corpus can be taken as input data, the annotated text type can be taken as supervision, and a preset classification model can be trained to obtain the first text type recognition model. The preset classification model can include but not limited to support vector machine model, random forest model, etc., and the present embodiment does not make a specific limitation in this regard.

[0089] Specifically, the first corpus can be obtained by text type annotation based on a first black word library, and the first black word library can be established before training the first text type recognition model, and the first black word library can include a plurality of black words. When annotating the corpus in the first corpus, the first black word library can be obtained first, and each black word in the first black word library can be matched with each corpus in the first corpus to determine whether the corpus includes the black word in the first black word library. When a corpus includes one or more black words in the first black word library, the type of the black word in the first black word library can be used to annotate the text type of the corresponding corpus to obtain the annotated corpus. When a corpus does not include the black word in the first black word library, the corpus can be marked as non-advertising text.

[0090] For example, the type of the black word in the first black word library can include but not limited to social advertising text, promotion advertising text, resource advertising text and transaction advertising text. When a corpus in the first corpus includes a black word in the first black word library, the type of the black word can be determined, and the text type of the corpus can be annotated as the same type as the black word. For example, the text "jia wei" is in the first black word library, and its type is social advertising text, so the text type of the corpus "jia wei xxxxxx" matched with (i.e. including) "jia wei" can be annotated as social advertising text.

[0091] It should be noted that if a corpus is hit by more than two types of black words, i.e., the corpus includes multiple types of black words, the corpus can be text type labeled in the order of social advertising text>resource advertising text>promotion advertising text>transaction advertising text. For example, if the corpus "add micro xxxxx, get hundreds of stone" is hit by the social advertising black word "add micro" and the resource advertising black word "hundreds of stone", since the social advertising text has a higher priority, the text type of the corpus can be labeled as a social advertising text.

[0092] It should be noted that the above division of the text type corresponding to the corpus in the corpus library into non-advertising text, social advertising text, promotion advertising text, resource advertising text and transaction advertising text is only an example. In some possible embodiments, the text type corresponding to the corpus can also be divided into other types, and the advertising type can also not be divided, and only the text type corresponding to the corpus can be divided into non-advertising text and advertising text. The embodiments of the application do not make specific limitations in this regard.

[0093] In actual applications, the first black word library can not only include general domain black words, but also can include specific domain exclusive labels and black words when the method provided by the embodiments of the application is applied to a specific domain text type recognition model. For example, specific advertisements in the game domain, including but not limited to game slang (such as resource merchants selling fruits, and out of marble).

[0094] It can be understood that by establishing a specific domain exclusive label and black word library, the recognition accuracy of the text type recognition model for specific domain text can be improved. In the game domain, not only the coverage of different types of advertisements for different games by the text type recognition model can be improved, but also the recognition accuracy of specific game advertisements can be improved.

[0095] In one possible embodiment, the first corpus can also be obtained after text type labeling based on expert marking. When labeling the corpus in the first corpus, a plurality of expert marked corpora can be obtained first. The expert marked corpus can include a corresponding expert mark, and the expert mark can be a text type of the expert mark. The corpus in the first corpus that is the same as the expert marked corpus can be labeled as the text type in the corresponding expert mark.

[0096] Exemplarily, for the expert-labeled corpus, five options can be provided for the expert to select during expert labeling: 1. non-advertising text, 2. social advertising text, 3. promotion advertising text, 4. resource advertising text, and 5. transaction advertising text. The expert selects which option to label the corpus with the corresponding expert label, for example, when labeling a corpus, the expert selects the option "social advertising text", and the corpus can be labeled with the expert label of social advertising text to obtain an expert-labeled corpus. If there is a corpus in the first corpus library that is the same as the expert-labeled corpus, the text type of the corpus in the first corpus library can be labeled as social advertising text. Alternatively, if there is no corpus in the first corpus library that is the same as the expert-labeled corpus, the expert-labeled corpus can be inserted into the first corpus library, and the corresponding text type can be labeled as social advertising text.

[0097] It can be understood that by adding expert labels, the training data of the text type recognition model can be enriched, thereby improving the recognition effect of the trained model.

[0098] In one possible embodiment, the first corpus library can also be obtained after text type labeling based on suspected black text corpora with larger abnormality coefficients (as an example of suspected preset type text corpora), wherein the suspected black text corpus is a corpus that has a high probability of being advertising text. When labeling the corpora in the first corpus library, a suspected black text model (as an example of a suspected preset type text recognition model) can be used to determine the suspected black text corpora in the first corpus library. The corpora in the first corpus library that are the same as the suspected black text corpora can be labeled as advertising text. Since social advertising accounts for the highest proportion, about 95%, among all advertising texts identified in actual applications, in the case of distinguishing between advertising types, the corpora in the first corpus library that are the same as the suspected black text corpora can be labeled as social advertising text by default.

[0099] It should be noted that the specific content of determining the suspected black text corpus by using the suspected black text model will be described in detail later. After determining the suspected black text corpus, manual auditing can be performed to determine whether it is misidentified, and the suspected black text corpus that is misidentified can be relabeled. Specifically, the suspected black text corpus determined by the suspected black text model can be manually checked to determine whether it is not social advertising text, and if it is not social advertising text, the suspected black text corpus can be relabeled as non-advertising text.

[0100] It can be understood that by using the similarity between the contents and the differences between the black production players and the normal players in terms of behavior characteristics, and adding suspected black text labels, the training data of the text type recognition model can be further enriched, and the recognition effect of the trained model can be improved.

[0101] It should be noted that, in practical applications, the first annotated corpus can be obtained by any one or a combination of the following methods: annotation based on the first black word corpus, annotation based on suspected black text corpus, and annotation based on expert-marked corpus. This embodiment of the invention does not impose any specific restrictions on this.

[0102] In step S210, the evaluation metrics data may include, but are not limited to, precision and recall from public chat data. Public chat refers to the game's public chat channel, whose content is visible to multiple players; public chat data is the content displayed in this public chat channel. Precision and recall from public chat data are calculated based on the most realistic public chat content and can be used to assess the model's judgment ability on the current game's public chat.

[0103] Specifically, a certain number of player messages can be randomly selected from all public chat messages within the first historical time period. A first text type recognition model is used to identify each player's message, determining which messages are identified as advertising text. Furthermore, each black word in the first black word database is matched against each player's message to identify messages containing black words as genuine advertising text. Then, the precision and recall rates are calculated by counting the number of player messages identified as advertising text by the first text type recognition model, the number of player messages identified as advertising text that are also genuine advertising text, and the total number of genuine advertising text messages. The specific calculation formulas are as follows:

[0104]

[0105] Among them, K 11 K represents the accuracy of the public screen data. 12 n1" represents the recall rate on the public screen data, n1′ represents the number of player messages that the first text type recognition model judges as advertising text and that are real advertising text, n1′ represents the number of player messages that the first text type recognition model judges as advertising text, and n1 represents the number of real advertising texts.

[0106] It should be noted that the first historical time period and number of player messages extracted can be preset according to actual needs. For example, it can be set to randomly extract 1,000 player messages from all public chat messages in the last 5 days for precision and recall calculations, etc. This embodiment of the invention does not impose specific limitations on this.

[0107] In the indicator data acquisition step S210, after obtaining the evaluation indicator data of the first text type recognition model, it can be determined whether the evaluation indicator data meets the standard. If it does, it can be determined that the first text type recognition model meets the standard; if not, it can be determined that the first text type recognition model does not meet the standard.

[0108] Specifically, it can be determined whether the precision on the public screen data is greater than or equal to a first precision threshold, whether the recall on the public screen data is greater than or equal to a first recall threshold, and if the precision on the public screen data is greater than or equal to the first precision threshold and the recall on the public screen data is greater than or equal to the first recall threshold, it can be determined that the first text type recognition model meets the standard, otherwise it can be determined that the first text type recognition model does not meet the standard. The first precision threshold and the first recall threshold can be pre-set according to actual needs, for example, both can be set to a value between 0.98-0.99, and the embodiments of the present application do not make specific limitations.

[0109] In actual application, a normalization model effect evaluation index calculation program can be set to periodically calculate and report the evaluation index data of the first text type recognition model to the processor, so that the processor can determine whether the model meets the standard according to the reported evaluation index data, and update the model according to the determination result.

[0110] In actual application, the calculation program can also periodically calculate and report the evaluation index data of the first text type recognition model and the comparison result of the corresponding threshold to the processor, so that the processor can determine whether the model meets the standard according to the reported comparison result, and update the model according to the determination result.

[0111] The period of the calculation program reporting the evaluation index data and / or the comparison result can be pre-set according to actual needs, for example, it can be set to 1 hour, 2 hours, etc., and the embodiments of the present application do not make specific limitations.

[0112] S220: cleaning step, cleaning the first black word library in the case that the first text type recognition model does not meet the standard.

[0113] In the cleaning step S220, in the case that it is determined that the first text type recognition model does not meet the standard, the first black word library (i.e. the instance of the first text library) can be cleaned, that is, whether the number of associated players of each black word in the first black word library meets the expectation is checked, and the black word with more associated players is removed from the first black word library.

[0114] It can be understood that the purpose of cleaning the black word library is to avoid writing incorrect black words. For example, if "hello" is mistakenly written into the first black word library, all the corpus containing "hello" will be labeled as advertising text, which may cause the text type recognition model trained to judge all the corpus containing "hello" as advertising text.

[0115] In a possible embodiment, the cleaning step S220 can include: obtaining historical sending data in a first preset time period, the historical sending data including a plurality of historical sending corpora and role information of sending the historical sending corpora; matching each black word in the first black word library with each historical sending corpus respectively, and counting a first total number of roles corresponding to each historical sending corpus that is matched successfully; removing a black word in the first black word library corresponding to a first total number of roles greater than a first preset threshold, to obtain a cleaned first black word library.

[0116] Specifically, the historical sending data can be real data actually generated in an online environment. Taking a game scenario as an example, the historical sending data can be player speech data in a game in a first preset time period, the player speech data can include a plurality of speech texts of game players and role information of the game players sending the speech texts. The first preset time period can be set in advance according to actual needs, for example, can be set to 30 days, 40 days, and the like, and the present embodiment does not make a specific limitation in this regard.

[0117] Specifically, for each black word in the first black word library, the speech text containing the black word in the speech text of the game player in the first preset time period can be found, and the corresponding role information can be determined, and then the total number of roles of the determined role information can be counted, denoted as a first total number of roles. If the first total number of roles corresponding to the black word is greater than a first preset threshold, the black word is removed from the first black word library; if the first total number of roles corresponding to the black word is less than or equal to the first preset threshold, the black word is retained.

[0118] It can be understood that after the above operation is performed on each black word in the first black word library, a cleaned first black word library can be obtained. The first preset threshold involved in the present embodiment can be set in advance according to actual needs, for example, can be set to 100, and the present embodiment does not make a specific limitation in this regard.

[0119] The method of cleaning the first black word library is illustrated below. First, player speech data in an online environment in 30 days can be extracted, and the total number of roles sending the speech text containing the black word to be cleaned in the first black word library (denoted as A) is calculated. If A is greater than a first preset threshold (for example, can be set to 100), the black word can be marked as a pending black word, and the black word is removed from the first black word library, to avoid intercepting the black word involving too many players. Alternatively, the black word can also be immediately alarmed to notify the relevant person in charge. The relevant person in charge can audit the pending black word to determine whether it is a black word written in error, if not, it can be manually restored as a formal black word and reinserted into the first black word library.

[0120] In actual application, the effective time of the black word corresponding to the first role whose total number is greater than the first preset threshold in the first black word library can be changed to the database default start date, and the black word with an effective period greater than the current day (i.e., the effective black word) in the first black word library can be obtained to obtain the cleaned first black word library.

[0121] It can be understood that by cleaning the black word library, the black word written in error can be removed from the black word library, and the accuracy of the black word library is improved.

[0122] S230: updating step, updating the first corpus based on the cleaned first black word library to obtain a second corpus.

[0123] In the updating step S230, after cleaning the first black word library, the first corpus can be updated according to the cleaned first black word library, that is, whether the labeling of each corpus in the first corpus is correct is checked, and the labeling information of the corpus with incorrect labeling is updated.

[0124] It can be understood that the purpose of updating the corpus is to avoid the corpus with incorrect labeling. For example, if "hello" is mistakenly written into the first black word library, all the corpora containing "hello" in the first corpus will be incorrectly labeled as advertising text, and when the black word library is cleaned, "hello" is removed from the first black word library, so the labeling information of the corpus containing "hello" in the first corpus also needs to be updated to non-advertising text.

[0125] In one possible embodiment, the updating step S230 can include updating the labeling information of each corpus in the first corpus based on the cleaned first black word library to obtain a second corpus.

[0126] Specifically, for each corpus in the first corpus, it can be determined whether the corpus contains only each black word removed from the first black word library when the black word library is cleaned, but does not contain any black word in the cleaned first black word library, if so, it can be determined that the corpus is non-advertising, and the labeling information of the corpus can be updated to non-advertising text; if not, the labeling information of the corpus is not modified.

[0127] It can be understood that by updating the corpus, the corpus with incorrect labeling can be corrected, the accuracy of the labeling information is improved, and the precision and recall of the text type recognition model trained based on the corpus on text type recognition are improved.

[0128] In actual application, in combination with the reference description attached Figure 3In a case where it is determined that the first text type recognition model is substandard, it can be determined whether the first black word library and the first corpus are cleaned and updated. If not, a program for cleaning the first black word library and updating the first corpus can be started, i.e., the cleaning step S220, the updating step S230, and subsequent steps are performed. If the cleaning and updating are completed, an alarm can be triggered again to request manual review.

[0129] S240: retraining step, retraining the model based on the second corpus to obtain a target text type recognition model.

[0130] In the retraining step S240, the second corpus obtained by updating the first corpus can be used to retrain the text type recognition model based on the initially constructed model structure, and the trained model can be evaluated, and the model that meets the evaluation standard can be used as the target text type recognition model.

[0131] In one possible embodiment, the retraining step S240 can include: retraining the model based on the second corpus to obtain a second text type recognition model; and model testing the second text type recognition model, and in a case where the testing is passed, the second text type recognition model is used as the target text type recognition model.

[0132] Specifically, the second text type recognition model can be obtained by retraining the text type recognition model based on the initially constructed model structure using the second corpus. The specific method of training the second text type recognition model based on the second corpus can refer to the specific method of training the first text type recognition model based on the first corpus in the index data obtaining step S210, which will not be repeated here.

[0133] Specifically, in the model testing of the second text type recognition model, the test index data corresponding to the second text type recognition model can be calculated, which can include one or more of the precision and recall on the public screen data, the precision and recall on the global corpus, the precision and recall on the new corpus, and the precision and recall on the training corpus.

[0134] The specific calculation method of the precision and recall on the public screen data can refer to the related content of calculating the precision and recall on the public screen data of the first text type recognition model in the index data obtaining step S210, which will not be repeated here.

[0135] The precision and recall on the global corpus are calculated based on the global corpus, and can be used to investigate the generalization ability of the model. Specifically, a certain number of corpora can be randomly extracted from the global corpus for precision and recall calculation to obtain the precision and recall on the global corpus. The global corpus can include corpora used for model training (i.e., corpora in the first corpus) and corpora not used for model training. Precision and recall calculation can be performed on a portion of corpora used for model training and not used for model training. The specific calculation formula is as follows:

[0136]

[0137] wherein K 21 represents the precision on the global corpus, K 22 represents the recall on the global corpus, n2" represents the number of corpora judged as advertisement text by the second text type recognition model and being real advertisement text, n2' represents the number of corpora judged as advertisement text by the second text type recognition model, and n2 represents the number of corpora of real advertisement text. Specifically, the corpora labeled as advertisement text in the extracted corpora are real advertisement text, and the corpora labeled as non-advertisement text are real non-advertisement text. The labeling method of the corpora can refer to the related content of the labeling of the first corpus in the index data acquisition step S210. The embodiments of the present application will not be repeated here.

[0138] It should be noted that the number of corpora extracted from the global corpus can be pre-set according to actual needs, for example, 1000 corpora can be randomly extracted for precision and recall calculation, etc. The embodiments of the present application do not make specific limitations.

[0139] wherein the precision and recall on the new corpus are calculated based on the newly inserted corpora in a certain time period, and can be used to investigate whether the new model is superior to the old model on the new corpus after completing the training. Specifically, a certain number of corpora can be randomly extracted from the new corpus in a certain time period for precision and recall calculation to obtain the precision and recall on the new corpus. The specific calculation formula is as follows:

[0140]

[0141] wherein K 31 represents the precision on the new corpus, K 32indicates the recall rate on the new corpus, n3"indicates the number of corpus judged as advertisement text and being real advertisement text by the second text type recognition model, n3'indicates the number of corpus judged as advertisement text by the second text type recognition model, and n3indicates the number of real advertisement text. Specifically, the corpus marked as advertisement text is real advertisement text, and the corpus marked as non-advertisement text is real non-advertisement text. The marking method of the corpus can refer to the related content of marking the first corpus in the index data obtaining step S210, and the embodiment of the present application will not be repeated here.

[0142] It should be noted that the time period and the number of the extracted corpus can be pre-set according to actual needs, for example, 500 corpus in the new corpus within 12 hours or on the same day can be randomly extracted for precision and recall rate calculation, and the like, and the embodiment of the present application does not make specific limitation.

[0143] The precision and recall rate on the training corpus are calculated based on the corpus used in the model training, and can be used to examine whether the model training meets the standard, and is the most basic test. Specifically, a certain number of corpus can be randomly extracted from the corpus used in the model training (i.e. the corpus in the first corpus), and the precision and recall rate can be calculated to obtain the precision and recall rate on the training corpus, and the specific calculation formula is as follows:

[0144]

[0145] K 41 indicates the precision on the training corpus, K 42 indicates the recall rate on the training corpus, n4"indicates the number of corpus judged as advertisement text and being real advertisement text by the second text type recognition model, n4'indicates the number of corpus judged as advertisement text by the second text type recognition model, and n4indicates the number of real advertisement text. Specifically, the corpus marked as advertisement text is real advertisement text, and the corpus marked as non-advertisement text is real non-advertisement text.

[0146] It should be noted that the number of corpus extracted from the first corpus can be pre-set according to actual needs, for example, 1000 corpus can be randomly extracted for precision and recall rate calculation, and the like, and the embodiment of the present application does not make specific limitation.

[0147] Specifically, after the test index data corresponding to the second text type recognition model is calculated, it can be judged whether the test index data meets the corresponding test condition, if the corresponding test condition is met, it can be determined that the test is passed, otherwise, it can be determined that the test is not passed.

[0148] Exemplarily, assuming that the test index data includes 8 index data of precision and recall on the public screen data, precision and recall on the global corpus, precision and recall on the new corpus, and precision and recall on the training corpus, it can be respectively determined whether the precision on the public screen data is greater than or equal to a first precision threshold, whether the recall on the public screen data is greater than or equal to a first recall threshold, whether the precision on the global corpus is greater than or equal to a second precision threshold, whether the recall on the global corpus is greater than or equal to a second recall threshold, whether the precision on the new corpus is greater than or equal to a third precision threshold, whether the recall on the new corpus is greater than or equal to a third recall threshold, whether the precision on the training corpus is greater than or equal to a fourth precision threshold, and whether the recall on the training corpus is greater than or equal to a fourth recall threshold. If the above test conditions are all met, it can be determined that the test is passed, and the second text type recognition model meets the standard; otherwise, an alarm mechanism can be triggered, and manual review can be requested. For example, a second alarm message can be output, which is used to indicate that the second text type recognition model does not meet the standard.

[0149] The first precision threshold, the first recall threshold, the second precision threshold, the second recall threshold, the third precision threshold, the third recall threshold, the fourth precision threshold, and the fourth recall threshold can all be pre-set according to actual needs, for example, can all be set to a value between 0.98 and 0.99, and the embodiments of the present application do not make specific limitations thereto.

[0150] It can be understood that in the case that the model does not meet the standard, by automatically cleaning and updating the corpus and retraining the text type recognition model based on the updated corpus, an automatic iteration link of the model is established, and the accuracy of the model retrained for text type recognition can be improved.

[0151] In one possible embodiment, the model updating method can further include a model deployment step of deploying a preset text type recognition model to an online environment in the case that the first text type recognition model does not meet the standard.

[0152] In combination with the accompanying drawings Figure 3 In the case that it is determined that the first text type recognition model does not meet the standard, the preset text type recognition model can be deployed as a bottom-line model in the online environment to replace the first text type recognition model that does not meet the standard in the online environment, and an alarm mechanism is immediately triggered to notify the relevant person in charge. The preset text type recognition model can be a model that is pre-set and verified to have good effect, which is used to replace the model that does not meet the standard in an emergency to ensure the accuracy of the text type recognition model interface in the online environment.

[0153] Specifically, the preset text type recognition model can be determined according to actual needs, and the structure and acquisition method of the preset text type recognition model are not limited in the embodiments of the present application. In the case that the first text type recognition model does not meet the standard, a first alarm message can be output, which is used to indicate that the first text type recognition model does not meet the standard.

[0154] In one possible embodiment, in the case that the first text type recognition model crashes, the model deployment step can also be performed to deploy the preset text type recognition model to the online environment as a bottom-line model. For example, the evaluation index data of the first text type recognition model can be calculated in real time, including but not limited to the precision and recall on the public screen data. When the precision and / or recall on the public screen data is lower than a certain threshold (indicating that the evaluation index data is severely deteriorated), the preset text type recognition model can be deployed to the online environment. The threshold can be pre-set according to actual needs, and the embodiments of the present application do not make specific limitations thereto.

[0155] In one possible embodiment, in the case that the first text type recognition model update time is overdue, the model deployment step can also be performed to deploy the preset text type recognition model to the online environment as a bottom-line model. For example, since the evaluation index data of the first text type recognition model is reported, the time consumption of the first text type recognition model update process can be recorded, and when the time consumption is greater than a certain time value, the preset text type recognition model can be deployed to the online environment. The time value can be pre-set according to actual needs, and the embodiments of the present application do not make specific limitations thereto.

[0156] It can be understood that in the case that the first text type recognition model does not meet the standard, the first text type recognition model crashes, or the first text type recognition model update time is overdue, the bottom-line model deployed in the online environment can automatically fuse when risks are encountered, ensuring the accuracy of the text type recognition model interface in the online environment and ensuring the normal operation of the online environment.

[0157] In one possible embodiment, in combination with the accompanying drawings Figure 4 The model update method can further include the following steps:

[0158] S250: Corpus acquisition step, in the case that the first text type recognition model meets the standard, a third corpus is acquired, and the third corpus is obtained by updating the first corpus based on the second black word library (as an instance of the second text library) and the newly added corpus set within the second preset time period.

[0159] The second black word library includes the first black word library and a newly added black word set in a second preset time period (as an example of a newly added text set), and the newly added black word set includes texts marked as belonging to advertising texts in the second preset time period.

[0160] In actual application, the first text type recognition model can be updated according to the following steps in combination with the reference description attached Figure 3 When it is determined that the first text type recognition model meets the standard, it can be determined whether the update time is reached. When the update time is not reached, only the model index evaluation report of this time can be submitted. When the update time is reached, the black word library checking and parallel program and the corpus insertion parallel program can be started to perform the corpus acquisition step S250 and subsequent steps.

[0161] In the corpus acquisition step S250, when it is determined that the first text type recognition model meets the standard, the black word library and the corpus can be updated regularly to optimize and train the model in the online environment based on the updated corpus to obtain an optimized model. The period of updating the black word library and the corpus can be set in advance according to actual needs, for example, it can be set to be updated once a day or twice a day, and the update time can also be set in advance according to actual needs, for example, it can be set to 12:00 and 24:00, etc., and the present embodiment does not make specific limitation thereto.

[0162] Specifically, in combination with the reference description attached Figure 5 The corpus acquisition step S250 can include:

[0163] S510: acquiring a second black word library and a newly added corpus set in a second preset time period, the second black word library including the first black word library and a newly added black word set in the second preset time period.

[0164] Optionally, in step S510, acquiring the second black word library can include: acquiring the newly added black word set in the second preset time period; and adding the newly added black word set to the first black word library to obtain the second black word library.

[0165] Specifically, each black word in the newly added black word set can be a text marked as belonging to an advertising text in the second preset time period, and the newly added black word set can be determined by manual operation. The newly added black word set is inserted into the first black word library to obtain the second black word library. The second preset time period can be set in advance according to actual needs, and the present embodiment does not make specific limitation thereto. Preferably, the second preset time period can be set as the time period between the previous update time and the present update time.

[0166] Exemplarily, before reaching the update time, the relevant person in charge can determine the time period between the previous update time and the current update time, and the new text belonging to the advertising text appearing in the online environment or other places during the time period can also be found by digging into the specific game black production jargon, and the specific game exclusive advertising text can be found, and the above two types of text are marked to form the newly added black word set.

[0167] In one possible embodiment, the relevant person in charge can also manually search the database for the corresponding transformed black words of each black word in the newly added black word set, such as variant characters, variant characters, added variant symbols, etc., and add the transformed black words to the newly added black word set to increase the richness of the black word library.

[0168] Exemplarily, there are two paths for the generation of variant characters, one is homophonic characters, i.e. the same pronunciation, such as "micro -> Wei"; the other is structurally similar characters, such as "message -> message" (of course, "micro -> Wei" is also). Therefore, the variant characters can be stored in the database, and when new advertising text has no possibility of generating variant characters, the database can be increased. For example, homophonic characters and structurally similar characters can be searched on the Internet, such as "substitute for value" this text is added to the black word library, and the word "Chong" often appears in the form of variant characters, so the website can search for homophonic characters "worm, Chong" and structurally similar characters "gun, gun, gun, gun" and the like, so as to generate several transformed texts, such as "substitute for worm" and "substitute for gun".

[0169] In one possible embodiment, since some texts after transformation, such as "add micro" transformed into "Ga Wei" may become the name of a certain hero in the game or a normal player's common language, after determining the transformed text, the transformed text can also be cleaned to remove such texts. The specific cleaning process can refer to the specific content of cleaning the first black word library in the cleaning step S220, and the embodiments of the present application will not be described here.

[0170] Alternatively, in step S510, obtaining the second black word library can include: obtaining the newly added black word set in the second preset time period; cleaning the newly added black word set to remove mislabeled texts in the newly added black word set; adding the cleaned newly added black word set to the first black word library to obtain the second black word library.

[0171] Specifically, after manually determining the newly added black word set, the black words in the entire newly added black word set can also be cleaned to remove the mislabeled black words in the entire newly added black word set. Insert the cleaned newly added black word set into the first black word library to obtain the second black word library. The specific cleaning process can also refer to the specific content of cleaning the first black word library in the cleaning step S220, and the embodiments of the present application will not be described here.

[0172] ​It can be understood that by deeply mining specific game black production jargon, finding specific exclusive black words, and by imitating black and gray production through operations such as variant characters, variants, adding variant symbols, etc. on the basis of existing black words, the richness of the black word library can be increased, and the black production countermeasures can be captured in advance, thereby improving the countermeasures of the model optimized and trained against black production, and improving the coverage of the model for new forms of advertisements.

[0173] In step S510, the newly added corpus set can include part or all of the corpus actually generated in the online environment. For example, the corpus not repeated in the first corpus library among the corpus actually generated in the online environment in the second preset time period can be taken as the newly added corpus. The newly added corpus set can also include the possible appearing advertisement texts predicted in advance by the relevant person in charge. Taking the game scene as an example, the possible appearing advertisement texts can be predicted in advance through the names of resources, benefits, etc. in the game, such as “huangjing” and “huangjin” variant characters that can appear in future advertisements, and “recharge 100 yuan to get 5w huangjing” and other jargon that can become future advertisement texts.

[0174] It can be understood that by deeply mining specific game black production jargon and predicting possible appearing advertisement texts in advance, the richness of the corpus library can be increased, and the black production countermeasures and new forms of advertisements can be captured in advance, thereby further improving the coverage of the model optimized and trained for new forms of advertisements, preventing the occurrence of advertisements, and improving the countermeasures of the model against black production, so that the text type recognition model can timely capture and alarm new forms of advertisements that frequently countermeasure and ultimately intercept.

[0175] S520: text type labeling of each corpus in the newly added corpus set based on the second black word library.

[0176] In step S520, the method of text type labeling of each corpus in the newly added corpus set based on the second black word library can refer to the method of labeling the corpus in the first corpus library based on the first black word library in the index data acquisition step S210, and the present embodiment will not be described here.

[0177] S530: updating the labeling information of each corpus in the first corpus library based on the newly added black word set.

[0178] In step S530, each black word in the newly added black word set can be matched with each corpus in the first corpus library respectively to determine whether the black word in the newly added black word set is included in each corpus. When a corpus includes a black word in the newly added black word set, the labeling information of the corpus can be modified according to the type of the black word.

[0179] Specifically, if the annotation information of a certain corpus is non-advertising text, and the corpus is hit by a certain black word in the newly added black word set, the annotation information of the corpus can be modified to advertising text, or the annotation information of the corpus can be modified to the type corresponding to the hit black word. For example, the annotation information of a certain corpus is non-advertising text, and the corpus is hit by the black word "jia wei" (whose type is social advertising text) in the newly added black word set, and the annotation information of the corpus can be modified to social advertising text.

[0180] Specifically, if the annotation information of a certain corpus is advertising text with a lower priority, the corpus is hit by a certain black word in the newly added black word set, and the black word has a type with a higher priority, the annotation information of the corpus can be modified to the type corresponding to the hit black word. For example, the annotation information of a certain corpus is promotion advertising text, and the corpus is hit by the black word "jia wei" (whose type is social advertising text) in the newly added black word set, and the annotation information of the corpus can be modified to social advertising text.

[0181] It should be noted that if a corpus is hit by more than two types of black words in the newly added black word set, the corpus can be annotated with the text type in the order of priority: social advertising text > resource advertising text > promotion advertising text > transaction advertising text.

[0182] Specifically, if a certain corpus is not hit by a black word in the newly added black word set, or if a certain corpus is hit by a black word in the newly added black word set but its annotation information is already advertising text or the highest priority social advertising text, the annotation information of the corpus is not modified.

[0183] S540: Add the annotated newly added corpus set to the first corpus library with updated annotation information to obtain a third corpus library.

[0184] In step S540, the annotated newly added corpus set and the first corpus library with updated annotation information can be merged to obtain the third corpus library.

[0185] S260: Optimization training step, based on the third corpus library, the first text type recognition model is optimized and trained to obtain a target text type recognition model.

[0186] In the optimization training step S260, the third corpus library can be used to perform model training link with the first text type recognition model as the initial model to obtain the target text type recognition model.

[0187] It should be noted that the specific method of training the target text type recognition model based on the third corpus library can refer to the existing model training method, and the embodiments of the present application will not be repeated here.

[0188] It can be understood that the embodiment of the present application can periodically complete automatic updating of the model, improve the coverage of the model to new forms of advertisements, and the confrontation ability of the model to black production by establishing a game-specific tag and a black word library, and establishing a corresponding black word generation link and an updating link of the black word library and the corpus in the case that the first text type recognition model meets the standard.

[0189] In one possible embodiment, in combination with the description of the accompanying drawings Figure 6 In the corpus acquisition step S250, before the step S530, the step S550 and the step S560 can also be included:

[0190] S550: determining the suspected black text corpus (as an instance of the suspected preset type text corpus) in the new corpus set by using the suspected black text model (as an instance of the suspected preset type text corpus recognition model).

[0191] In step S550, part or all of the corpus in the new corpus set can be input into the suspected black text model to obtain the output suspected black text corpus.

[0192] In one possible embodiment, determining the suspected black text corpus in the new corpus set by using the suspected black text model can include: calculating the text distance between each corpus in the new corpus set two by two; dividing the new corpus set into at least one corpus subset according to the text distance, and the text distance between any two corpora in each corpus subset is less than a preset distance; determining whether each corpus subset is a suspected black text set, and obtaining the abnormality coefficient of each corpus subset determined as a suspected black text set; and regarding each corpus in the corpus subset whose abnormality coefficient is greater than a second preset threshold as a suspected black text corpus.

[0193] Specifically, the text distance between two corpora can be calculated by the following formula:

[0194]

[0195] Dis(A,A') = ∞, |length(A)-length(A')|>m

[0196] Wherein, L(A,A') represents the edit distance between two corpora A and A', length(A) represents the text length of corpus A, and length(A') represents the text length of corpus A'. m represents a preset length threshold, which can be pre-set according to actual needs, for example, it can be set as m=5, and the embodiment of the present application does not make specific limitation.

[0197] Specifically, by calculating the pairwise text distances between each corpus in the newly added corpus, a distance matrix between corpora can be obtained. Then, algorithms such as agglomerative clustering can be used to divide the newly added corpus into at least one subset, denoted as the corpus subset. (N is the number of corpora). Each subset of the corpus... Let A represent a set containing a certain corpus, where A is a corpus within the set. i The set is labeled with i = 1, ..., N. For each subset of the corpus... The following conditions must be met: The text distance between any two corpora within a set is less than a preset distance d. For any two corpora from different sets, the text distance between the corpora is greater than or equal to the preset distance d. In particular, if the text distance between a corpus and any other corpus is greater than or equal to the preset distance d, then that corpus forms its own set.

[0198] The value of the preset distance d can be preset according to actual needs. For example, it can be set to a value between 0.1 and 0.2. This embodiment of the invention does not make a specific limitation on this.

[0199] Specifically, determining whether each subset of corpus is a suspected black text set may include: obtaining sending status data corresponding to newly added corpus sets within a third preset time period, the sending status data including role information of each corpus sent in the newly added corpus set; for each subset of corpus, determining the total number of times each corpus in the subset was sent and the total number of second roles based on the sending status data; calculating the sending non-repetition rate and abnormal behavior data corresponding to the subset of corpus based on the sending status data; and determining that the subset of corpus is a suspected black text set if the total number of times sent, the total number of second roles, the sending non-repetition rate, and the abnormal behavior data meet preset conditions.

[0200] To clearly illustrate the method provided in the embodiments of the present invention, the following concepts can be defined in advance:

[0201] Advertising text collection The set of texts that the first text type recognition model identifies as advertisements.

[0202] Send set G A Equipment collection That is, set G A For any given corpus, the device that sent it will record it in the set. among.

[0203] Send set G A Character Collection That is, set GA Any corpus in the set G When.

[0204] The set G A The set of Internet Protocol (IP) addresses of the set G The set G A Any corpus in the set G When.

[0205] Specifically, the sending situation data can be real data actually generated in the online environment. For example, in a game scenario, the sending situation data can be the player speech data of each corpus in the newly added corpus set in the third preset time period. The sending situation data can include the speech text of multiple game players and the role information of the game players sending the speech text.

[0206] The third preset time period can be pre-set according to actual needs. For example, for a game with a small amount of text, the third preset time period can be set to 24 hours, and for a game with a large amount of text, the third preset time period can be set to 1 hour. The embodiments of the present application do not make specific limitations on this.

[0207] Specifically, for each corpus subset G A , the speech text containing each corpus in the corpus subset G A in the speech text of the game player in the third preset time period can be found, and the corresponding role information can be determined. Then the number of speech texts containing each corpus in the corpus subset G A can be counted, denoted as the total sending times S T (G A ), and the total number of roles sending each corpus in the corpus subset G A can be counted, denoted as the second total number of roles R T (G A ).

[0208] Specifically, the sending non-repetition rate can include one or more of the device dimension sending non-repetition rate, the role dimension sending non-repetition rate, and the IP dimension sending non-repetition rate. The abnormal behavior data can include one or more of the device dimension abnormal behavior data, the role dimension abnormal behavior data, and the IP dimension abnormal behavior data. The abnormal behavior data can be the total number of sending any advertising text in the newly added corpus set.

[0209] Specifically, for each corpus subset G A , the sending situation data can be used to determine the set GA Equipment collection Character Collection and IP address set Alternatively, we can first determine the set of advertising texts in the newly added corpus, identified based on the first text type recognition model.

[0210] Then, the device set can be statistically analyzed based on the transmission data. Send advertising text collection The total number of any ad text in China and the equipment set The various devices D in i i = 1...N D (N D For equipment collection Calculate the corresponding transmission non-repetition rate based on the number of devices. And calculate the average non-repetition rate of transmission. The calculation formula is as follows:

[0211]

[0212]

[0213] The role set can be statistically analyzed based on the sending data. Send advertising text collection The total number of any ad text in China And the character collection The various characters R i i = 1...N R (N R For character collection Calculate the corresponding non-repetition rate for sending messages based on the number of characters in the message. And calculate the average non-repetition rate of transmission. The calculation formula is as follows:

[0214]

[0215] IP address set can be statistically analyzed based on sending data. Send advertising text collection The total number of any ad text in China and the set of IP addresses Each IP address in i i = 1...N IP (N IP For IP address set Calculate the corresponding non-repetition rate for sending IP addresses. And calculate the average non-repetition rate of transmission. The calculation formula is as follows:

[0216]

[0217] Specifically, for each subset of the corpus, it can be determined whether the corresponding total number of transmissions, the total number of second characters, the non-repetition rate of transmissions, and the abnormal behavior data meet the preset conditions. If the total number of transmissions, the total number of second characters, the non-repetition rate of transmissions, and the abnormal behavior data all meet the preset conditions, the subset of the corpus is determined to be a suspected black text set.

[0218] The preset conditions can be pre-set according to actual needs. For example, they can be set to: the total number of sends being greater than a preset number; the total number of second roles being less than a first preset number; the non-repetition rate of sends being less than a preset repetition rate; and the total number of any advertising texts sent from the newly added corpus being greater than a second preset number. The preset number of sends, the first preset number, the preset repetition rate, and the second preset number can all be pre-set according to actual needs, and this embodiment of the invention does not impose specific limitations on them.

[0219] For example, the total number of transmissions S can be determined. T (G A Is it greater than the threshold T? s Determine the total number R of the second character. T (G A Is it less than the threshold T? R Determine the set of advertising texts to send. The total number of any ad text in China Is it greater than the threshold T? S,D Determine the average non-repetition rate of the sent data. Is it less than the threshold T? p,D Determine the set of advertising texts to send. The total number of any ad text in China Is it greater than the threshold T? S,R Determine the average non-repetition rate of the sent data. Is it less than the threshold T? p,R Determine the set of advertising texts to send. The total number of any ad text in China Is it greater than the threshold T? S,IP Determine the average non-repetition rate of the sent data. Is it less than the threshold T? p,IP The corpus subset G is determined if the following conditions are met. A A collection of suspected black text:

[0220] (1)S T (G A )>T S This indicates that the corpus subset G Aeach corpus in the corpus subset G

[0221] (2)R T (G A )<T R , indicating that if each corpus in the corpus subset G A is banned, the number of affected roles is less than a certain threshold.

[0222] (3) any one of the following three conditions is met: indicating that the sending non-repetition degree of the device, role, and IP dimensions is low. Generally, the lower the non-repetition rate of sending text under a device, role, and IP, the higher the abnormality degree of the device, role, and IP.

[0223] (4) any one of the following three conditions is met: indicating that the device, role, and IP that sent the corpus subset G A have sent advertising text.

[0224] wherein, the values of T S , T R , T p,D , T p,R , T p,IP , T S,D , T S,R , and T S,IP may be pre-set according to actual needs, and the embodiments of the present application do not make specific limitations thereon.

[0225] Specifically, after determining each corpus subset of the suspected black text set, the total sending times S T (G A ) corresponding to each corpus subset can be obtained as a corresponding abnormality coefficient for representing the abnormality degree of the corresponding corpus subset. When the total sending times S T (G A ) is greater than a second preset threshold, each corpus in the corresponding corpus subset can be regarded as a suspected black text corpus and can be added to the corpus library. The second preset threshold can be pre-set according to actual needs, for example, can be set to 100, 150, etc., and the embodiments of the present application do not make specific limitations thereon.

[0226] The working principle of the suspected black text model provided by the embodiments of the present application is illustrated below. Taking 1 hour as the third preset time period, the sending condition data table processed in the online environment in the last 1 hour is as follows:

[0227]

[0228]

[0229] It can be seen that, Assume the preset distance d = 0.2, corpus A' satisfies Dis(A, A') = 2 / 19 < d = 0.2. Wherein, 19 is the minimum value of the length of "Is brother, add my WeChat xxxxxx" and "Is brother, add my WeChat xxxxxx". The numerator 2 indicates that only the characters "micro" to "We" and "signal" to "core" need to be edited from "Is brother, add my WeChat xxxxxx" to "Is brother, add my WeChat xxxxxx", so the text distance between the two corpora is 2 / 19, which is less than the preset distance d, so the corpus A' and the corpus A can be divided into the same corpus subset G A′ And the text distance between corpus B and corpus A and corpus A' is greater than the preset distance d, so corpus B can be divided into a corpus subset alone.

[0230] For the corpus subset G A′ As analyzed above, the set contains corpus A = 'Is brother, add my WeChat xxxxxx' and corpus A' = 'Is brother, add my WeChat xxxxxx', and the following assumes that the set contains only corpus A and corpus A'.

[0231] Calculate the number of times S A′ of all corpora in the corpus subset G T (G A′ ) in the third preset time period. It is assumed that A appears 1100 times and A' appears 100 times, so S T (G A′ ) = 1200 times. Calculate the total number of roles R A′ of all corpora in the corpus subset G T (G A′ ). It is assumed that only roles R1 and R2 have sent corpora in the corpus subset G A′ , so R T (G A′ ) = 2.

[0232] Calculate the device set that sends any advertising text (i.e. corpus A) As shown in the above table, devices D1 and D2 will be calculated into the device set . Therefore, the advertising text sent by devices D1 and D2 will be calculated into the total number . Assuming that the device set contains only D1 and D2, and D1 has sent 1000 corpora A (which are advertising texts), and D2 has sent 100 corpora A and 100 corpora A' (which are non-advertising texts), then

[0233] All devices D in the device set i i = 1...N D Calculate the sending non-repetition rate And calculate the average value. Assuming that D1 sends 1000 corpus A, the sending non-repetition rate is 1 / 1000 = 0.001, D2 sends 100 corpus A and 100 corpus A', the sending non-repetition rate is 2 / 200 = 0.01, then the average value

[0234] The total number of role dimension sending any advertising text can be calculated in a similar way The sending non-repetition rate of the role dimension The total number of IP dimension sending any advertising text And the sending non-repetition rate of the IP dimension

[0235] If the corpus subset G A′ Corresponding S T (G A′ ), R T (G A′ ), And Satisfy the above four conditions (1)-(4), it can be determined that the corpus subset G A′ is a suspected black text set. And, since the corpus subset G A′ Corresponding S T (G A′ ) = 1200 > 100, it can be determined that all corpus in the corpus subset G A′ is a suspected black text corpus.

[0236] S560: Based on the suspected black text corpus, label the text type of each corpus in the new corpus set.

[0237] In step S560, the same corpus in the new corpus set as the suspected black text corpus can be labeled as an advertising text, or in the case of distinguishing the advertising type, the same corpus in the new corpus set as the suspected black text corpus can be labeled as a social advertising text. The specific labeling method can refer to the specific content of the method provided by the embodiment shown in Figure 2 The embodiments of the present application will not be repeated here.

[0238] ​It can be understood that by utilizing the similarity between contents and the difference between the black production and the normal player behavior characteristics, the suspected black text corpus can be determined, the accuracy of the suspected black text corpus can be improved, and the training data of the text type recognition model can be enriched, and the recognition effect of the model obtained by training can be improved.

[0239] In one possible embodiment, the reference is made to the description attached hereto Figure 6 In the corpus obtaining step S250, before the step S530, the step S570 and the step S580 can also be included.

[0240] S570: Obtain the expert-labeled corpus in the newly added corpus set.

[0241] In the step S570, the corpus labeled by the experts in a fourth preset time period can be obtained as the expert-labeled corpus. The fourth preset time period can be pre-set according to actual requirements, and the present embodiment is not limited in this regard. Preferably, the fourth preset time period can be set to be the same as the second preset time period.

[0242] S580: Perform text type labeling on each corpus in the newly added corpus set based on the expert-labeled corpus.

[0243] In the step S580, the specific labeling method can refer to the specific content of the method provided in the embodiment shown in Figure 2 The present embodiment is not described herein again.

[0244] It can be understood that by adding the expert labeling, the training data of the text type recognition model can be further enriched, and thus the recognition effect of the model obtained by training can be improved.

[0245] In one possible embodiment, after the suspected black text corpus and the expert-labeled corpus are determined, the suspected black text corpus and the expert-labeled corpus can also be cleaned to avoid the labeling error of the corpus. The specific cleaning process can refer to the specific content of cleaning the first black word library in the cleaning step S220, and the present embodiment is not described herein again.

[0246] It should be noted that although the present embodiment describes three ways of text type labeling based on the black word library, the suspected black text corpus and the expert-labeled corpus, in actual application, only one or more ways can be used, and the present embodiment is not limited in this regard. For example, since the expert-labeled corpus is small, there can be only 1-2 pieces per week, which is a small order of magnitude compared with the hundreds of pieces of the black word library and the suspected black text corpus per day, and thus in actual application, the way based on the expert-labeled corpus can be ignored. When the corpus library is updated every day, only the ways based on the black word library and the suspected black text corpus can be used, and when the corpus library is updated for the last time every week, the way based on the expert-labeled corpus can be considered.

[0247] In actual application, when considering the three ways of text type labeling based on the black word library, the suspected black text corpus and the expert labeled corpus simultaneously, the suspected black text corpus can be used for labeling first, then the expert labeled corpus can be used for labeling, and finally the black word library can be used for labeling. Specifically, the corpus hit by the suspected black text corpus can be labeled as social advertising text, the corpus hit by the expert labeled corpus can be labeled as the advertising type corresponding to the expert label, and the corpus hit by the text in the black word library can be labeled as the type corresponding to the text in the black word library. If a corpus is hit multiple times, the corpus can be labeled as the text type with the highest priority. Only when a corpus is not hit by any text in the black word library and does not belong to the expert labeled corpus and the suspected black text corpus, the corpus can be labeled as non-advertising text.

[0248] It should be noted that other contents of text type labeling of each corpus in the newly added corpus set in the embodiments of the present application can refer to the specific contents of the method provided by the embodiments shown in the specification, and the embodiments of the present application will not be repeated here. Figure 2

[0249] In one possible embodiment, the corpus set can be obtained by combining the corpus set shown in the specification with the corpus set shown in the specification. Figure 3 After the corpus set obtaining step S250, the obtained third corpus set can also be checked to obtain a checked third corpus set. Correspondingly, the checked third corpus set can be used to take the first text type recognition model as an initial model to execute a model training link to obtain a target text type recognition model.

[0250] Specifically, the checking of the third corpus set is mainly aimed at the corpus labeled based on the suspected black text corpus and the expert labeled corpus, and the checking of the third corpus set is mainly to avoid the occurrence of a corpus that is incorrectly judged as an advertising text. For example, if a corpus frequently appearing in a map game is "I sent a coordinate", if this corpus is incorrectly labeled as an advertising text, it may cause all the coordinates sent by the players to be intercepted, resulting in very serious consequences, so it is necessary to determine whether the corpus is a suspected false advertising text, if so, the corpus can be removed from the third corpus set and an alarm can be given, or the labeling information of the corpus can be modified as non-advertising text. The method of checking the third corpus set is similar to the method of cleaning the first black word library, and the embodiments of the present application will not be repeated here.

[0251] It can be understood that by checking the corpus set, the corpus labeled incorrectly according to the expert labeled corpus and the suspected black text corpus can be corrected, the accuracy of the labeling information of the corpus can be improved, and the precision and recall of the text type recognition model trained based on the corpus set can be further improved.

[0252] ​In one possible embodiment, the model updating method further comprises the following steps: Figure 4

[0253] S270: an online testing step, testing the target text type recognition model online, and deploying the target text type recognition model to an online environment in case of passing the test.

[0254] In the online testing step S270, the trained target text type recognition model can be deployed to a shadow environment (a kind of testing environment), and a shadow environment testing process is triggered. When the target text type recognition model passes the test in the shadow environment, it can be deployed to the online environment to replace the model in the online environment. When the target text type recognition model fails to pass the test in the shadow environment, it is not deployed to the online environment, and the online environment still uses the current model.

[0255] In actual applications, the model updating method can further comprise the following steps: Figure 3 After the target text type recognition model is obtained through model training, a shadow environment testing parallel program can be executed to determine whether the effect of the target text type recognition model (i.e., a new model) is better than that of the original model in the online environment. If so, the target text type recognition model is used to replace the original model and is deployed to the online environment. Otherwise, the original model is maintained. After deployment, an effect evaluation report of the target text type recognition model can be submitted.

[0256] Specifically, the model shadow environment testing is to deploy the target text type recognition model in an environment named shadow, and maintain the deployment of the original model in the formal environment, so as to compare and evaluate the precision and recall rates of the target text type recognition model and the original model. The specific implementation scheme is as follows:

[0257] 1. After the target text type recognition model is trained, it is deployed in an environment named shadow. The environment can receive 100% of the traffic copy of the online environment, and record the logs into a database, but the model prediction result is not directly output to the online environment.

[0258] 2. The original model is still deployed in the online environment, and the logs are recorded into the database, and the output result is directly fed back to the online environment.

[0259] ​3. input the test corpus into the two models respectively, compare the output results of the two models on the same test corpus recorded in the log, evaluate the difference between the evaluation index data of the target text type recognition model and the original model, if the evaluation index data of the target text type recognition model is better than that of the original model, it can be determined that the target text type recognition model passes the test in the shadow environment, and can replace the original model to be deployed in the online environment, otherwise it can be determined that the target text type recognition model fails the test in the shadow environment, and the original model deployed in the online environment remains unchanged. The test corpus can be the corpus actually generated in the online environment.

[0260] Specifically, the evaluation index data can include one or more of the precision and recall on the public screen data, the precision and recall on the global corpus, the precision and recall on the new corpus, and the precision and recall on the training corpus. For example, assuming that the evaluation index data includes the precision and recall on the public screen data, the precision and recall on the global corpus, and the precision and recall on the new corpus, when the three precisions and recalls of the target text type recognition model are higher than those of the original model, it can be determined that the target text type recognition model passes the test in the shadow environment; otherwise, it can be determined that the target text type recognition model fails the test in the shadow environment.

[0261] It can be understood that through the model shadow environment test, the new model can be safely, conveniently and automatically tested without affecting the production environment, and the effect of the new model in the real production environment can be simulated. Compared with manual online testing, the safety and convenience are improved.

[0262] Deploying the model update link provided by the embodiment of the application in the game field and comparing the original model with the new model obtained by using the model update method provided by the embodiment of the application can know that: the evaluation index of the new model is greatly improved, the recognition accuracy and recall rate of the new model for the advertisement text are increased from 95% of the original to more than 99%, and the recognition rate of different games and different types of advertisements can reach more than 99%; the business index exceeds the expectation, the proportion of advertisement messages in places such as in-game chat, alliance announcement, email, nickname signature and the like in all messages is reduced to 0%, the discovery and processing time of new forms of advertisements is shortened to 20 minutes, the interception proportion is increased from less than 1% to more than 80%, and the cold start time of new games is shortened to 1 day; the labor cost is greatly reduced, and the manual participation time is shortened from 1 hour per day to 10 minutes per day.

[0263] To sum up, the embodiment of the present application provides a new architecture including data labels, text corpus, algorithm model, service interface and the like, and a model optimization method based on the architecture, in the case of model reaching the standard and not reaching the standard, different methods are used to update the text type recognition model, a complete model automatic iteration link is established, automatic training and testing of the model are realized, the precision and recall of the text type recognition model in text type recognition are improved. And when encountering risks, it can be automatically fused, and new forms of frequently confronted advertisements can be captured, alarmed, automatically updated algorithm and finally intercepted in time.

[0264] In addition, through expert marking and suspected black text corpus, the automatic cleaning and updating of the corpus library are improved, and subsequent model training and effect evaluation are connected, and the monitoring and alarm link of the whole link is added, so that the safety and transparency of the automatic operation are ensured, and the efficiency of the model automatic test iteration is improved.

[0265] An embodiment of the present application also provides a computer readable storage medium which can be arranged in an electronic device to save a computer program / instruction related to a model updating method, and the computer program / instruction is loaded and executed by a processor to implement the model updating method provided by the above method embodiment.

[0266] Optionally, in the embodiment of the present application, the storage medium can include but is not limited to: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk and various storage program codes.

[0267] An embodiment of the present application also provides a computer program product, which includes a computer program / instruction, when the computer program product runs on an electronic device, the computer program / instruction is loaded and executed by a processor to implement the steps of the model updating method provided in the above various optional embodiments.

[0268] It should be noted that: the above-mentioned sequence of the embodiments of the present application is only for description, not representing the advantages and disadvantages of the embodiments. And the above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be executed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous.

[0269] The various embodiments described in the specification are progressive in nature, and identical or similar parts among the various embodiments can be mutually referred to. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are described relatively simply because they are substantially similar to the method embodiments, and the relevant parts can be referred to the description of the method embodiments.

[0270] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed to relevant hardware by a program. The program can be stored in a computer readable storage medium, and the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0271] The above only describes the preferred embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A model updating method for an electronic device, comprising: The method comprises the following steps: an index data acquisition step, acquiring evaluation index data of a first text type recognition model in an online environment, and determining whether the first text type recognition model meets the standard according to the evaluation index data; wherein the first text type recognition model is obtained by training based on a first corpus, and the first corpus is obtained by text type annotation based on at least a first text library, and the first text library comprises texts marked as belonging to a preset type; a cleaning step, cleaning the first text library if the first text type recognition model does not meet the standard; an updating step, updating the first corpus based on the cleaned first text library to obtain a second corpus; a retraining step, retraining the model based on the second corpus to obtain a target text type recognition model; a corpus acquisition step, comprising: acquiring an added corpus set in a second preset time period; determining suspected preset type text corpus in the added corpus set by using a suspected preset type text recognition model, comprising: calculating the text distance between each corpus in the added corpus set; dividing the added corpus set into at least one corpus subset according to the text distance, and the text distance between any two corpora in each corpus subset is less than a preset distance; determining whether each corpus subset is a suspected preset type text set, and obtaining the abnormality coefficient of each corpus subset determined as a suspected preset type text set; wherein the sending situation data corresponding to the added corpus set in a third preset time period is obtained, the sending situation data comprises role information of sending each corpus in the added corpus set, for each corpus subset, the total sending times of each corpus in the corpus subset and the second total number of roles are determined according to the sending situation data, the sending non-repetition rate and abnormal behavior data corresponding to the corpus subset are calculated according to the sending situation data, and the corpus subset is determined as a suspected preset type text set if the total sending times, the second total number of roles, the sending non-repetition rate and the abnormal behavior data meet the preset conditions; each corpus in the corpus subset with an abnormality coefficient greater than a second preset threshold is regarded as a suspected preset type text corpus; text type annotation is performed on each corpus in the added corpus set based on the suspected preset type text corpus.

2. The method of claim 1, wherein, The cleaning step comprises: acquiring historical sending data in a first preset time period, the historical sending data comprising a plurality of historical sending corpora and role information of sending the historical sending corpora; matching each text in the first text library with each historical sending corpus, and counting the first total number of roles corresponding to each historical sending corpus matched successfully; removing the texts in the first text library corresponding to the first total number of roles greater than a first preset threshold to obtain the cleaned first text library.

3. The method of claim 2, wherein, The updating step comprises: updating the annotation information of each corpus in the first corpus based on the cleaned first text library to obtain a second corpus.

4. The method of claim 1, wherein, The retraining step comprises: retraining a model based on the second corpus to obtain a second text type recognition model; performing model testing on the second text type recognition model, and deploying the second text type recognition model to an online environment in a case where the testing is passed.

5. The method of claim 1, wherein, The method further comprises: a model deployment step of deploying a preset text type recognition model to an online environment in a case where the first text type recognition model is not up to standard.

6. The method of claim 1, wherein, The corpus obtaining step further comprises: In a case where the first text type recognition model is up to standard, a third corpus is obtained, the third corpus being obtained by updating the first corpus based on a second text corpus and a newly added corpus set in a second preset time period, the second text corpus comprising the first text corpus and a newly added text set in the second preset time period, the newly added text set comprising texts labeled as belonging to a preset type in the second preset time period; The method further comprises: an optimization training step of performing optimization training on the first text type recognition model based on the third corpus to obtain a target text type recognition model.

7. The method of claim 6, wherein, The corpus obtaining step further comprises: obtaining a second text corpus, the second text corpus comprising the first text corpus and a newly added text set in a second preset time period; performing text type labeling on each corpus in the newly added corpus set based on the second text corpus; updating the labeling information of each corpus in the first corpus based on the newly added text set; adding the newly added corpus set after labeling to the first corpus after updating the labeling information to obtain a third corpus.

8. The method of claim 7, wherein, The obtaining of the second text corpus comprises: obtaining a newly added text set in a second preset time period; cleaning the newly added text set to remove mislabeled texts in the newly added text set; adding the cleaned newly added text set to the first text corpus to obtain a second text corpus.

9. The method of claim 7, wherein, The corpus obtaining step further comprises: obtaining expert-labeled corpora in the newly added corpus set; performing text type labeling on each corpus in the newly added corpus set based on the expert-labeled corpora.

10. The method of claim 1 or 6, wherein, The method further comprises: an online testing step of performing online testing on the target text type recognition model, and deploying the target text type recognition model to an online environment in a case where the testing is passed.

11. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, is arranged to perform the method of any one of claims 1 to 10. The processor executes the computer program to implement the model updating method of any one of claims 1-10.

12. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the model updating method of any one of claims 1-10.

13. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the model updating method of any one of claims 1-10. The computer program / instruction is executed by the processor to implement the model updating method of any one of claims 1-10.

Images