A method and system for peer review system collusion attack sample generation
By collecting and preprocessing target text and colluder text, constructing a candidate attack vocabulary and replacing words to generate adversarial samples, the problem of collusion attacks in peer review systems is solved, the success rate of adversarial samples is improved, and the algorithm complexity is reduced.
Patent Information
- Application Number
- CN202211652747.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-21
- Publication Date
- 2025-11-28
- Estimated Expiration
- 2042-12-21
AI Technical Summary
In existing peer review systems, adversarial sample generation methods that involve collusion between authors and reviewers to increase paper acceptance rates lack specificity, leading to unfair allocation results.
By collecting target text and colluder text, performing text preprocessing, searching for target attack words, constructing a candidate attack word list, and generating adversarial samples by replacing words according to the word list order, the adversarial attack effect is simulated.
The generated adversarial examples reduce the quality of peer review assignments, increase the success rate of adversarial examples, and reduce the time and complexity of the algorithm without being detected by humans.
Smart Images
Figure CN115809656B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of natural language processing of computer science, and particularly relates to a collusion attack sample generation method and device for a peer review system. BACKGROUND
[0002] Peer review is a process in which qualified peers evaluate, score and rank submitted projects in a field, so as to evaluate the author's writing level or whether the project application meets the standards of the subject field. It is a typical application of natural language processing (NLP). Since peer review can avoid the intervention of laymen, it plays an important role in ensuring the quality of reviewed projects and is considered to be the only scientific evaluation method in the field of scientific research management.
[0003] Although peer review has achieved great success in scientific research evaluation, how to use the peer review mode to effectively evaluate information fairly and efficiently is a useful and challenging open problem. In order to improve the transparency of peer review, enhance objectivity, improve the fairness of review results, reduce the pressure of reviewers, and improve the efficiency of the review process, researchers have been making various innovations and explorations in various fields of peer review mechanism. In order to promote high-quality peer review, it is crucial to distribute submitted papers to qualified reviewers. Past studies have shown that small changes in peer review quality can have far-reaching effects. One of the most widely used automatic distribution algorithms today is the TPMS (Toronto Paper Matching System) system, which uses natural language processing techniques to calculate the similarity between submitted papers and the publications of previous papers by reviewers. The distribution goal is to maximize the global total similarity, so as to measure the qualifications of reviewers.
[0004] Many past works on reviewer assignment develop algorithms to maximize cumulative similarity, i.e., the sum of similarities of all assigned reviewers and all papers. However, since the authors of submitted papers have the initiative to control their papers, in order to obtain good reviews from reviewers and thus improve the acceptance rate of their papers, the authors of the papers are likely to collude with the reviewers they know and submit well-designed adversarial samples to increase the probability that the peer review process will assign the submitted papers to colluding reviewers. Although existing text adversarial attack methods can generate adversarial samples quickly, there is no method for the peer review scene. Therefore, there is an urgent need for a collusion attack sample generation method for a peer review system. SUMMARY
[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a method and apparatus for generating adversarial attack samples for peer review systems, thereby achieving the goal of assigning adversarial samples to the intended colluders during the allocation process of the peer review system.
[0006] A method for generating samples of collusion-based adversarial attacks for peer review systems includes the following steps:
[0007] Step 1: Collect the target text and the texts of the colluders among the reviewers, and perform text preprocessing on the target text and the texts of the colluders to obtain the preprocessed word set;
[0008] Step 2: Find target attack words from the word set;
[0009] Step 3: Construct a candidate attack word list based on the target attack word;
[0010] Step 4: Replace words according to the order of the attack word list to generate adversarial samples.
[0011] Furthermore, the present invention also includes inputting the target text and adversarial sample into a peer review system respectively, comparing the results of assigning reviewers to them, in order to evaluate the effectiveness of the peer review system in assigning adversarial samples.
[0012] Furthermore, the specific method for step 1 is as follows:
[0013] Input target text P document P ori Document R of the colluding reviewers, and the author R of the target text. tar The author R who collected the target text tar Document P ori And the accomplice R of the target text P att Document R, for document P ori Perform stop word filtering and part-of-speech standardization on document R to obtain the processed word set;
[0014] Furthermore, the specific method for step 2 is as follows:
[0015] Two methods are combined to construct the candidate attack word list: one is to use a thesaurus to look up target words, and the other is to calculate word similarity based on a semantic dictionary. The specific combined operation is as follows:
[0016] First, collect documents P in descending order of word frequency. ori The vocabulary set P w R, a collection of collusion reviewer documents w Obtain the word set P w Thesaurus A w Then the thesaurus A w and word set Rw Take the intersection Find the word set R again w In the word set P w The words with a semantic distance gap of no more than δ are taken as The target attack word set is obtained in addition to
[0017] Further, the specific method of step 3 is as follows:
[0018] Construct a replacement word table, construct a candidate attack word table from the target attack word set, and use the replacement word pair (p w ,r w ) in the target attack word set to form a replacement word table That is, the candidate attack word table is obtained
[0019] The p w in the replacement word pair (p w ,r w ) is a word in the word set P w , and the word set P w is a word in the word set R w .
[0020] Further, the specific method of step 4 is as follows:
[0021] According to the order of the attack word table, replace the words to generate the adversarial text. According to the order in the candidate attack word table , replace the word p ori in the document P w with r w , thereby iteratively obtaining the adversarial text P att , where p w represents the word in the wth replacement word pair; calculate the attack budget |W'| according to the set α value and the word p w , |W'| ≤ p w · α; each time a group of word pairs (p ori ,r w ) is replaced in the document P w , the variable changed is incremented by one, until changed is greater than the attack budget |W'|, and the adversarial sample P att is returned.
[0022] A collusion attack sample generation system for a peer review system, comprising an original allocation module, an adversarial attack module, and a log module.
[0023] Original allocation module: used to display the peer review allocation result of the target text, which receives the target text input by the user and returns the peer review allocation result according to the selected similarity calculation method and allocation method;
[0024] The anti-attack module is configured to simulate the effect when the target text is subjected to the anti-attack, receives the attack parameters alpha and delta input by the user, and obtains the target text transmitted by the original allocation module, disturbs the target text according to an anti-sample generation algorithm to construct an anti-sample, and returns the anti-sample.
[0025] The log module is configured to obtain the original target sample and the anti-sample, compare them through a text difference measurement algorithm, and intuitively display the effect of the anti-attack through a comparison index.
[0026] The embodiments of the present application have the following beneficial effects:
[0027] The embodiments of the present application provide a collusion anti-attack sample generation method and device for a peer review system. The present application mainly aims at the problem that the peer review system is subjected to collusion between a text submitter and a reviewer, and constructs an anti-text to manipulate the result of the peer review allocation system. The present application explores a method of collusion anti-attack in a peer review scenario, which includes collecting target text and the text of a colluder in a reviewer and performing text preprocessing; finding a target attack word; constructing a candidate attack word table; and replacing a word according to the order of the attack word table to generate an anti-sample. The present application replaces the synonyms of the target text and the colluder text with the synonyms, makes the least changes to the sentences of the target text, generates an anti-sample that will not arouse human awareness in the maximum extent, reduces the quality of the peer review allocation, improves the success rate of the anti-sample, and reduces the time and complexity of the algorithm. BRIEF DESCRIPTION OF DRAWINGS
[0028] Figure 1 An application scenario schematic diagram of the collusion anti-attack sample generation method for the peer review system provided by the embodiments of the present application;
[0029] Figure 2 A flowchart schematic diagram of the collusion anti-attack sample generation method for the peer review system provided by the embodiments of the present application;
[0030] Figure 3 A structure schematic diagram of the collusion anti-attack sample generation device for the peer review system provided by the embodiments of the present application. DETAILED DESCRIPTION
[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.
[0032] Firstly, the application scenarios that can be provided by the present application are introduced, such as generating word-level adversarial samples for a peer review system.
[0033] As shown in the figure, a collusion adversarial attack sample generation method for a peer review system comprises the following steps: Figure 2
[0034] Step 1, collect target text and colluder text in participating reviewers, and perform text preprocessing on the target text and the colluder text to obtain a preprocessed word set;
[0035] Step 2, find a target attack word from the word set;
[0036] Step 3, construct a candidate attack word table according to the target attack word;
[0037] Step 4, replace words according to the order of the attack word table to generate an adversarial sample.
[0038] The first embodiment of the present application is as follows:
[0039] As shown in the figure, the present embodiment provides a collusion adversarial attack sample generation method for a peer review, which comprises at least the following steps: Figure 1
[0040] 201 Collect target text and colluder text in participating reviewers and perform text preprocessing.
[0041] Specifically, the peer review system first collects the text of all reviewers, and the data set input by the user is the target text P ori =[a1,a2,…,ak], the text of honest reviewers R1, R2 and R3, and the text of colluding reviewers R4=[b1,b2,…,bm] and R5=[c1,c2,…,cn], wherein k, m and n are the lengths of the texts.
[0042] 202 Find a target attack word.
[0043] Specifically, the user inputs the numbers of colluding reviewers R4 and R5, the number of colluding reviewers is 2, the maximum word meaning distance δ<0.5, and one of the methods for finding a target attack word is to use a synonym dictionary (such as WordNet) to look up the target attack word, and the other is to find the similarity of words based on a semantic dictionary. By using the synonym set, the target word a1 in P ori is found to be synonymous with b1 in the text R4, and the target word a2 in P ori is found to have a word meaning distance δ<0.5 with c1 in the text R5.
[0044] 203 Construct a candidate attack word table;
[0045] Specifically, for target words a1 and synonym b1, target words a2 and word c1, first, the words that are both synonyms and the word meaning distance delta <0.5 are listed in the candidate attack word list, then the words that only belong to synonyms are listed in the candidate attack word list, and finally the words that only belong to the word meaning distance delta <0.5 are listed in the candidate attack word list. The replacement word pair (a1, b1) and (a2, c1) is generated by the target word and the corresponding word in the candidate attack word list, and the replacement word list is generated
[0046] 204, according to the order of the candidate attack word list, the words are replaced to generate the adversarial sample.
[0047] Specifically, P att a1 is replaced by b1, a2 is replaced by c1, and the adversarial sample is returned as P att =[b1, c1, …, ak].
[0048] The application embodiment further provides an application scenario and an embodiment running interface diagram for implementing the above method embodiment.
[0049] Referring to Figure 1 , the application scenario diagram of the application embodiment is shown.
[0050] As Figure 1 shown, when there is no attack, the peer review system collects the text of all reviewers and calculates the similarity between the text and the submitted paper, and the result is assigned to the target paper P ori is assigned to R1, R2 and R3 three honest reviewers, when there is an attack, the author of the target paper and the reviewers R4 and R5 in the reviewer pool collude, this embodiment changes the original submission paper into an adversarial text through a certain algorithm, and the assignment system calculates the similarity between the adversarial text and all reviewers including and in the reviewer pool, and then assigns it to R4 and R5, so as to achieve the purpose of collusion attack with reviewers.
[0051] The embodiment provides a collusion attack method in a peer review scene, the method comprising collecting target text and text of colluders in participating reviewers and performing text preprocessing; finding target attack words; constructing a candidate attack word list; and replacing words according to the order of the attack word list to generate an adversarial sample. The present application replaces the synonyms and near synonyms of the target text and the colluder text, makes the smallest changes to the sentences of the target text, generates an adversarial sample that will not arouse human awareness in the maximum extent, reduces the quality of peer review distribution, improves the success rate of the adversarial sample, and reduces the time and complexity of the algorithm.
[0052] The second embodiment of the present application is:
[0053] As Figure 3As shown, the embodiment provides a peer review collusion attack sample generation device, comprising:
[0054] The original allocation module 301 is used to display the peer review allocation result of the target text, receives the target text input by the user, and returns the peer review allocation result according to the selected similarity calculation method and allocation method; wherein the similarity calculation method includes TF-IDF, LDA and LSI.
[0055] Specifically, first, the peer review system collects all the texts of the reviewers, and the data set input by the user is the target text P ori =[a1,a2,…,ak], the texts of the honest reviewers R1, R2 and R3, the texts of the colluding reviewers R4=[b1,b2,…,bm] and R5=[c1,c2,…,cn], wherein k, m and n are the lengths of the texts. The similarity of the submitted text and the reviewer text is calculated by selecting TF-IDF, LDA or LSI, and P ori has the highest similarity with the texts of R1, R2 and R3, so according to the allocation method of maximizing the global total similarity, the target text P ori is allocated to the three honest reviewers R1, R2 and R3.
[0056] The adversarial attack module 302 is used to simulate the effect when the target text is subjected to adversarial attack, receives the attack parameters α and δ input by the user, and obtains the target text transmitted by the original allocation module, performs perturbation on the target text according to the adversarial sample generation algorithm to construct an adversarial sample, and returns the adversarial sample;
[0057] Specifically, the user inputs the numbers of the colluders R4 and R5, the number of colluders is 2, and the maximum word meaning distance δ<0.5. By replacing the words a1 and a2 in P ori with the synonymous words b1 and c1 in R3 and R4, and the words with a word meaning distance δ<0.5 in R3 and R4, for the target word a1 and the candidate attack word b1, and the target word a2 and the attack word c1, first, the words that are both synonymous and have a word meaning distance δ<0.5 are listed in the candidate attack word table, then the words that are only synonymous are listed in the candidate attack word table, and finally the words that are only with a word meaning distance δ<0.5 are listed in the candidate attack word table, according to the order of the attack word table, the words are replaced, a1 in P att is replaced by b1, and a2 is replaced by c1, and the returned adversarial sample is P att =[b1,c1,…,ak].
[0058] The user inputs the numbers R4 and R5 of the colluding reviewers, the number of colluding reviewers is 2, the maximum word meaning distance δ is less than 0.5, one of the methods for finding the target attack word is to use a thesaurus (such as WordNet) to look up the attack target word, and the other method is to find the similarity degree of the word based on a semantic dictionary. The target word a1 in P ori is found by using the synonym set to find the synonym b1 in the text R4, and the target word a2 in P ori and the word c1 with a word meaning distance δ less than 0.5 in the text R5 are found. For the target word a1 and the synonym b1, and the target word a2 and the word c1, first, the words that are both synonyms and have a word meaning distance δ less than 0.5 are listed in the candidate attack word table, then the words that are only synonyms are listed in the candidate attack word table, and finally, the words that are only with a word meaning distance δ less than 0.5 are listed in the candidate attack word table.
[0059] The log module 303 is used to obtain the original target sample and the adversarial sample, compare them by using a text difference measurement algorithm, and intuitively display the effect of the adversarial attack by using a comparison index.
[0060] Specifically, the target text and the adversarial sample are compared by using the average word mover's distance (AWMD) and the average sentence similarity distance (ASSD). The calculation formula of the average word mover's distance (AWMD) is as follows:
[0061]
[0062] The calculation formula of the average sentence similarity distance (ASSD) is as follows:
[0063]
[0064] The device for generating colluding adversarial attack samples for a peer review system provided in this embodiment is characterized in that it comprises: an original allocation module 301, which is used to display the peer review allocation result of a target text, receives the target text input by a user, and returns the peer review allocation result according to a selected similarity calculation method and an allocation method; wherein the similarity calculation method comprises TF-IDF, LDA and LSI. An adversarial attack module 302, which is used to simulate the effect when the target text is subjected to an adversarial attack, receives the attack parameters α and δ input by a user, obtains the target text transmitted by the original allocation module, disturbs the target text to construct an adversarial sample according to an adversarial sample generation algorithm, and returns the adversarial sample. A log module 303, which is used to obtain the original target sample and the adversarial sample, compare them by using a text difference measurement algorithm, and intuitively display the effect of the adversarial attack by using a comparison index.
[0065] An embodiment of the present application also provides a computer readable storage medium comprising a stored computer program, wherein the computer readable storage medium controls a device in which the computer readable storage medium is located to perform the method for generating collusion attack samples of a peer review system as described above when the computer program is run.
[0066] In the above-described embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.
[0067] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other manners. Among them, the above-described device embodiments are only schematic, for example, the division of the modules can be a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed ones can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms. The modules described as separate components can be or can not be physically separated, and the components displayed as modules can be or can not be physical modules, that is, they can be located in one place, or can be distributed to a plurality of modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment scheme.
[0068] In addition, each functional module in each embodiment of the present application can be integrated in one processing module, or each module can exist physically, or two or more modules can be integrated in one module. The above integrated module can be realized in the form of hardware or in the form of software functional module.
[0069] The above is the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and modifications can be made, and these improvements and modifications are also considered to be within the protection scope of the present application.
[0070] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware. The program can be stored in a computer readable storage medium, and when the program is executed, the processes of the above-mentioned embodiment methods can be included. The storage medium can be a magnetic disc, an optical disc, a Read-Only Memory (ROM) or a Random Access Memory (RAM).
Claims
1. A method for peer review system collusion attack sample generation, characterized in that Comprising the following steps: Step 1, collect the target text P of the author R tar The target text P of the author R ori The target text P of the author R ori The target text P of the author R att The target text P of the author R ori and the document R, and perform stop word filtering and part-of-speech standardization operations on the target text P and the document R to obtain a processed word set; Step 2, find the target attack word from the word set; Step 3, construct a candidate attack word list according to the target attack word; Step 4, replace the words according to the order of the candidate attack word list to generate the adversarial attack sample; Step 2 specific method as follows: Two methods are combined to construct the target attack word set, one is to use the synonym dictionary to look up to obtain the target attack word, and the other is to calculate the similarity of the word based on the semantic dictionary to obtain the target attack word; The specific combination operation is as follows: First, collect the target text P in descending order of word frequency. ori The vocabulary set P w Collect the vocabulary of document R. w Obtain the word set P w Thesaurus A w Then the thesaurus A w and word set R w Taking the intersection yields Then find the vocabulary set R w Chinese and vocabulary collection P w Words whose semantic distance does not exceed δ are considered as The supplementation yields the target attack word set; Step 3 specific method as follows: The replacement word table is constructed, a candidate attack word table is constructed through a target attack word set, and a replacement word pair (p w , w ) is composed according to the target attack word set to constitute the replacement word table That is, the candidate attack word table is obtained The replacement word pair (p) w ,r w p in ) w For the vocabulary collection P w The word in the text, r w For the vocabulary set R w Words in the text; Step 4 specific method as follows: According to the order of the candidate attack word table, a word is replaced to generate an adversarial text; according to the order in the candidate attack word table , a word p ori in the target text P w is replaced by r w , thereby iteratively obtaining an adversarial attack sample P att , calculating an attack budget |W'| according to a set α value and the word p w , and each replacement of a group of words (p w , r w ) in the target text P ori causes a variable changed to be accumulated once, until changed is greater than the attack budget |W'|, and the adversarial attack sample P att is returned; The said alpha and delta are attack parameters input by the user.
2. The method of claim 1, wherein Also includes: The target text P is respectively ori and adversarial attack sample P att The data is input into the peer review system, and the results of the assigned reviewers are compared to evaluate the effectiveness of the peer review system in assigning adversarial attack samples.
3. A collusion attack sample generation system for a peer review system, the system comprising: The system is used to implement the method as claimed in any one of claims 1-2, specifically comprising: original allocation module, adversarial attack module and log module; Original allocation module: used to show the peer review allocation result of the target text, this module receives the target text input by the user, and returns the peer review allocation result according to the selected similarity calculation method and allocation method; Adversarial attack module: used to simulate the effect when the target text is subjected to adversarial attack, this module receives the attack parameters alpha and delta input by the user, and obtains the target text transmitted by the original allocation module, and disturbs the target text to construct the adversarial attack sample according to the adversarial attack sample generation algorithm, and returns the adversarial attack sample; Log module, used to obtain the original target sample and the adversarial attack sample, compare them through the text difference measurement algorithm, and intuitively show the effect of the adversarial attack through the comparison index.