Apparatus and method for detecting source code generated by artificial intelligence

Abstract

The present disclosure provides a source code detection device and method capable of checking in advance for legal disputes or security vulnerabilities caused by source code generated by artificial intelligence by performing the steps of: dividing an input source code into a plurality of tokens; performing neural network operations based on a large language model on the divided plurality of tokens to estimate a token selection probability for each token; obtaining indicator score information for the source code including a confusion degree representing the difference in distribution between a token predicted to be generated by the artificial intelligence based on a previously input token and a currently input token based on the estimated token selection probability; generating a feature vector composed of indicator score information and statistical information including the length of the source code; and inputting the feature vector into a machine learning neural network model to determine whether the author of the source code is a human or an artificial intelligence, thereby accurately distinguishing whether the creator of the source code is a human or an artificial intelligence.

Description

Technology Field

[0001] The present disclosure relates to an AI-generated source code detection device and method, and more specifically, to an AI-generated source code detection device and method using code characteristic analysis and machine learning. Background Technology

[0002] Recent advancements in artificial intelligence technology are particularly notable for the rapid development of Big Language Models (MBMs). Big Language Models dramatically improve natural language processing and generation capabilities by learning from vast amounts of text data. These models demonstrate performance approaching human levels in language comprehension and generation and are being used in various application fields. In particular, the influence of Big Language Models is growing in the field of source code generation. By learning the grammar and patterns of programming languages, these models can generate source code that is natural and accurate, just as if written by a human.

[0003] Source code generation through such artificial intelligence is significantly improving the productivity of the software development process. AI models have the potential to reduce the workload of developers and maximize project efficiency by rapidly writing complex algorithms, automating repetitive coding tasks, and contributing to improved code quality.

[0004] However, problems may arise, such as AI models generating source code protected by copyright and licenses learned during training, or generating source code with security vulnerabilities; this can negatively impact the reliability of the software. Therefore, to enhance the reliability of software development and prevent various issues—such as copyright protection and security problems—that may occur during the development process, there is a need for technology capable of distinguishing between source code written by humans and source code written by AI, thereby detecting source code generated by AI models. The problem to be solved

[0005] The purpose of the present disclosure is to provide a source code detection device and method capable of distinguishing and determining whether the creator of source code is a human or an artificial intelligence.

[0006] The purpose of the present disclosure is to provide a source code detection device and method for distinguishing the creator of source code based on various indicator scores representing familiarity with a neural network model, along with statistical information about the source code. means of solving the problem

[0007] According to one embodiment of the present disclosure, a source code detection device comprises: a memory; and a processor that executes at least a portion of an operation according to a program stored in the memory, wherein the processor divides an input source code into a plurality of tokens, performs neural network operations based on a large language model on the divided plurality of tokens to estimate a token selection probability for each token, and obtains indicator score information for the source code including a confusion degree representing the difference in distribution between a token predicted to be generated by an artificial intelligence based on a previously input token and a currently input token based on the estimated token selection probability; generates a feature vector composed of the indicator score information and statistical information including the length of the source code; and inputs the feature vector into a machine-learned neural network model to determine whether the author of the source code is a human or an artificial intelligence.

[0008] The above indicator score information may include a log-likelihood representing the suitability of the artificial intelligence of the plurality of tokens, a rank representing the selection ranking of the input token among the plurality of tokens that can be selected next based on the previously input token, and an entropy representing the prediction difficulty representing the level at which each sequentially input token is difficult to select from the previously input token.

[0009] The above statistical information may include the length of the source code, along with the average length of multiple variable names and the average length of multiple function names included in the source code.

[0010] The above machine-learned neural network model can be pre-trained based on training data labeled as to whether the author of the source code is a human or an artificial intelligence.

[0011] A source code detection method according to another embodiment of the present disclosure is a method performed by a processor, comprising: dividing an input source code into a plurality of tokens; performing neural network operations based on a large language model on the divided plurality of tokens to estimate a token selection probability for each token; and obtaining indicator score information for the source code including a confusion degree representing the difference in distribution between a token predicted to be generated by an artificial intelligence based on a previously input token and a currently input token based on the estimated token selection probability; generating a feature vector composed of the indicator score information and statistical information including the length of the source code; and inputting the feature vector into a machine-learned neural network model to determine whether the author of the source code is a human or an artificial intelligence. Effects of the invention

[0012] The source code detection device and method of the present disclosure can check in advance for legal disputes or security vulnerabilities caused by source code generated by artificial intelligence by accurately distinguishing whether the creator of the source code is a human or an artificial intelligence based on various indicator scores representing familiarity with a neural network model along with statistical information about the source code. Brief explanation of the drawing

[0013] FIG. 1 shows a configuration of a source code detection device according to one embodiment, classified according to operation. Figure 2 shows an example of the detailed configuration of the token probability analysis model of Figure 1. Figure 3 shows an example of the detailed configuration of the statistical analysis model of Figure 1. FIG. 4 illustrates a source code detection method according to one embodiment. FIG. 5 is a diagram illustrating a computing environment including a computing device according to one embodiment. Specific details for implementing the invention

[0014] Hereinafter, specific embodiments according to embodiments of the present disclosure will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, apparatuses, and / or systems described herein. However, this is merely illustrative and the present invention is not limited thereto.

[0015] In describing the embodiments of the present disclosure, detailed descriptions of known technology related to the present invention are omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of the embodiments. Furthermore, terms described below are defined with consideration of their functions in the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe specific embodiments and should not be limiting. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as “include” or “compose” are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described. Additionally, terms such as “...part,” “...unit,” “module,” and “block” described in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software.

[0016] FIG. 1 shows a configuration of a source code detection device according to one embodiment classified according to operation, FIG. 2 shows an example of a detailed configuration of the token probability analysis model of FIG. 1, and FIG. 3 shows an example of a detailed configuration of the statistical analysis model of FIG. 1.

[0017] Referring to FIG. 1, a source code detection device according to one embodiment may include a source code acquisition module (11), a tokenization module (12), a token probability analysis model (13), an indicator score calculation module (14), a statistical analysis module (15), a feature vector generation module (16), and a generator identification model (17).

[0018] The source code acquisition module (11) acquires the source code (W) to be analyzed. Here, the source code (W) is a code that must be determined whether it is a code generated by a human or a code generated by artificial intelligence, and it may be a code written in any programming language. The source code acquisition module (11) may be implemented as a communication module that receives the source code (W) from a storage medium or external device where the source code (W) is stored, and in some cases, it may be implemented as an input module.

[0019] The tokenization module (12) receives the source code (W) obtained by the source code acquisition module (11), and the authorized source code is converted into a plurality of tokens (w1, w2, …, w N It is split by separating with ). Here, as an example, the input source code (W = {w1, w2, …, w N This is explained by assuming that}) is divided into N tokens.

[0020] The token probability analysis model (13) is N tokens (w1, w2, …, w) obtained from the source code. N ) Estimates the probability that each will be selected. The token probability analysis model (13) is implemented as a neural network model trained based on a Large Language Model (LLM), and N tokens (w1, w2, …, w) that are input in succession are analyzed. N Estimate and output the probability that each will be selected.

[0021] A massive language model is a type of artificial intelligence model that is generally configured to perform tasks involving the processing of various languages ​​by learning large-scale text data. Currently, it is also being utilized to generate source code, similar to many well-known generative models. In this way, a generative model based on a massive language model that generates source code is configured to select appropriate code that must be added following the previously generated code based on probability through neural network operations. The reason the token probability analysis model (13) is implemented based on a massive language model is that, in most cases, the source code generated by artificial intelligence is generated by a generative model based on a massive language model. That is, by utilizing the neural network model used to generate source code, N tokens (w1, w2, …, w) are generated by the neural network model. N This is to accurately estimate the probability that each one will be selected. That is, the token probability analysis model (13) can be constructed using a generative neural network model trained to generate source code.

[0022] The token probability analysis model (13) utilizes the characteristics of this large language model to divide N tokens (w1, w2, …, w) from the source code (W). N When each of the following is input sequentially, the token selection probability (P(w), which is the conditional probability that the token input following the previous tokens is selected in the generative model, is given by... i |w1, w2, … , w i-1 Estimate and output the token selection probability (P(w). Here, the token selection probability (P(w)) is estimated. i |w1, w2, … , w i-1 The reason this is output as a conditional probability is that it represents the probability that the next token will be selected based on the previously selected token.

[0023] In the token probability analysis model (13), the probability of selecting a token (P(w)) i |w1, w2, … , w i-1 When )) is obtained, the indicator score calculation module (14) calculates the obtained token selection probability (P(wi |w1, w2, … , w i-1 Calculate various indicator scores based on )). Here, the indicator scores may consist of scores for various indicators that can represent the familiarity with the source code (W) of the token probability analysis model (13), which is a neural network model.

[0024] The indicator score calculation module (14) may include at least one of a confusion calculation module (21), a log-likelihood calculation module (22), a rank calculation module (23), and an entropy calculation module (24), as illustrated in FIG. 3.

[0025] The confusion calculation module (21) calculates the token selection probability (P(w) i |w1, w2, … , w i-1 Perplexity (PP(W)) is calculated for each token based on )). Perplexity (PP(W)) is an indicator for measuring the performance of a language model, where the generative model generating source code (W) [calculates] the previously input tokens (w1, w2, …, w i-1 Tokens predicted to be generated based on ) and currently input tokens (w i It represents the error between ). That is, confusion (PP(W)) represents the difference in distribution between the tokens predicted to be generated by the generative model and the tokens currently input. Therefore, if the confusion (PP(W)) is low, it indicates that tokens predicted by the neural network model have been input consecutively, which means that the source code (W) has a high familiarity with the neural network model. In other words, it can be seen that the source code (W) is likely to have been generated by the neural network model. On the other hand, if the confusion (PP(W)) is high, it means that tokens not predicted by the neural network model have been input, and in this case, it can be said that the source code (W) was likely generated by a human.

[0026] The confusion level (PP(W)) can be calculated according to Equation 1.

[0027]

[0028] (Here, exp(·) represents the exponential function.)

[0029] The log-likelihood calculation module (22) calculates the log-likelihood, which indicates the goodness of fit for the neural network model. Generally, the log-likelihood (LL(W)) is used as an indicator of how well a generative model trained on a dataset consisting of multiple source codes can represent the dataset, but here, it is utilized in reverse, using multiple tokens (w1, w2, …, w) obtained from the source code (W). N It is used to measure the goodness of fit, indicating how well it was obtained to fit the generative model.

[0030] The log-likelihood (LL(W)) can be calculated as shown in Equation 2.

[0031]

[0032] Conversely to confusion (PP(W)), if the log-likelihood (LL(W)) is high, it means that the source code (W) has high familiarity with the neural network model, and therefore it is highly likely that it was generated by the neural network model. Conversely, if the log-likelihood (LL(W)) is low, it means that it is highly likely that it was generated by a human.

[0033] The rank calculation module (23) calculates a rank score by evaluating the absolute rank of each token in the source code based on the previous context. The rank calculation module (23) calculates the rank score based on the tokens (w1, w2, …, w) that were previously input to the generative model. i-1 Multiple tokens that can be selected next based on ) (w i,1 , … , w i,M Token selection probability for each (P(w) i |w1, w2, … , w i-1 Check the probability of selecting the confirmed token (P(w)). i |w1, w2, … , w i-1 Multiple tokens (w) that can be selected based on ))i,1 , … , w i,M ) are sorted sequentially. In this case, the token with the highest probability of selection has the highest rank with the smallest value (here, for example, 1), and the token with the lowest probability of selection has the lowest rank with the largest value (M). Then, depending on the previously input token, multiple tokens (w i,1 , … , w i,M Among ) currently entered token(w i Check the rank of ) and obtain Rank(W) as in Equation 3.

[0034]

[0035] Rank(w i |w1, w2, … , w i-1 The lower the )) the token(w i Since this means that ) is highly likely to be selected by the neural network model, Rank(Rank(w i |w1, w2, … , w i-1 A low Rank(w) indicates that the source code (W) is likely to have been generated in the neural network model, and the Rank(Rank(w i |w1, w2, … , w i-1 If )) is high, it indicates that the source code (W) is likely to have been created by a human.

[0036] Finally, the entropy calculation module (24) is a plurality of tokens (w1, w2, …, w N The prediction difficulty for each ) is accumulated to calculate the entropy (H(W)). Entropy (H(W)) is an indicator used in information theory to measure the uncertainty of a system. Accordingly, the entropy calculation module (24) calculates the previously input tokens (w1, w2, …, w) among a plurality of tokens that the neural network model can select. i-1 Based on ), the currently entered token (w i Other tokens besides ) (w j , here w j ≠ w iToken entropy (H(w) representing the probability of selecting ) i |w1, w2, … , w i-1 Calculate )) first according to mathematical formula 4.

[0037]

[0038] Here, V represents the size of the token set containing all tokens that the neural network model can select.

[0039] N tokens (w1, w2, …, w) obtained from source code (W) N Token entropy (H(w) for each i |w1, w2, … , w i-1 When )) is calculated, the entropy calculation module (24) calculates the token entropy (H(w) calculated as in Equation 5. i |w1, w2, … , w i-1 The entropy (H(W)) for the source code can be calculated as the average of )).

[0040]

[0041] The tokenization module (12), the token probability analysis model (13), and the indicator score calculation module (14) can be said to be an indicator score extraction module that integrates to extract an indicator score from source code.

[0042] Meanwhile, the statistical analysis module (15) obtains various information that can be obtained directly from the source code. Referring to FIG. 3, the statistical analysis module (15) may include a source code length analysis module (31), a variable name length analysis module (32), and a function name length analysis module (33).

[0043] The source code length analysis module (31) checks the length of the source code itself. The source code length analysis module (31) checks the total number of authorized tokens (N) to evaluate the complexity and scale of the source code. Then, the variable name length analysis module (32) checks the length of each name for multiple variables included in the source code and calculates the average of the checked variable name lengths. Similarly, the function name length analysis module (33) checks the length of each name for multiple functions included in the source code and calculates the average of the checked variable name lengths.

[0044] Generally, when artificial intelligence generates source code, it tends to generate source code in a very standard manner. In contrast, when a person writes source code, there is a tendency to simplify it by abbreviating variable names or function names for convenience, and there is also a tendency to write source code as short as possible. Therefore, if the length of the source code and the average length of variable names and function names are shorter than the statistically verified average length of source code or variable names and function names, it can be inferred that the source code was written by a person, and conversely, if it is longer, it is likely that the source code was generated by artificial intelligence. However, the length of the source code can vary significantly depending on the task performed by each code, and the length of variable names and function names can also vary significantly depending on the tendencies of the person writing the source code. Therefore, it is difficult to verify the source code length or the average length of variable names and function names obtained from the statistical analysis module (15) simply by comparing absolute values.

[0045] And this is various indicator information obtained from the indicator score calculation module (14), namely confusion (PP(W)), log-likelihood (LL(W)), and rank (Rank(w)). i |w1, w2, … , w i-1 The same applies to )) and entropy (H(W)).

[0046] Therefore, here, neural network operations are performed using a generator discrimination model (17), which is a neural network model pre-trained in a machine learning manner, on various indicator information obtained from the indicator score calculation module (14) and statistical information obtained from the statistical analysis module (15), thereby determining whether the source code was written by a human or generated by artificial intelligence. Additionally, the feature vector generation module (16) generates a feature vector based on indicator information and statistical information so that the generator discrimination model (17) can easily receive various indicator information and statistical information.

[0047] The feature vector generation module (16) obtains indicator information, such as confusion (PP(W)), log-likelihood (LL(W)), and rank (Rank(w), from the indicator score calculation module (14). i |w1, w2, … , w i-1 A feature vector is generated by combining )) and entropy (H(W)), and the source code length, variable name, and function name average length, which are statistical information obtained from the statistical analysis module (15). Here, the feature vector is, for example, confusion (PP(W)), log-likelihood (LL(W)), and rank (Rank(w)). i |w1, w2, … , w i-1 It can be generated by simply arranging the average length of variable names and function names, entropy (H(W)), source code length, and, in some cases, by generating feature vectors with various combinations. Additionally, feature vectors can be constructed by applying different weights to each piece of information.

[0048] The generator determination model (17) receives a feature vector and performs neural network operations to determine whether the source code was written by a human or by artificial intelligence, and outputs the determination result. The generator determination model (17) can determine the generator of the source code by implementing a neural network model that is pre-trained using a training dataset in which, for example, source code written by a human is labeled with a truth value of 1 and source code written by artificial intelligence is labeled with a truth value of 0.

[0049] A source code detection device of one embodiment can accurately determine whether the creator of the source code is a human or an artificial intelligence based on various indicator scores representing familiarity with a neural network model, along with statistical information about the source code. Therefore, by detecting source code generated by artificial intelligence, legal disputes or security vulnerabilities can be checked in advance.

[0050] In the illustrated embodiments, each component may have different functions and capabilities in addition to those described above and may include additional components not described. Additionally, in one embodiment, each component may be implemented using one or more physically separated devices, or by one or more processors or a combination of one or more processors and software, and may not be clearly distinguished in specific operation as in the illustrated examples.

[0051] And the source code detection device illustrated in FIG. 1 may be implemented within a logic circuit by hardware, firmware, software, or a combination thereof, or may be implemented using a general-purpose or specific-purpose computer. The device may be implemented using a hardwired device, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc. Additionally, the device may be implemented as a system-on-chip (SoC) including one or more processors and controllers.

[0052] Furthermore, the source code detection device may be installed in the form of software, hardware, or a combination thereof on a computing device or server equipped with hardware elements. A computing device or server may refer to various devices that include, in whole or in part, communication devices such as communication modems for communicating with various devices or wired / wireless communication networks, memory for storing data for executing programs, and microprocessors for executing programs to perform calculations and commands.

[0053] FIG. 4 illustrates a source code detection method according to one embodiment.

[0054] Referring to FIG. 4, a source code detection method of one embodiment first receives a source code (W) to be analyzed (51). Then, the length of the input source code (W) and the length of the variable name of each of the multiple variables included in the source code (W) and the length of the function name of each of the multiple functions are checked (52). Once the variable name length and the function name length are checked, the average value of the variable name length and the average value of the function name length are calculated and obtained as statistical information about the source code (W) along with the length of the source code (W) (53).

[0055] Meanwhile, in the source code detection method, indicator score information can be obtained separately from the process of obtaining statistical information. To obtain indicator score information, tokenization is first performed to divide the source code (W) into multiple tokens (54). Then, N tokens (w1, w2, …, w) obtained from the source code (W) are used. N By inputting ) into a neural network model trained based on a large language model and performing neural network operations, N tokens (w1, w2, …, w N The token selection probability (P(w) representing the probability that each one is selected i |w1, w2, … , w i-1 Estimates )) (55).

[0056] And the estimated token selection probability (P(w i |w1, w2, … , w i-1 A plurality of indicator scores including confusion (PP(W)) are calculated from )) to obtain indicator score information (56). Here, the indicator score information includes log-likelihood (LL(W)) and rank (Rank(w)) along with confusion (PP(W)). i |w1, w2, … , w i-1 At least one of )) and entropy (H(W)) may be included.

[0057] Confusion (PP(W)) is an indicator representing the difference in distribution between the tokens predicted to be generated by the generative model and the currently input tokens, and can be calculated according to Equation 1, and log-likelihood (LL(W)) is a number of tokens (w1, w2, …, w N ) can be obtained according to Equation 2 as an indicator of how well it is acquired to fit the generative model. And Rank(Rank(w i |w1, w2, … , w i-1 )) is the previously entered tokens (w1, w2, …, w i-1 Multiple tokens that can be selected next based on ) (w i,1 , … , w i,M Among the input tokens (w i) represents the ranking in which it is selected and can be calculated as shown in Equation 3. Meanwhile, entropy (H(W)) is for each sequentially input token (w i ) is the previously entered token(w1, w2, …, w i-1 It can be calculated according to mathematical formulas 4 and 5 as an indicator of prediction difficulty indicating a level that is difficult to select from ).

[0058] When statistical information and indicator score information are obtained, a feature vector is generated using the obtained statistical information and indicator score information (57). Here, the feature vector includes confusion (PP(W)), log-likelihood (LL(W)), and rank (Rank(w)) included in the indicator score information. i |w1, w2, … , w i-1 It can be generated as a one-dimensional vector containing at least one of )) and entropy (H(W)), the length of source code (W) included in statistical information, the average length of variable names, and the average length of function names. However, in some cases, a feature vector may be generated by applying different weights to each indicator score included in the indicator score information, the length of source code (W) included in statistical information, the average length of variable names, and the average length of function names.

[0059] When a feature vector is generated, the generated feature vector is input into a neural network model that has been pre-trained with training data labeled as whether the source code is written by a human or generated by an artificial intelligence, and by performing neural network operations, the author of the input source code is determined to be a human or an artificial intelligence and a judgment result is output (58) /

[0060] Although FIG. 4 describes each process as being executed sequentially, this is merely an illustrative description, and a person skilled in the art can apply various modifications and variations by changing the order described in FIG. 4, executing one or more processes in parallel, or adding other processes, within the scope of not departing from the essential characteristics of the embodiment of the present invention.

[0061] FIG. 5 is a diagram illustrating a computing environment including a computing device according to one embodiment.

[0062] In the illustrated embodiments, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those described below. The illustrated computing environment (90) may include a computing device (91) to perform the source code detection method illustrated in FIG. 4. In one embodiment, the computing device (91) may be one or more components included in the source code detection device illustrated in FIG. 1.

[0063] A computing device (91) includes at least one processor (92), a computer-readable storage medium (93), and a communication bus (95). The processor (92) may enable the computing device (91) to operate according to the exemplary embodiment described above. For example, the processor (92) may execute one or more programs (94) stored in the computer-readable storage medium (93). The one or more programs (94) may include one or more computer-executable instructions, and the computer-executable instructions may be configured to enable the computing device (91) to perform operations according to the exemplary embodiment when executed by the processor (92).

[0064] The communication bus (95) interconnects various other components of the computing device (91), including the processor (92) and the computer-readable storage medium (93).

[0065] The computing device (91) may also include one or more input / output interfaces (96) and one or more communication interfaces (97) that provide an interface for one or more input / output devices (98). The input / output interfaces (96) and communication interfaces (97) are connected to a communication bus (95). The input / output devices (98) may be connected to other components of the computing device (91) through the input / output interfaces (96). An exemplary input / output device (98) may include an input device such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or an output device such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (98) may be included inside the computing device (91) as a component constituting the computing device (91), or it may be connected to the computing device (91) as a separate device distinct from the computing device (91).

[0066] Although the present invention has been described in detail above through representative embodiments, those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the technical spirit of the appended claims.

Claims

Claim 1 A device comprising: a memory; and a processor that executes at least a portion of an operation according to a program stored in the memory, wherein the processor divides an input source code into a plurality of tokens, performs neural network operations based on a large language model on the divided plurality of tokens to estimate a token selection probability for each token, and obtains indicator score information for the source code including a confusion level representing the difference in distribution between a token predicted to be generated by an artificial intelligence based on a previously input token and a currently input token based on the estimated token selection probability; generates a feature vector composed of the indicator score information and statistical information including the length of the source code; and inputs the feature vector into a machine-learned neural network model to determine whether the author of the source code is a human or an artificial intelligence, wherein the indicator score information includes a log-likelihood representing the suitability of the plurality of tokens to the artificial intelligence, a rank representing the selection ranking of the input token among a plurality of tokens that can be next selected based on a previously input token, and an entropy representing the prediction difficulty representing the level at which each sequentially input token is difficult to select from the previously input token, and wherein the processor obtains N of the source code (W). Token(w1, w2, …, w N The token selection probability (P(w), which is the conditional probability for each i |w1, w2, … , w i-1 Based on )), the above log-likelihood (LL(W)) is mathematically Source code detection device that calculates according to Claim 2 delete Claim 3 A source code detection device according to claim 1, wherein the statistical information includes the length of the source code, the average length of a plurality of variable names and the average length of a plurality of function names included in the source code. Claim 4 In claim 1, the machine-learned neural network model is a source code detection device that is pre-trained based on learning data labeled as to whether the author of the source code is a human or an artificial intelligence. Claim 5 In claim 1, the processor obtains N tokens (w1, w2, …, w) from the source code (W). N The token selection probability (P(w), which is the conditional probability for each i |w1, w2, … , w i-1 Based on )), the above confusion level (PP(W) is mathematically A source code detection device that calculates according to (where exp(·) represents an exponential function). Claim 6 delete Claim 7 In claim 1, the processor obtains N tokens (w1, w2, …, w) from the source code (W). N Among ) previously entered tokens(w1, w2, …, w i-1 Multiple tokens that can be selected next based on ) (w i,1 , … , w i,M Token selection probability for each (P(w) i |w1, w2, … , w i-1 Check )) to sort sequentially, and multiple sequentially sorted tokens (w i,1 , … , w i,M Among ) currently entered token(w i By checking the rank of ), the above rank (Rank(W)) is mathematically Source code detection device that calculates according to Claim 8 In paragraph 1, the processor is among a plurality of tokens selectable by the artificial intelligence a previously input token (w1, w2, …, w i-1 Based on ), the currently entered token (w i Other tokens besides ) (w j , here w j ≠ w i Token entropy (H(w) representing the probability of selecting ) i |w1, w2, … , w i-1 )) is a mathematical formula (where V is the size of the token set containing all tokens that the neural network model can select) calculated according to the calculated token entropy (H(w i |w1, w2, … , w i-1 Source code detection device that calculates the above entropy as the average of )). Claim 9 A method performed by a processor comprising: dividing an input source code into multiple tokens; performing neural network operations based on a large language model on the divided multiple tokens to estimate a token selection probability for each token; obtaining indicator score information for the source code including a confusion level representing the difference in distribution between a token predicted to be generated by an artificial intelligence based on a previously input token and a currently input token based on the estimated token selection probability; and generating a feature vector composed of the indicator score information and statistical information including the length of the source code. and input the feature vector into a machine-learned neural network model to determine whether the author of the source code is a human or an artificial intelligence, wherein the indicator score information includes a log-likelihood representing the suitability of the plurality of tokens to the artificial intelligence, a rank representing the selection ranking of the input token among a plurality of tokens that can be next selected based on the previously input token, and an entropy representing the prediction difficulty representing the level at which each sequentially input token is difficult to select from the previously input token, and the step of obtaining the indicator score information includes N tokens (w1, w2, …, w) obtained from the source code (W). N The token selection probability (P(w), which is the conditional probability for each i |w1, w2, … , w i-1 Based on )), the above log-likelihood (LL(W)) is mathematically Source code detection method calculated according to Claim 10 delete Claim 11 In claim 9, the source code detection method includes statistical information including the length of the source code, the average length of multiple variable names and the average length of multiple function names included in the source code. Claim 12 In claim 9, the above-mentioned determining step is a source code detection method performed using a machine learning neural network model pre-trained based on training data in which the author of the source code is a human or an artificial intelligence. Claim 13 In claim 9, the step of obtaining the indicator score information comprises N tokens (w1, w2, …, w) obtained from the source code (W). N The token selection probability (P(w), which is the conditional probability for each i |w1, w2, … , w i-1 Based on )), the above confusion level (PP(W) is mathematically A source code detection method calculated according to (where exp(·) represents an exponential function). Claim 14 delete Claim 15 In claim 1, the step of obtaining the indicator score information comprises N tokens (w1, w2, …, w) obtained from the source code (W). N Among ) previously entered tokens(w1, w2, …, w i-1 Multiple tokens that can be selected next based on ) (w i,1 , … , w i,M Token selection probability for each (P(w) i |w1, w2, … , w i-1 Check )) to sort sequentially, and multiple sequentially sorted tokens (w i,1 , … , w i,M Among ) currently entered token(w i By checking the rank of ), the above rank (Rank(W)) is mathematically Source code detection method calculated according to Claim 16 In claim 1, the step of acquiring the indicator score information comprises the previously input tokens (w1, w2, …, w) among a plurality of tokens selectable by the artificial intelligence. i-1 Based on ), the currently entered token (w i Other tokens besides ) (w j , here w j ≠ w i Token entropy (H(w) representing the probability of selecting ) i |w1, w2, … , w i-1 )) is a mathematical formula (where V is the size of the token set containing all tokens that the neural network model can select) calculated according to the calculated token entropy (H(w i |w1, w2, … , w i-1 Source code detection method for calculating the above entropy as the average of )).

Images