The invention discloses a malicious
software homology analysis method based on a behavior tree, provides a new definition for behaviors of malicious
software, is not limited to analysis of API continuous short sequences, describes behavior characteristics of the malicious
software from each behavior and a relationship between the behaviors, and is richer in behavior
semantics. The method comprisesthe following steps: firstly, calling a sequence from an API generated during malicious
software execution, and constructing the behavior tree for reflecting a malicious
software behavior model by adopting an Inductive Miner
algorithm; secondly, extracting behavior characteristics from each behavior tree, generating family weighted behavior characteristics, converting the
behavior trees into similarity vectors based on a similarity
algorithm, and finally, training a family classification model by applying a naive Bayes classification
algorithm. According to the method, the problems of lack ofa control structure and existence of
noise in the API sequence in previous malicious software
homology analysis based on the API sequence can be solved, and the malicious software family classification capability is improved.