Test method for software containing deep neural network

Abstract

The invention discloses a test method for software containing a deep neural network, which comprises the following steps of: firstly, analyzing software characteristics of the software containing the deep neural network, and designing a mutation operator according to the software characteristics; and then applying the mutation operator to software containing the deep neural network for mutation to generate a software variant. And then analyzing and judging whether the current test case set meets the test sufficiency requirement or not by using the software variant. If yes, the current test case set is directly used for conducting software testing on the software containing the deep neural network. And if not, generating a new test sample based on the metamorphic test technology and the generative adversarial network model to expand the test case set. And finally, performing software testing on the software containing the deep neural network by using the expanded test case set, and analyzing and judging whether the expanded test case set meets the test sufficiency requirement or not. If yes, proving that the software is fully tested, and ending the test. And if not, repeating the steps to continue to expand the test case set.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of software testing, and in particular to a testing method for software containing a deep neural network. BACKGROUND

[0002] Software testing is an important means to guarantee software quality in the process of software development and maintenance. Testers divide software testing into white-box testing and black-box testing according to whether the test cases are designed based on the source code. Among them, the software testing based on the source code is called white-box testing, and compared with other software testing methods, the white-box testing technology is easier to find software faults. Commonly used white-box testing methods include coverage testing technology, domain testing technology, symbolic testing technology, plug-in testing technology and mutation testing technology, etc. Black-box testing is to test software from the outside of software, and the main advantage is that the test cases conform to the expected use of the program or system, which is more natural and easy to understand. Commonly used black-box testing methods include equivalence class partitioning method, cause-effect diagram method, orthogonal experiment method, boundary value analysis method, decision table driving method, function testing, state testing, etc.

[0003] In the field of traditional software testing, it is often necessary to consider whether the designed test samples are sufficient and can meet the multiple requirements of software testing to ensure the sufficiency of testing. At present, testers generally use control flow or data flow analysis to define test sufficiency criteria, such as branch coverage, definition-use coverage, etc., and apply them to guide the subsequent test case design. At the same time, when designing test cases, mutation testing analysis technology is introduced to measure the test sufficiency from another angle, which can be used to evaluate the test sufficiency of the current test case set and expand the test case set.

[0004] In traditional software testing, the test result is generally determined by comparing the actual output and the expected output (Test Oracle). However, in the application scenario of artificial intelligence, the Oracle is often unknown, making it difficult to implement traditional software testing. In order to solve the problem of test Oracle, scholars first proposed the concept of metamorphic testing in 1998, that is, when the correct result of a certain test input cannot be determined, the domain knowledge of the software is analyzed to find a certain relationship that is satisfied by the results of multiple test inputs to test the software. That is, metamorphic testing can test the program by checking whether the relationship (i.e., metamorphic relationship) between the execution results of multiple test samples is satisfied.

[0005] In recent years, deep learning has made breakthrough progress in speech recognition, computer vision and other applications, and more and more technical personnel introduce deep learning technology when developing software to solve deeper problems. The concept of deep learning was first proposed by scholars at the University of Toronto in 2006, which refers to a machine learning process based on sample data to obtain a deep network structure containing multiple levels through a defined training method. The deep network structure obtained by deep learning contains a large number of single elements (neurons), each neuron is connected to a large number of other neurons, and the connection strength (weight) between neurons is modified during the learning process and determines the function of the network. The deep network structure obtained by deep learning meets the characteristics of neural networks, so the deep network is a deep neural network, that is, a deep neural network. Its purpose is to establish a model to simulate the neural connection structure of the human brain, and when processing images, sounds and texts, the data features are described through multiple transformation stages in layers, and then the data is explained. Therefore, how to test software containing deep neural networks becomes a focus of technical personnel, especially how to fully guarantee the software test containing deep neural networks, and how to generate new test samples to guarantee the sufficiency of the test when it is insufficient. SUMMARY

[0006] The following gives a brief summary of one or more aspects to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all contemplated aspects, and neither is it intended to identify key or critical elements of all aspects nor to delineate the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to the more detailed description given later.

[0007] The purpose of the present application is to solve the above problems, and a test method for software containing deep neural networks is provided. The traditional software test mutation analysis technology and the characteristics of deep neural networks are combined to mutate the neural network, so as to measure the test sufficiency of software containing deep neural networks. At the same time, the traditional mutation test technology and the technology of generative adversarial network are combined to constantly generate new test samples and expand the test case set, so as to guarantee the sufficiency of the test.

[0008] The technical scheme of the present application is:

[0009] The present application provides a test method for software containing deep neural networks, comprising the following steps:

[0010] Step S1: analyze the software features of software containing deep neural networks, and design a mutation operator according to the software features;

[0011] Step S2: applying a mutation operator to the deep neural network software to mutate the neural network, to generate a software mutant;

[0012] Step S3: using the software mutant generated in step S2 to analyze and determine whether the current test case set meets the test sufficiency requirement; if yes, directly using the current test case set to test the deep neural network software; if no, executing the next step S4;

[0013] Step S4: generating new test samples based on the transmutation testing technology and the generative adversarial network model, to expand the test case set;

[0014] Step S5: using the expanded test case set to test the deep neural network software, and using the software mutant generated in step S2 to analyze and determine whether the expanded test case set meets the test sufficiency requirement; if yes, proving that the software is sufficiently tested, and ending the testing of the deep neural network software; if no, repeating steps S2-S4 to continue expanding the test case set until the test sufficiency requirement is met.

[0015] According to an embodiment of the testing method for the deep neural network software, in step S1, the testing method for the deep neural network software generates a new mutation operator by using the redundant difference method when designing the mutation operator, including the following steps:

[0016] Step S11: designing two corresponding test case sets according to the two existing software mutants, and testing the deep neural network software using the designed test case sets to activate the neurons of the neural network in the software;

[0017] Step S12: comparing and analyzing the activation differences of the neurons in the neural network caused by different test case sets, and collecting a neuron set in which the neurons are affected differently by different test sets;

[0018] Step S13: mutating the collected neuron set to obtain a mutation operator.

[0019] According to an embodiment of the testing method for the deep neural network software, in step S13, the testing method for the deep neural network software mutates the neurons in the neuron set by modifying the offset, modifying the activation function, shielding the input dimension, increasing or decreasing a single neuron, and changing the connection weight, to generate a new mutation operator.

[0020] According to an embodiment of the test method for the deep neural network software, in step S3 or S5, when the test method for the deep neural network software analyzes and judges whether the test case set to be tested meets the test sufficiency requirement by using the software mutant generated in step S2, the test case is used to run the software before mutation and the software after mutation respectively, the running results are compared, and then the proportion of the software mutant killed by the test case set to be tested is calculated according to the comparison result, so as to obtain the mutation score of the test case set to be tested; wherein the test method for the deep neural network software uses the mutation score of the test case set to be tested to evaluate the test sufficiency requirement of the test case set to be tested.

[0021] According to an embodiment of the test method for the deep neural network software, when the test method for the deep neural network software uses the mutation score of the test case set to be tested to evaluate the test sufficiency requirement of the current test case set, a threshold value is set to evaluate whether the current test case set meets the test sufficiency requirement through the set threshold value; wherein,

[0022] If the mutation score of the current test case set reaches the set threshold value, it is proved that the current test case set meets the test sufficiency requirement;

[0023] If the mutation score of the current test case set does not reach the set threshold value, it is proved that the current test case set does not meet the test sufficiency requirement, and the test case set needs to be expanded.

[0024] According to an embodiment of the test method for the deep neural network software, in step S4, when the test method for the deep neural network software expands the test case set based on the metamorphic testing technology and the generative adversarial network model, first, the domain knowledge of the deep neural network software is analyzed to find the metamorphic relationship based on the domain knowledge, then the corresponding original test sample is obtained based on the metamorphic relationship, and finally the trained generative adversarial network model is used to generate a plurality of new test samples from the original test sample and supplement them to the test case set; wherein the original test sample includes sample parameters, test environment and program running result of the original test sample, and the new test sample is consistent with the sample parameters of the corresponding original test sample but different in test environment.

[0025] According to an embodiment of the test method for the deep neural network software, the metamorphic relationship is divided into equivalent metamorphic relationship, non-equivalent metamorphic relationship and time sequence metamorphic relationship, and when the test method for the deep neural network software determines the metamorphic relationship of the software, the corresponding metamorphic relationship is determined according to the type of the used neural network model and the corresponding domain knowledge.

[0026] According to an embodiment of the test method for the deep neural network software according to the present application, when a new problem occurs in the software test of the deep neural network software using the expanded test case set in step S5, first, return to steps S2 and S3 to evaluate the test sufficiency of the current test case set through steps S2 and S3; if the current test case set does not meet the test sufficiency requirement, continue to execute step S4 to expand the test case set until the software is fully tested; if the current test case set meets the test sufficiency requirement, it is proved that the software is fully tested, and the test of the deep neural network software is ended.

[0027] According to an embodiment of the test method for the deep neural network software according to the present application, when the test method for the deep neural network software changes the connection weight of the neurons in the neuron set in the form of changing the connection weight, the connection between the neurons is changed, the parameter value of the connection between the neurons is randomly exchanged, and the connection weight is changed in percentage, so as to realize the variation of the neurons in the neuron set.

[0028] The present application also provides a computer readable medium storing computer program codes, which, when executed by a processor, implement the method as described above.

[0029] The present application also provides a test device for the deep neural network software, comprising:

[0030] a memory for storing instructions executable by the processor; and

[0031] a processor for executing the instructions to implement the method as described above.

[0032] The present application has the following beneficial effects compared with the prior art: the test method for the deep neural network software according to the present application combines the traditional software test mutation analysis technology and the characteristics of the deep neural network, mutates the neural network, and analyzes the generated software mutant, so as to measure the test sufficiency of the software containing the deep neural network. Meanwhile, the traditional transmutation test technology and the technology of the adversarial generation network are combined to continuously generate new test samples and expand the test case set, so as to guarantee the sufficiency of the test. Through the present application, the test sufficiency of the software containing the deep neural network can be guaranteed, the robustness and reliability of the simulation software are improved, and the quality and efficiency of the simulation work are improved. BRIEF DESCRIPTION OF DRAWINGS

[0033] The above features and advantages of the present application will be better understood by reading the detailed description of the embodiments of the present application in conjunction with the drawings, in which: In the drawings, the components are not necessarily drawn to scale, and components of like structure or function can have like designations but different numbers.

[0034] Figure 1 is a flow chart illustrating an embodiment of the test method for deep neural network software according to the present application.

[0035] Figure 2 is a flow chart illustrating an embodiment of the mutation operator generation method according to the present application. DETAILED DESCRIPTION

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some examples or embodiments of the present application, and for those skilled in the art, the present application can also be applied to other similar scenarios without creative labor, unless it is obvious from the language environment or otherwise stated. In the drawings, the same reference numbers represent the same structures or operations, unless the context clearly indicates otherwise.

[0037] As shown in the present application and claims, unless the context clearly indicates otherwise, the words "one", "a", "an", and / or "the" do not mean to specify a single number, but can also include a plurality. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements explicitly identified, and these steps and elements do not constitute an exclusive list, and the method or device can also include other steps or elements.

[0038] Unless specifically stated otherwise, the relative arrangement of components and steps, numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present application. At the same time, it should be understood that the sizes of the various parts shown in the drawings are not drawn in proportion to the actual proportions. The technology, methods and devices known to those skilled in the relevant art can not be discussed in detail, but in appropriate cases, the technology, methods and devices should be considered as part of the authorized description. In all examples shown and discussed here, any specific value should be interpreted as merely exemplary, and not as a limitation. Therefore, other examples of exemplary embodiments can have different values. It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0039] In the detailed description of the embodiments of the application, the cross-sectional view of the device structure is partially enlarged without the general proportion for the convenience of illustration, and the schematic view is only an example, which should not limit the scope of protection of the application herein. In addition, three-dimensional spatial dimensions including length, width and depth should be included in actual production.

[0040] In the description of the present application, it should be understood that the orientation words such as "front, back, up, down, left, right", "transverse, vertical, perpendicular, horizontal" and "top, bottom" and the like indicate the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, without the opposite description, these orientation words do not indicate and imply that the indicated device or element must have a specific orientation or be constructed and operated in a specific orientation, therefore it cannot be understood as a limitation on the scope of protection of the present application; the orientation words "inner, outer" refer to the inner and outer of the contour of each component itself.

[0041] An embodiment of a testing method for software containing a deep neural network is disclosed herein, Figure 1 is a flow chart showing an embodiment of the testing method for software containing a deep neural network of the present application. Please refer to Figure 1 The following is a detailed description of each step of the testing method for software containing a deep neural network.

[0042] Step S1: analyze the software features of software containing a deep neural network, and design mutation operators according to the software features.

[0043] In the traditional software testing mutation analysis process, the tester generally designs mutation operators according to the characteristics of the program under test first, and the mutation operators generally make only minor changes to the program under test under the premise of meeting the syntax. Then, a large number of mutants can be generated by applying mutation operators to the program under test, and after identifying equivalent mutants, if the existing test cases cannot eliminate all non-equivalent mutants, new test cases need to be designed and added to the test case set to improve the test sufficiency. In addition to being used for sufficiency evaluation of test case set, mutation testing can also simulate real defects of the software under test by using mutation defects, so as to assist in evaluating the effectiveness of the test method proposed by the researchers. Therefore, in this embodiment, when applying the traditional software testing mutation analysis technology to the testing of software containing a deep neural network, first, the characteristics of the deep neural network are analyzed to analyze the software features of the software containing the deep neural network, and then the mutation operators are designed according to the software features.

[0044] In an embodiment, a redundant difference method is used to generate new mutation operators. Figure 2 is a flow chart showing an embodiment of the mutation operator generation method of the present application, and the following will be described in detail in combination with Figure 2 , the detailed description of this embodiment is as follows:

[0045] Step S11: Two corresponding test case sets are designed according to the two existing software variants, and the software containing the deep neural network is tested using the designed test case sets to activate the neurons of the neural network in the software.

[0046] Step S12: The activation differences of the neurons in the neural network caused by different test case sets are compared and analyzed, and a neuron set that is affected differently by different test sets is collected.

[0047] When generating a new mutation operator by using the redundant difference method, generally, the test case sets of two existing variants are used to obtain a new mutation operator. By comparing the activation differences of the neurons in the neural network caused by different test samples in the corresponding test case sets of the two variants, a neuron set that is affected differently by different test case sets is found. Such a neuron set is generally associated with domain knowledge, and deep neural networks applied in different fields will produce different affected neuron sets. Therefore, mutating the neuron set can effectively simulate the possible potential defects of the deep neural network in a specific field.

[0048] Step S13: The collected neuron set is mutated to obtain a mutation operator.

[0049] In this step, the neurons in the neuron set can be mutated by modifying the offset, modifying the activation function, masking the input dimension, adding or deleting a single neuron, and changing the connection weight, so as to generate a new mutation operator. When the connection weight is used to mutate the neurons in the neuron set, the connection weight can be changed by connecting between part of the neurons, randomly exchanging the parameter values of part of the neuron connections, and increasing or decreasing the connection weight by percentage, so as to realize the mutation of the neurons in the neuron set.

[0050] Step S2: Apply the mutation operator to the software containing the deep neural network to mutate the neural network and generate a software variant.

[0051] Step S3: Use the software variant generated in step S2 to analyze and determine whether the current test case set meets the test sufficiency requirement; if yes, directly use the current test case set to test the software containing the deep neural network; if not, execute the next step S4.

[0052] In this embodiment, after generating the new software mutant through step S2, the test cases in the current test case set are used to run the software before mutation and the software after mutation respectively, and the running results are compared, and then the proportion of the current test case set killing the software mutant is calculated according to the comparison result, so as to obtain the mutation score of the current test case set, which is used to evaluate the test sufficiency requirement of the current test case set.

[0053] In the evaluation of the test sufficiency requirement of the current test case set by using the mutation score of the current test case set, a threshold (for example, 90%) is set to evaluate whether the current test case set meets the test sufficiency requirement by using the set threshold. If the mutation score of the current test case set reaches the set threshold (for example, reaches 90%), it is proved that the current test case set meets the test sufficiency requirement. If the mutation score of the current test case set does not reach the set threshold (for example, does not reach 90%), it is proved that the current test case set does not meet the test sufficiency requirement, and the subsequent step of expanding the test case set needs to be performed.

[0054] Step S4: generating new test samples based on the transmutation test technology and the adversarial generative network model to expand the test case set.

[0055] In this embodiment, in the analysis of the domain knowledge of the software containing the deep neural network, the transmutation relationship based on the domain knowledge is first found, then the corresponding original test sample is obtained based on the transmutation relationship, and finally a plurality of new test samples are generated from the original test sample by using the trained adversarial generative network model. The sample parameters, test environment and program running result of the original test sample are consistent with the sample parameters of the corresponding original test sample, and the test environment is different.

[0056] The transmutation test technology refers to that when the correctness of the test input cannot be determined, the domain knowledge of the software is analyzed, a certain determined relationship (i.e. transmutation relationship) satisfied by the running results of a plurality of test inputs is found, and the software is tested. The transmutation test checks whether the relationship (i.e. transmutation relationship) between the execution results of a plurality of test samples is satisfied to test the program. The transmutation relationship is divided into equivalent transmutation relationship, non-equivalent transmutation relationship and time sequence transmutation relationship. When the transmutation relationship of the software is determined, the corresponding transmutation relationship can be determined according to the type of the used neural network model and the corresponding domain knowledge.

[0057] For example, in the case of autonomous driving software testing, different road conditions images (i.e. test samples) acquired by sensors need to be tested. After obtaining the metamorphic relations from domain knowledge (for example, the same road conditions, the same reaction of the autonomous driving car in different weather backgrounds), the original image input (i.e. the original test sample) is obtained, including image parameters, background (i.e. test environment) and feedback of the autonomous driving system to the image. If the background in the image is sunny, the trained GAN model is used to automatically add rain or cloudy background effects to the original image, thereby generating an image with the same content but different background styles as a new test sample. Finally, it is necessary to analyze whether the feedback of the autonomous driving system to the generated new test sample is consistent with the feedback of the corresponding original image. If so, the generated new test sample is expanded to the test case set.

[0058] Step S5: using the expanded test case set to test the software containing the deep neural network, and using the software mutant generated in step S2 to analyze and determine whether the expanded test case set meets the test sufficiency requirement; if so, it is proved that the software is fully tested, and the test of the software containing the deep neural network is ended; if not, steps S2-S4 are repeated to continue expanding the test case set until the test sufficiency is met.

[0059] In this embodiment, after using the expanded test case set to test the software containing the deep neural network, it is also necessary to use the software mutant generated in step S2 to analyze and determine whether the expanded test case set meets the test sufficiency requirement. The specific analysis method is consistent with the method of using the software mutant generated in step S2 to analyze and determine whether the current test case set meets the test sufficiency requirement in step S3 described above, which will not be repeated here. If the expanded test case set meets the test sufficiency requirement, it is proved that the software is fully tested, and the test of the software containing the deep neural network is directly ended at this time, and the newly appeared problems are improved. If the expanded test case set does not meet the test sufficiency requirement, steps S2-S4 are repeated to expand the test case set until the software is fully tested.

[0060] The present specification also provides a computer readable medium storing computer program code, which, when executed by a processor, implements the test method for software containing a deep neural network as described above.

[0061] The present specification also provides a test device for software containing a deep neural network, comprising a memory storing instructions executable by a processor, and a processor for executing the instructions in the instruction memory to implement the test method for software containing a deep neural network as described above.

[0062] The previous description of the disclosure is provided to enable any persons skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0063] Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality, without reference to a specific

[0064] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein can be implemented or performed with a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor can be a microprocessor, but in the alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0065] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium can reside as discrete components in a user terminal.

[0066] In one or more exemplary embodiments, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. Storage media can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

Claims

1. A testing method for software containing deep neural networks, characterized in that, Includes the following steps: Step S1: Analyze the software characteristics of software containing deep neural networks, and design mutation operators based on the software characteristics; Step S2: Apply the mutation operator to the software containing the deep neural network to mutate the neural network and generate a software mutant; Step S3: Analyze the software variants generated in step S2 to determine whether the current test case set meets the test sufficiency requirements; if yes, directly use the current test case set to perform software testing on the software containing deep neural networks; if no, proceed to the next step S4. Step S4: Generate new test samples based on metamorphic testing technology and adversarial generative network model to expand the test case set; Step S5: Use the expanded test case set to perform software testing on the software containing deep neural networks, and use the software variant generated in step S2 to analyze and determine whether the expanded test case set meets the test adequacy requirements; if yes, it proves that the software has been adequately tested, and the testing of the software containing deep neural networks ends; if not, repeat steps S2 to S4 to continue expanding the test case set until the test adequacy is met.

2. The testing method for software containing deep neural networks according to claim 1, characterized in that, In step S1, the testing method for software containing deep neural networks uses a redundant difference method to generate new mutation operators when designing mutation operators, including the following steps: Step S11: Design two corresponding test case sets based on the two existing software variants, and use the designed test case sets to test the software containing deep neural networks to activate the neurons of the neural network in the software; Step S12: Compare and analyze the differences in activation of neurons in the neural network caused by different test case sets, and collect sets of neurons whose effects on neurons differ depending on the test set. Step S13: Mutate the collected set of neurons to obtain the mutation operator.

3. The testing method for software containing deep neural networks according to claim 2, characterized in that, In step S13, the testing method for software containing deep neural networks uses methods such as modifying offsets, modifying activation functions, masking input dimensions, adding or removing individual neurons, and changing connection weights to mutate neurons in the neuron set, thereby generating new mutation operators.

4. The testing method for software containing deep neural networks according to claim 1, characterized in that, In step S3 or S5, when the testing method for software containing deep neural networks uses the software variants generated in step S2 to analyze and determine whether the test case set under test meets the test sufficiency requirements, the test cases are used to run the software before and after mutation, and the running results are compared. Then, based on the comparison results, the proportion of software variants killed by the test case set under test is calculated, thereby obtaining the mutation score of the test case set under test. The testing method for software containing deep neural networks uses the mutation score of the test case set under test to evaluate the test sufficiency requirements of the test case set under test.

5. The testing method for software containing deep neural networks according to claim 4, characterized in that, The testing method for software containing deep neural networks uses the variation score of the tested test case set to evaluate the test sufficiency requirements of the current test case set. A threshold is set to evaluate whether the current test case set meets the test sufficiency requirements; where, If the variation score of the current test case set reaches the set threshold, it proves that the current test case set meets the test sufficiency requirements. If the variation score of the current test case set does not reach the set threshold, it proves that the current test case set does not meet the test sufficiency requirements and needs to be expanded.

6. The testing method for software containing deep neural networks according to claim 1, characterized in that, In step S4, when the testing method for software containing deep neural networks expands the test case set based on metamorphic testing technology and adversarial generative network model, it first analyzes the domain knowledge of the software containing deep neural networks, finds the metamorphic relationship based on the domain knowledge, then obtains the corresponding original test samples based on the metamorphic relationship, and finally uses the trained adversarial generative network model to generate multiple new test samples based on the original test samples and supplements them to the test case set. Among them, the original test samples include the sample parameters, test environment and program running results of the original test samples, and the new test samples have the same sample parameters as the corresponding original test samples but different test environments.

7. The testing method for software containing deep neural networks according to claim 5, characterized in that, Metamorphic relationships are categorized into equivalent metamorphic relationships, non-equivalent metamorphic relationships, and temporal metamorphic relationships. The testing method for software containing deep neural networks determines the corresponding metamorphic relationship based on the type of neural network model used and the corresponding domain knowledge when determining the software's metamorphic relationship.

8. The testing method for software containing deep neural networks according to claim 1, characterized in that, In step S5, when new problems arise during software testing of software containing deep neural networks using the expanded test case set, the process first returns to steps S2 and S3 to evaluate the sufficiency of the current test case set. If the current test case set does not meet the sufficiency requirements, step S4 is executed to expand the test case set until the software is fully tested. If the current test case set meets the test sufficiency requirements, then the software has been adequately tested, and the testing of the software containing deep neural networks is complete.

9. The testing method for software containing deep neural networks according to claim 3, characterized in that, The testing method for software containing deep neural networks mutates neurons in a neuron set by changing connection weights. This is achieved by altering the connection weights between some neurons, randomly swapping the parameter values ​​of some neuron connections, and increasing or decreasing the connection weights by a percentage.

10. A computer-readable medium storing computer program code, characterized in that, The computer program code, when executed by a processor, implements the method as described in any one of claims 1-9.

11. A testing device for software containing deep neural networks, characterized in that, include: Memory is used to store instructions that can be executed by the processor; as well as A processor for executing the instructions to implement the method as described in any one of claims 1-9.

Images