Segmented linear neural network robustness verification method and device

Abstract

The application relates to a segmented linear neural network robustness verification method and device. The method comprises the following steps: receiving a robustness verification task of a neural network model to be verified; constructing a mixed integer constraint problem description based on the robustness verification task; performing adaptive bit width fixed point quantization on continuous variables of the mixed integer constraint problem description, generating an Ising model and extracting corresponding coupling parameters; loading the coupling parameters, performing Ising energy minimization solving on the Ising model, and obtaining corresponding spin values according to a solving result; and performing inverse transformation on the spin values to obtain a robustness verification result of the neural network model. The scheme solves the problems of high computational complexity, low verification efficiency and high energy consumption of the robustness verification of the segmented linear neural network.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security technology, and in particular to a method and apparatus for verifying the robustness of a piecewise linear neural network. Background Technology

[0002] Deep neural networks (DNNs) are widely used in fields with high security requirements, such as computer vision, natural language processing, autonomous driving, and medical diagnosis. Research shows that DNNs are highly sensitive to input perturbations; even small perturbations in the input data can lead to completely incorrect model outputs, posing a significant security threat to high-risk applications. Therefore, robustness verification of neural networks—proving that their outputs are stable within a given perturbation range, or finding specific counterexamples that lead to misclassification—has become a core requirement for artificial intelligence security.

[0003] In related technologies, robustness verification of neural networks is mainly achieved through the following three methods: The first is an exact verification method based on Satisfiability Modulus Theory (SMT). This method, based on SMT / satisfiability solution verification, represents the neural network as a combination of linear constraints and activation functions, and uses branch and bound / conflict learning to search for feasible counterexamples. The second is an optimization verification method based on Mixed Integer Programming (MIP). This method formalizes the verification problem into a MIP problem and uses a general solver for exact solution. The third is an approximate verification method based on abstract interpretation / relaxation bounds (interval propagation, linear relaxation, dual bounds, etc.), which replaces exact feasibility search with relaxation / upper bounds to improve verification speed. However, when using the above methods to robustly verify neural networks, the time complexity increases exponentially with the number of neurons and segments when solving combinatorial optimization problems with a large number of discrete variables. This results in a certain verification delay for scenarios such as online verification of autonomous driving and real-time compliance review of financial models. Furthermore, the energy consumption and cost are high, with general-purpose CPUs / GPUs incurring significant energy consumption and memory access overhead for branch and bound and MIP problem solving. Acceleration also relies heavily on stacked computing power, leading to high costs. Finally, scalability is insufficient: classical solvers are prone to severe degradation when dealing with diverse network structures, complex activation functions, and increased perturbation domain dimensions.

[0004] In summary, the robustness verification of piecewise linear neural networks in related technologies suffers from high computational complexity, low verification efficiency, and high energy consumption.

[0005] There is currently no effective solution to the problems of high computational complexity, low verification efficiency and high energy consumption in the robustness verification of piecewise linear neural networks in related technologies. Summary of the Invention

[0006] The robustness verification method and apparatus for piecewise linear neural networks provided by the embodiments of the present invention at least solve the problems of high computational complexity, low verification efficiency and high energy consumption in robustness verification of piecewise linear neural networks.

[0007] The embodiments of this invention provide a method for robustness verification of a piecewise linear neural network, comprising: Receive robustness verification tasks for neural network models to be verified; A hybrid integer constraint problem description is constructed based on the robustness verification task described above; Adaptive bit-width fixed-point quantization is performed on the continuous variables described in the mixed integer constraint problem to generate the Ising model and extract the corresponding coupling parameters. Load the coupling parameters and solve the Ising energy minimization problem of the Ising model, and obtain the corresponding spin value based on the solution result; The spin values ​​are inversely transformed to obtain the robustness verification results of the neural network model.

[0008] In some embodiments, a hybrid integer constraint problem description is constructed based on the robustness verification task, including: Based on the robustness verification task, robustness verification constraints of the neural network model are extracted. These robustness verification constraints include network computation constraints and perturbation domain constraints. The network computation constraints and perturbation domain constraints are converted into a set of linear inequalities; Selection and auxiliary variables are introduced into the piecewise linear activation function of the neural network model; A standardized description of the mixed integer constraint problem is constructed based on the set of linear inequalities, the selection variables, and the auxiliary variables.

[0009] In some embodiments, adaptive bit-width fixed-point quantization is performed on the continuous variables described by the mixed integer constraint problem to generate the Ising model and extract the corresponding coupling parameters, including: The continuous variables in the description of the mixed integer constraint problem are estimated by the fixed-point interval boundary propagation algorithm to obtain the quantization range, and the quantization range is subjected to adaptive bit-width fixed-point quantization processing. The quantized continuous variables are then encoded based on multiple binary variables. The inequality constraints in the description of the mixed integer constraint problem are transformed into an equality structure by using non-negative relaxation variables. Construct a unified binary variable vector, which includes all original binary variables, quantized bits of continuous variables, and relaxation bits corresponding to non-negative relaxation variables; Based on the robustness verification task, the initial optimization objective and constraint violation penalty term of the neural network model are determined, and a quadratic unconstrained binary optimization model is constructed based on the initial optimization objective and the constraint violation penalty term. The quadratic unconstrained binary optimization model is transformed into the Ising Hamiltonian form using matrix transformation, resulting in the Ising model. The weight matrix and bias vector of the Ising model are optimized to obtain the corresponding coupling parameters.

[0010] In some embodiments, the adaptive bit width is determined by the number of effective segment intervals of the linear segmented activation function of the neural network model.

[0011] In some embodiments, the Ising model is solved by minimizing the Ising energy, and the corresponding spin value is obtained based on the solution result, including: The Ising model and the corresponding coupling parameters are loaded into the preset Ising solver; The Ising solver performs a physical evolution of the Ising model based on the coupling parameters, thereby achieving the solution of minimizing the Ising energy of the Ising model; Phase detection is performed on the output pulse of the Ising solver to obtain the spin value.

[0012] In some embodiments, performing an inverse transformation on the spin values ​​to obtain the robustness verification results of the neural network model includes: The spin values ​​are inversely transformed into candidate values ​​of the original binary variable and the quantized continuous variable; The input perturbation is reconstructed based on the original binary variables and the candidate values ​​of the continuous variables, and forward propagation is performed to verify whether the neural network model has classification flipping. If classification flip exists, output the corresponding adversarial example and return a verification failure signal; If no category flipping occurs, a verification pass signal is returned when the stopping condition is met.

[0013] An embodiment of the present invention also provides a piecewise linear neural network robustness verification apparatus for performing the above-described piecewise linear neural network robustness verification method, comprising: The host interface module is used to receive robustness verification tasks for the neural network model to be verified. A constraint construction module is used to construct a hybrid integer constraint problem description based on the robustness verification task; The Ising mapping module is used to perform adaptive bit-width fixed-point quantization on the continuous variables described by the mixed integer constraint problem, generate the Ising model, and extract the corresponding coupling parameters. The Ising solver module is used to load the coupling parameters and perform Ising energy minimization on the Ising model, and obtain the corresponding spin value based on the solution result; The result decoding module is used to perform an inverse transformation on the spin values ​​to obtain the robustness verification results of the neural network model.

[0014] In some embodiments, the Ising solver module performs Ising energy minimization on the Ising model using a coherent Ising machine with a built-in optical parametric oscillator.

[0015] In some embodiments, the device employs a heterogeneous integrated architecture, where the constraint construction module and the Ising mapping module are integrated on the same FPGA chip, the optical parametric oscillator array of the Ising solving module is opto-interconnected to the FPGA chip via an optical fiber array, and the phase detector of the result decoding module is electrically connected to the FPGA chip via a high-speed ADC.

[0016] In some embodiments, the FPGA chip includes: The interval arithmetic hard core unit is used to perform fixed-point interval boundary propagation operations; The Big M linearization circuit is used to generate auxiliary variables and constraint coefficients; The coupling parameter generation unit is used to convert the description of a mixed integer constraint problem into a coupling matrix and a bias vector.

[0017] In some embodiments, the constraint construction module includes: The boundary propagation submodule is used to generate the upper and lower bounds of the pre-activation values ​​of each neuron in the neural network model using a fixed-point interval boundary propagation algorithm. The segmented selection encoding submodule is used to prune infeasible segments based on the upper and lower bounds of the pre-activation value, and generate binary variables corresponding to the effective segmented intervals of each neuron in the neural network model. The auxiliary variable generation submodule is used to convert the product of the binary variable and the continuous variable into the auxiliary variables and constraint coefficients required for equivalent linear constraints using a hardware parallel architecture.

[0018] In some embodiments, the Ising solver module includes: An optical parametric oscillator array includes multiple periodically polarized lithium niobate optical parametric oscillator ring cavities, the physical cavity length of each optical parametric oscillator ring cavity being a preset length, and the optical parametric oscillator array being used to generate multiple coherent pulses, the pulse phase and amplitude of which correspond to the Ising spin state. An optocoupler network includes a photodetector, an analog-to-digital converter, a digital signal processor, a digital-to-analog converter, and an electro-optic modulator. The optocoupler network is used for programmable coupling of the coupling parameter set of the Ising model. The pump control circuit includes a tunable laser source and a power control circuit. The pump control circuit is used to perform hardware-level adjustment of the pump power, cavity loss and injection noise of the optical parametric oscillator array to achieve physical annealing evolution. An environmental stabilization unit, including a temperature control circuit and an active vibration isolation platform, is used for optical path and phase locking, temperature control, and vibration suppression.

[0019] In some embodiments, the apparatus further includes a model storage module, the model storage module comprising: A weight memory, wherein the weight memory is used to store the weight matrix of the neural network model; Bias memory, the bias memory being used to store the bias vector of the neural network model; Piecewise linear parameter memory, wherein the piecewise linear parameter memory is used to store the interval parameters of the piecewise linear activation function of the neural network model.

[0020] In some embodiments, the apparatus further includes a control and scheduling module, which is connected to the host interface module, the constraint construction module, the Ising mapping module, the Ising solving module, and the result decoding module, respectively. The control and scheduling module is used to control the neural network model to perform a robustness verification task.

[0021] The robustness verification method and apparatus for piecewise linear neural networks provided by the embodiments of this invention constructs a hybrid integer constraint problem description through a robustness verification task based on a neural network model. The continuous variables in the hybrid integer constraint problem description are adaptively bit-width fixed-point variables to generate an Ising model and extract coupling parameters. Based on these coupling parameters, the Ising model is solved for Ising energy minimization, and spin values ​​are output. The robustness verification result of the piecewise linear neural network is obtained by inverse transformation of the spin values. This solves the problems of high computational complexity, low verification efficiency, and high energy consumption in robustness verification of piecewise linear neural networks. By mapping the robustness verification constraint problem of the piecewise neural network to an Ising model and using physical parallel evolution to achieve Ising energy minimization, it can replace the traditional branch-and-bound and integer programming solutions of CPU / GPU, significantly reducing computational complexity and energy consumption, and significantly improving the efficiency and accuracy of robustness verification. It can adapt to the verification requirements of high-security scenarios. By transforming the nonlinear constraints of piecewise linear activation functions into linear constraints, the complex robustness verification problem can be converted into a quantifiable and solvable form, reducing the computational complexity of the solution. Adaptive bit-width fixed-point quantization of continuous variables reduces computational load and energy consumption while maintaining verification accuracy. Simultaneously, the mixed-integer constraint problem is transformed into an Ising model, and coupling parameters are extracted, achieving a fit between the constraint problem and the Ising solution. Loading the coupling parameters and achieving Ising energy minimization through physical evolution significantly improves the solution speed and reduces energy consumption by leveraging parallel solution characteristics. Energy convergence ensures solution accuracy and avoids getting trapped in local optima. Attached Figure Description

[0022] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other embodiments based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a method for verifying the robustness of a piecewise linear neural network, as described in an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram of the structure of a piecewise linear neural network robustness verification device according to an embodiment of the present invention. Detailed Implementation

[0025] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0026] To address the problems of high computational complexity, low verification efficiency, and high energy consumption in robustness verification of piecewise linear neural networks in related technologies, the embodiments of this invention provide a method and apparatus for robustness verification of piecewise linear neural networks.

[0027] like Figure 1 As shown, an embodiment of the present invention provides a method for verifying the robustness of a piecewise linear neural network, comprising: S101 receives the robustness verification task of the neural network model to be verified.

[0028] In the embodiments of this invention, the neural network model to be verified is a piecewise linear neural network model, which is a feedforward neural network where the activation functions of neurons in each layer are piecewise linear. The network structure of the piecewise linear neural network model adopts a multi-layer front-end structure, specifically including an input layer, hidden layers, and an output layer. The hidden layers have multiple layers and use piecewise linear activation functions. These piecewise linear activation functions are represented as activation functions composed of a finite number of linear segments, such as ReLU (two segments) and Hardtanh (three segments). Each segment of the piecewise linear activation function is defined by a slope. ,intercept and interval boundaries By definition, a piecewise linear activation function can convert the input pre-activation value Divide the input into multiple consecutive intervals, and define the pre-activation values ​​of the output and input within each interval. The relationship is linear. The pre-activation value is... , which indicates the first The linear combination output of layer neurons, through The calculation shows that, among which Indicates the first The weight matrix of the layer, Indicates the first The layer's bias vector, Indicates the first The output value (vector) after layer activation, i.e., the first layer... The input of the layer is from the first layer preactivation value The piecewise linear activation function is used to obtain the model. In the embodiments of this invention, the robustness of the above-mentioned piecewise linear neural network model is verified. Within a preset perturbation range, perturbation samples that can cause the output of the piecewise linear neural network model to be classified inverted are found, that is, the classification gap between the true class and the adversarial class is minimized, so that the piecewise linear neural network model makes as many classification errors as possible, in order to verify whether the piecewise linear neural network model has robustness.

[0029] In some embodiments, the robustness verification task of the piecewise linear neural network model includes the model parameters, the parameters of the piecewise linear activation function, and the input samples. ( (Indicates the dimension of the input sample), perturbation range (perturbation radius) The verification task description and robustness verification constraints, and the model parameters of the linear neural network model include the number of layers. Number of neurons per layer Weight matrix Bias The parameters of a piecewise linear activation function include the number of segments. Slope of each segment ,intercept and interval boundaries Input Sample That is, the input vector This represents the raw data to be processed (e.g., image pixels), limited by nominal points. Centered on, with radius of -norm perturbation sphere, i.e., satisfying .

[0030] S102, Construct a hybrid integer constraint problem description based on a robustness verification task.

[0031] In some embodiments, before constructing the description of the mixed integer constraint problem, it is necessary to first unify and integrate all the variables required for robustness verification to obtain the complete state vector. This provides a unified variable basis and a complete state vector for subsequent constraint construction and Ising mapping. It is a unified optimization variable composed of the inputs, activation values, pre-activation values, and auxiliary variables of all layers. This vector includes all information from the original input to the intermediate variables of each layer and finally to the output, forming the basic data structure for hardware formal verification and robustness analysis, and is a complete state vector. The mathematical expression is: in: This represents the total number of layers in the neural network model; The neural network model represents the first... The activation vector of the layer; The neural network model represents the first... The pre-activation value vector of the layer; The neural network model represents the first... Auxiliary variable vector of the layer.

[0032] In some embodiments, a hybrid integer constraint problem description is constructed based on a robustness verification task, including: Robustness verification constraints for extracting neural network models based on robustness verification tasks include network computation constraints and perturbation domain constraints. Transform network computation constraints and perturbation domain constraints into a set of linear inequalities; Introduce selection and auxiliary variables for the piecewise linear activation function of the neural network model; A standardized description of mixed-integer constraint problems is constructed based on a set of linear inequalities, selection variables, and auxiliary variables.

[0033] In the embodiments of this invention, based on the upper and lower bounds of the pre-activation value For each neuron, segmented intervals are selected to generate binary variables. This embodiment uses the ReLU activation function as an example. If a neuron's preactivation boundary... (Right now ), then force , If the intersection is ( If so, then retain two binary variables, satisfying... (Only one segment is activated).

[0034] In the embodiments of this invention, binary variables are used... With continuous variables product term Auxiliary variables required to transform into equivalent linear constraints The auxiliary variables are generated using the Big M linearization circuit, which incorporates multipliers and adders to support real-time generation. It can simultaneously generate four sets of linear constraints, the mathematical expressions of which are as follows: in, , These represent the upper and lower bounds of the large M value, respectively. They are configurable, ranging from 1 to 100, and adaptively adjust based on the pre-activation bound. , ), auxiliary variables The precision is 32-bit fixed-point.

[0035] In the embodiments of this invention, if the neural network model is based on the original input samples... For local robustness, it is required that... Centered on, the disturbance radius is of Within the perturbation sphere, any input that satisfies the perturbation constraints The predicted categories remain unchanged, that is: in: express of Disturb the neighborhood; express The corresponding real category; Indicates input The corresponding number The score for each output category.

[0036] Correspondingly, the "breach of robustness" of a neural network model is equivalent to the existence of some adversarial example. This makes a certain error category The output score is greater than or equal to the true category. Score, i.e. ,satisfy: There exists a certain Based on the above equivalence relation, the robustness verification problem is transformed into an optimization problem: minimizing the difference between the true class score and the candidate incorrect class score, with the objective function expressed as: in and These are the one-hot vectors representing the true class and the false class, respectively. For input The corresponding output layer (the first) The activation vector or output vector of a layer.

[0037] If the optimal value is less than or equal to 0, it means that there exists an input that satisfies the perturbation constraint, such that the score of the wrong class is not less than the score of the true class, thus violating robustness.

[0038] S103 performs adaptive bit-width fixed-point quantization on the continuous variables described by the mixed integer constraint problem, generates the Ising model, and extracts the corresponding coupling parameters.

[0039] In some embodiments, adaptive bit-width fixed-point quantization is performed on the continuous variables described by the mixed integer constraint problem to generate the Ising model and extract the corresponding coupling parameters, including: The continuous variables in the mixed integer constraint problem description are estimated using the fixed-point interval boundary propagation algorithm to obtain the quantization range. The quantization range is then subjected to adaptive bit-width fixed-point quantization. The quantized continuous variables are then encoded based on multiple binary variables. Inequality constraints in the description of mixed integer constraint problems are transformed into equality structures by using non-negative relaxation variables. Construct a unified binary variable vector, which includes all original binary variables, quantized bits of continuous variables, and relaxation bits corresponding to non-negative relaxation variables; The initial optimization objective and constraint violation penalty term of the neural network model are determined based on the robustness verification task, and a quadratic unconstrained binary optimization model is constructed based on the initial optimization objective and constraint violation penalty term. The quadratic unconstrained binary optimization model is transformed into the Ising Hamiltonian form using matrix transformation, resulting in the Ising model. The weight matrix and bias vector of the Ising model are optimized to obtain the corresponding coupling parameters.

[0040] In the embodiments of this invention, during optimization, all variables are guaranteed to satisfy mixed integer constraints such as robustness verification constraints and piecewise linear activation constraints. By penalizing the violation of constraints, the final solution meets all the feasibility requirements of the verification problem.

[0041] In some embodiments, continuous variables in the description of mixed-integer constrained problems (such as...) x , Adaptive bit-width fixed-point quantization encoding is performed, which converts continuous variables into a binary bit stream based on the number of effective segment intervals, while preserving the precision characteristics of the variables (using 32-bit fixed-point quantization). This makes it suitable for subsequent binary variable optimization needs, solving the problem that continuous variables cannot be directly input into a coherent Ising machine for solving, and ensuring that the quantized continuous variables can be used with binary selected variables. Collaborative participation in subsequent model construction. Furthermore, based on quantized bitstreams and standardized mixed-integer constraints (a set of linear inequalities), all linear constraints are converted into equivalent penalty terms, thus transforming the linear inequality constraints... Through nonnegative relaxation variables Transform into equality constraints and will constrain the amount of violation The constraint form is added as a penalty term to the objective function; a reasonable penalty logic is set for constraint violations to ensure that the constraints can be integrated into the optimization objective when constructing the subsequent quadratic unconstrained binary optimization model (QUBO). Simultaneously, the constraint form is simplified to adapt to the quadratic and linear term structures of the quadratic unconstrained binary optimization model, avoiding incompatibility between constraints and model format. A unified binary variable vector is constructed using multiple binary variable encodings (e.g., 8 bits corresponding to 8 bits of precision), containing all original binary variables, quantization bits, and relaxation variable bits. Then, the inequality constraints in the mixed integer constraint problem description are transformed into equality by introducing non-negative relaxation variables. The initial optimization objective (minimizing the classification margin) for neural network robustness verification is merged with the constraint violation penalty term to form a quadratic unconstrained binary optimization model. The resulting QUBO objective function is: in, Represents a vector of binary variables; Represents the coefficient matrix of the quadratic term; This represents the coefficient vector of the first-order term; for the single-output case... The initial optimization objective is to minimize ( This is the true class vector, used to determine whether the classification remains stable under a given perturbation. (For adversarial category vectors); constraint violation penalty term is (Constraint violation amount) In this embodiment, the penalty coefficient is... Take 100.

[0042] Furthermore, in the embodiments of this invention, the quadratic unconstrained binary optimization model is transformed into the Ising Hamiltonian form through matrix transformation to obtain the Ising model, and the coupling parameters in the Ising model, including the coupling matrix, are output. and bias vector The mathematical expression of the Ising model is as follows: in: Represents the coupling matrix; Represents the bias vector; and Represents the spin variable. .

[0043] In some embodiments, the adaptive bit width is determined by the number of effective segment intervals of the linear segmented activation function of the neural network model.

[0044] In some embodiments, after constructing the Ising model, it is further optimized based on the hardware parallelism of the coherent Ising machine to generate coupling parameters suitable for loading the coherent Ising machine. Hardware parallelism represents the number of spin values ​​that the coherent Ising machine can compute sequentially. Optionally, the optimization process includes at least one of sparsification, block partitioning, and fixed-point quantization. The sparsity optimization uses a threshold filtering method on the coupling matrix. Perform sparsification and traverse the coupling matrix. All elements are processed, and those with absolute values ​​less than a set threshold are removed (these elements have negligible impact on the Ising energy minimization solution), retaining only valid elements with absolute values ​​greater than or equal to the set threshold. Simultaneously, the position coordinates and values ​​of valid elements are recorded, forming a sparse matrix format (only valid elements and their corresponding positions are stored, invalid zero elements are not). This sparsification optimization process significantly reduces the amount of coupling parameters, decreasing the time required for subsequent loading into the coherent Ising machine. It also reduces the computational burden on the Ising solver, avoids invalid computations consuming hardware resources, and improves the efficiency of robustness verification. Block processing is applied to the sparsified coupling matrix. The processing is performed in uniform blocks according to the hardware solver size of the Coherent Ising Machine. In the embodiments of this invention, the parallel computing capability of the Coherent Ising Machine is combined to process the coupling matrix. The system is divided into several small sub-matrices, each corresponding to a parallel solution unit of the Coherent Ising Machine. This partitioning enables parallel loading and solving of coupling parameters, fully leveraging the hardware parallelism of the Coherent Ising Machine and further improving the solution speed. Simultaneously, it avoids timing conflicts and parameter overflow issues that can occur during Coherent Ising Machine loading due to excessively large single matrix sizes, ensuring the stability of parameter loading and solving. During fixed-point quantization, the sparsified and partitioned coupling matrices are processed... and bias vector Perform fixed-point quantization with a given number of bits, such as 8 or 14 bits (to maintain consistency with the overall parameter precision). Convert the floating-point values ​​of the coupled parameters to fixed-point values, setting a reasonable allocation of integer and decimal places to adapt to the fixed-point arithmetic architecture of the coherent Ising machine while ensuring parameter precision. Coherent Ising machines typically use fixed-point arithmetic to improve computational speed and reduce energy consumption. Fixed-point quantization avoids the incompatibility problem between floating-point parameters and the coherent Ising machine's arithmetic architecture, ensuring that the coherent Ising machine can accurately identify and compute the coupled parameters. Simultaneously, the quantized parameters can be directly converted into a binary bitstream, facilitating rapid loading into the corresponding Ising solver.

[0045] S104, load the coupling parameters and solve the Ising energy minimization problem for the Ising model, and obtain the corresponding spin value based on the solution result.

[0046] In some embodiments, coupling parameters are loaded and the Ising model is solved for Ising energy minimization. The corresponding spin values ​​are obtained based on the solution results, including: Load the Ising model and its corresponding coupling parameters into the preset Ising solver; The Ising solver performs a physical evolution of the Ising model based on coupling parameters, thereby achieving the solution of the Ising energy minimization of the Ising model; Phase detection is performed on the output pulse of the Ising solver to obtain the spin value.

[0047] In some embodiments, the Ising solver includes, but is not limited to: a Coherent Ising Machine (CIM), a quantum annealing computer (e.g., a D-Wave quantum annealer), an adiabatic quantum computer, a CMOS annealer, an FPGA-based annealing accelerator, a GPU-accelerated computing device, a TPU or AI-specific accelerator chip, a classical high-performance computer (HPC), a multi-core CPU server, a parallel computing cluster, a supercomputing platform, a gate-based quantum computer (including a superconducting quantum computer, an ion trap quantum computer, an optical quantum computer, a silicon-based quantum computer, a spin quantum computer, a topological quantum computer, etc.), a simulated quantum simulator, a digital quantum simulator, an optical parametric oscillator network system, a neuromorphic computing chip, a memristor array computing device, and other dedicated or general-purpose computing devices capable of minimizing the energy of the Ising model through quantum computing, simulated annealing, quantum annealing, adiabatic evolution, tensor network computing, Monte Carlo methods, branch and bound algorithms, variational quantum algorithms, or other optimization algorithms.

[0048] In the embodiments of this invention, the Ising solver takes a coherent Ising machine with a built-in optical parametric oscillator (OPO) as an example. The OPO can generate multiple coherent pulses, and the pulse phase and amplitude of the coherent pulses correspond to the Ising spin state, that is, the phase sign of the coherent optical pulse. The corresponding Ising spin state is used. The generated Ising model and the corresponding optimized coupling parameters are loaded into the preset Ising solver to complete the parameter initialization before solving, ensuring that the Ising solver can call the coupling parameters to solve the Ising model. The Ising solver performs physical evolution of the Ising model based on the applied coupling parameters. By simulating the energy change process of the Ising model, it gradually converges to the Ising energy minimum, thus achieving the Ising energy minimization solution. Taking a coherent Ising machine with a built-in optical parametric oscillator as an example, the optical parametric oscillator array 51 in the coherent Ising machine generates multiple coherent pulses. The pulse phase and amplitude of the coherent pulses correspond to the Ising spin state. Based on the applied coupling parameters, the coherent Ising machine adjusts the pump power of the tunable laser source, the cavity loss of the optical parametric oscillator ring cavity, and the injected noise through its built-in pump control circuit 53 to achieve physical annealing evolution. During the evolution process, the phase and amplitude of the coherent pulses adaptively adjust with the energy change of the Ising model, gradually approaching the spin state corresponding to the Ising energy minimum, and finally completing the Ising energy minimization solution. If other types of Ising solvers are used, their specific physical evolution methods can be adapted accordingly. After the Ising solver completes the energy minimization solution, phase detection is performed on its output pulse signal. Based on the Ising spin state corresponding to the pulse phase and amplitude, the corresponding spin value is extracted. This spin value is the optimal solution when the Ising model's energy is minimized, and is used to obtain the robustness verification results of the neural network model. The embodiment of this invention uses a coherent Ising machine with a built-in optical parametric oscillator for Ising solving. The relevant Ising machine hardware, due to its high-speed physical evolution characteristics, is suitable for scenarios with high verification efficiency requirements.

[0049] S105 performs an inverse transformation on the spin values ​​to obtain the robustness verification results of the neural network model.

[0050] In some embodiments, performing an inverse transformation on the spin values ​​to obtain robustness verification results for the neural network model includes: The spin values ​​are inversely transformed into candidate values ​​of the original binary variable and the quantized continuous variable; The input perturbation is reconstructed based on the original binary variables and candidate values ​​of continuous variables, and forward propagation is performed to verify whether the neural network model has classification flipping. If classification flip exists, output the corresponding adversarial example and return a verification failure signal; If no category flipping occurs, a verification pass signal is returned when the stopping condition is met.

[0051] In the embodiments of this invention, the spin variable is obtained by performing phase detection on the pulses output by the optical parametric oscillator array 51. Then the spin variable Convert to original binary variable Quantified candidate values ​​of continuous variables; based on the transformation relationship , spin vector Inverse transformation into a unified binary variable vector Then convert the binary variable vector The quantized bits in the data are concatenated and dequantized to obtain continuous variables (x, ...). , Candidate values ​​for ) will be extracted; the binary variable vector will be extracted. From the original binary bits, we obtain binary selection variables. Furthermore, based on the original binary variables Reconstructing input perturbations using candidate values ​​of continuous variables. Or, using intermediate variables, perform forward propagation to verify whether classification reversal exists, based on the inverse transform. , , Calculate the activation values ​​for each layer: Then based on the output layer activation value Calculate the classification results ;Will With the original input Classification results The comparison checks the feasibility of constraints and determines whether classification reversal exists. The feasibility check is performed using binary selection variables obtained through inverse transformation. Continuous variables ( , , The candidate values ​​are then checked to ensure they satisfy all standardized mixed integer constraints (perturbation domain constraints, network computation constraints, and binary selection variable constraints). Confirmation is made that no constraints are violated, providing a reliable basis for classification reversal judgment and improving accuracy. Finally, based on the classification reversal judgment result, a counterexample sample or validation pass information is output. This indicates the existence of classification flip, i.e., the existence of adversarial examples (valid counterexamples), indicating that the neural network model is not robust. In this case, the output is a reconstructed input perturbation. The corresponding disturbance amplitude is (represents the reconstructed input perturbation) Compared with the original input sample The amplitude difference (used to quantize the magnitude of the input perturbation), and the output values ​​before and after classification flip. and ;like This indicates that there is no classification flip, meaning the neural network model is relatively stable under input perturbations, i.e., the neural network model is stable under the original input samples. The system is locally robust. At this point, it is further determined whether the stopping conditions are met. The stopping conditions include the maximum solution time and no counterexamples after multiple samplings. If the conditions are met, a verification pass signal is sent back.

[0052] In the embodiments of this invention, the constraint problem of robustness verification of a piecewise linear neural network is mapped to the coupling parameters of an Ising model. These coupling parameters are then loaded, and the Ising model is solved by minimizing the Ising energy based on these parameters to obtain the corresponding spin values. Finally, an inverse transformation is performed on the spin values ​​to obtain the robustness verification result, i.e., whether the piecewise linear neural network is robust. Adaptive bit-width fixed-point quantization is used to generate an Ising model from the continuous variables described by the mixed integer constraint problem. This reduces data dimensionality and hardware computation costs while ensuring verification accuracy, significantly improving the speed of robustness verification and making it more compatible with hardware parallel architectures. Parallel exploration of the solution space through physical evolution of the Ising model enables high throughput in multiple problem scenarios. The computational process during verification is relatively simple, improving verification efficiency. Compared to large-scale branch search or integer programming solutions on CPUs / GPUs, this invention incorporates a coherent Ising machine based on an optical parametric oscillator, enabling coherent oscillation and feedback coupling. The physical parallel evolution of the optical parametric oscillator enables the minimization of Ising energy, reducing a large number of instruction-level operations and memory accesses. This makes it suitable for robustness verification of networks under energy constraints, significantly reducing energy consumption and heat dissipation pressure during the verification process.

[0053] like Figure 2 As shown, the embodiment of the present invention provides a piecewise linear neural network robustness verification device, which includes a host interface module 10, a constraint construction module 30, an Ising mapping module 40, an Ising solving module 50, and a result decoding module 60.

[0054] The host interface module 10 is used to receive the robustness verification task of the neural network model to be verified; the constraint construction module 30 is used to construct a mixed integer constraint problem description based on the robustness verification task; the Ising mapping module 40 is used to perform adaptive bit-width fixed-point quantization on the continuous variables of the mixed integer constraint problem description, generate the Ising model and extract the corresponding coupling parameters; the Ising solution module 50 is used to load the coupling parameters and perform Ising energy minimization on the Ising model, and obtain the corresponding spin value according to the solution result; the result decoding module 60 is used to perform inverse transformation on the spin value to obtain the robustness verification result of the neural network model.

[0055] The verification device of this invention can map the constraint problem of robustness verification of piecewise linear neural networks into loadable coupling parameters of a coherent Ising machine through a constraint construction module 30 and an Ising mapping module 40 in the form of a hardware structure. The Ising solution module 50 then loads these coupling parameters and performs Ising energy minimization on the Ising model to obtain the corresponding spin values. The result decoding module 60 performs an inverse transformation on the spin values ​​to obtain the robustness verification result, that is, to determine whether the piecewise linear neural network is robust. This allows the robustness verification of piecewise linear neural networks to be implemented in hardware, enabling repeated verification, calibration, and engineering deployment, and expanding the types of neural networks that can be equivalently verified. Adaptive bit-width fixed-point quantization is performed on the continuous variables described by the mixed integer constraint problem, and the quantization bit depth is dynamically adjusted according to the number of effective segment intervals to generate the Ising model. This reduces data dimensionality and hardware computation costs while ensuring verification accuracy, significantly improving the speed of robustness verification, and is more compatible with hardware parallel architectures.

[0056] In some embodiments, the Ising solver module 50 uses a coherent Ising machine with a built-in optical parametric oscillator to solve the Ising model for Ising energy minimization. Thus, the Ising solver module 50 explores the solution space in parallel through physical evolution using the coherent Ising machine, achieving high throughput in multiple problem scenarios. The computational process during verification is relatively simple, improving verification efficiency. Compared to large-scale branch search or integer programming solutions on CPUs / GPUs, this invention, by incorporating a coherent Ising machine based on an optical parametric oscillator, achieves coherent oscillation and feedback coupling. The physical parallel evolution of the optical parametric oscillator enables Ising energy minimization, reducing a large number of instruction-level operations and memory accesses. This makes it suitable for robust network verification under energy constraints, significantly reducing energy consumption and heat dissipation pressure during the verification process.

[0057] In some embodiments, the Ising solver module 50 may also employ other Ising solvers to solve the Ising model in this invention, such as including but not limited to: quantum annealing computers (e.g., D-Wave quantum annealers), adiabatic quantum computers, CMOS annealers, FPGA-based annealing accelerators, GPU-accelerated computing devices, TPU or AI-specific acceleration chips, classical high-performance computers (HPCs), multi-core CPU servers, parallel computing clusters, supercomputing platforms, gate-based quantum computers (including superconducting quantum computers, ion trap quantum computers, optical quantum computers, silicon-based quantum computers, spin quantum computers, topological quantum computers, etc.), simulated quantum simulators, digital quantum simulators, optical parametric oscillation network systems, neuromorphic computing chips, memristor array computing devices, and other dedicated or general-purpose computing devices capable of minimizing the energy of the Ising model through quantum computing, simulated annealing, quantum annealing, adiabatic evolution, tensor network computing, Monte Carlo methods, branch and bound algorithms, variational quantum algorithms, or other optimization algorithms. Therefore, any dedicated or general-purpose computing device or other technical solution that can equivalently replace the coherent Ising machine in this invention is considered to fall within the scope of protection of this patent.

[0058] In some embodiments, the device of the present invention adopts a heterogeneous integrated architecture, with the constraint construction module 30 and the Ising mapping module 40 integrated on the same FPGA chip. The optical parametric oscillator array 51 of the Ising solver module 50 is opto-interconnected to the FPGA chip via a fiber optic array, and the phase detector of the result decoding module 60 is electrically connected to the FPGA chip via a high-speed ADC. Thus, by adopting a heterogeneous integrated architecture and integrating the constraint construction module 30 and the Ising mapping module 40 on the same FPGA chip, the integration and operational efficiency of the data processing stage in the robustness verification process can be effectively improved. The opto-parametric oscillator array 51 of the Ising solver module 50 is opto-interconnected to the FPGA chip via a fiber optic array, which can fully leverage the advantages of the opto-optical hybrid computing architecture in high-speed interconnection, low-latency transmission, and high parallel computing power, thereby improving the speed of coupling parameter loading and Ising energy minimization solution. The phase detector of the result decoding module 60 is electrically connected to the FPGA chip via a high-speed ADC, enabling rapid acquisition and real-time processing of spin states. The overall architecture improves system hardware integration while reducing data transmission latency and hardware overhead, thereby enhancing the overall efficiency, real-time performance, and system stability of robustness verification of piecewise linear neural network models.

[0059] In some embodiments, the FPGA chip includes an interval operation hard core unit, a Big M method linearization circuit, and a coupling parameter generation unit. The interval operation hard core unit performs fixed-point interval boundary propagation operations; the Big M method linearization circuit generates auxiliary variables and constraint coefficients; and the coupling parameter generation unit converts the mixed-integer constraint problem description into a coupling matrix and a bias vector. Thus, integrating the interval operation hard core unit, the Big M method linearization circuit, and the coupling parameter generation unit into the FPGA chip enables the implementation of fixed-point interval boundary propagation, linearization, and coupling parameter generation within the FPGA chip. This significantly reduces computational latency and improves the speed and stability of mixed-integer constraint transformation and Ising model construction.

[0060] In some embodiments, the host interface module 10 interacts with the host computer 80 (e.g., an industrial control computer) or the test platform. That is, the host interface module 10 can receive robustness verification tasks sent by the host computer 80 or the test platform, transmit the received robustness verification tasks to other modules, and feed back the verification results to the host computer 80 or the test platform.

[0061] In some embodiments, the host interface module 10 is also used to transmit robustness verification results back to the host computer 80 to achieve closed-loop verification control and improve the reliability of robustness verification of the piecewise linear neural network model.

[0062] In some embodiments, the host interface module 10 employs one or more combinations of PCIe, Ethernet, and fiber optic links. This enables the host interface module 10 to achieve high-speed data transmission, adapt to the rapid transmission and reception of large amounts of data such as verification parameters and coupling matrices, ensuring the efficient progress of the verification process; and it supports remote interaction, enabling remote control and data feedback, thus improving the flexibility of use.

[0063] In some embodiments, the piecewise linear neural network robustness verification device further includes a model storage module 20, which is connected to the host interface module 10 and is used to store the robustness verification task transmitted by the host interface module 10. That is, the host interface module 10 can read the data stored in the model storage module 20 and can write the data obtained from the host computer 80 into the model storage module 20.

[0064] In some embodiments, the model storage module 20 includes a weight memory 21, a bias memory 22, and a piecewise linear parameter memory 23. The weight memory 21 stores the weight matrix of the piecewise linear neural network model; the bias memory 22 stores the bias vector of the piecewise linear neural network model; and the piecewise linear parameter memory 23 stores the interval parameters of the piecewise linear activation function.

[0065] In some embodiments, the model storage module 20 also includes a structure for storing other data, and the embodiments of the present invention are not listed here.

[0066] In some embodiments, the weight memory 21 adopts on-chip SRAM, which supports parallel read and write, and can ensure fast retrieval of weight data; its built-in data alignment circuit can convert the floating-point weights issued by the host computer 80 into the data precision corresponding to the weight matrix, and can also realize data verification (parity check) to avoid data transmission errors.

[0067] In some embodiments, the bias memory 22 uses the same on-chip SRAM as the weight memory 21. Its data precision is consistent with the weight matrix, and it corresponds one-to-one with the weight matrix. It also works synchronously with the weight memory 21 to ensure the synchronous calling of weights and biases during inter-layer calculations.

[0068] In some embodiments, the piecewise linear parameter memory 23 adopts an external DDR4 memory. The external DDR4 memory supports batch reading and can meet the fast call requirements of multiple activation parameters. It receives piecewise parameters (slope, intercept, and interval boundary) and provides the constraint construction module 30 with the piecewise information of the activation function.

[0069] In some embodiments, the constraint construction module 30 is connected to the model storage module 20. The constraint construction module 30 includes a boundary propagation submodule 31, a segment selection encoding submodule 32, and an auxiliary variable generation submodule 33. Specifically, the boundary propagation submodule 31 generates the upper and lower bounds of the pre-activation values ​​of each neuron in the neural network model using the Interval Bound Propagation (IBP) algorithm; the segment selection encoding submodule 32 performs infeasible segment pruning based on the upper and lower bounds of the pre-activation values ​​to generate binary variables corresponding to the effective segment intervals of each neuron in the neural network model; and the auxiliary variable generation submodule 33 uses a hardware parallel architecture to convert the product of the binary variables and continuous variables into auxiliary variables and constraint coefficients required for equivalent linear constraints.

[0070] In some embodiments, the fixed-point interval boundary propagation algorithm represents continuous variables in the robustness verification of the neural network model in a fixed-point format, and propagates the input perturbation range of the entire neural network model by calculating the value interval boundary (upper / lower bound) of each variable layer by layer, thereby quickly defining the legal value range of the variables.

[0071] In some embodiments, the segment selection encoding submodule 32 performs infeasible segment pruning based on the upper and lower bounds of the pre-activation value. First, based on the upper and lower bounds of the neuron's pre-activation value, it removes the segment intervals where the pre-activation value will not fall within the upper and lower bounds of the pre-activation value. The remaining segments are the valid segment intervals. Then, a 0 / 1 binary variable is generated for each valid segment interval to mark which valid segment interval the neuron finally falls into.

[0072] In some embodiments, the core of the hardware parallel architecture is to use multiple independent hardware units to execute verification tasks synchronously and in parallel. Leveraging the parallel characteristics of FPGA chips and the physical parallelism of the Coherent Ising machine, verification speed is improved, adapting to the verification requirements of large-scale piecewise linear neural networks. For example, in the auxiliary variable generation process, multiple parallel processing units are designed within the FPGA chip. Each unit independently processes one segment interval of one neuron, synchronously generating auxiliary variables and constraint coefficients without waiting for them one by one.

[0073] In some embodiments, the boundary propagation submodule 31 employs an FPGA hardware pipeline to implement fixed-point interval boundary propagation operations through interval operation hard core units, supporting fixed-point matrix-vector boundary propagation with a computational precision of 32-bit fixed-point numbers. This can reduce the size of variables in subsequent segmented encoding; the input of the boundary propagation submodule 31 is the weight matrix issued by the model storage module 20. Bias vector Input disturbance range and the perturbated input sample The range of values The output represents the upper and lower bounds of the preactivation values ​​of neurons in each layer. , and send to the segment selection encoding submodule 32.

[0074] In some embodiments, the segment selection encoding submodule 32 employs a binary encoding circuit with a built-in interval comparison unit, enabling it to quickly determine the pre-activation boundary and the segment boundary. The intersection of these points is used to prune infeasible segments to determine the effective segmentation interval for each neuron. The segmentation selection encoding submodule 32 uses the upper and lower bounds of the pre-activation values ​​generated by the boundary propagation submodule 31. For each neuron, segmented intervals are selected to generate binary variables. .

[0075] In some implementations, the auxiliary variable generation submodule 33 is used to generate binary variables using a hardware parallel architecture. With continuous variables product term Auxiliary variables required to transform into equivalent linear constraints With constraint coefficients. The auxiliary variable generation submodule 33 adopts the Big M method linearization circuit, with built-in multipliers and adders, supporting real-time generation of auxiliary variables.

[0076] In some embodiments, the Ising mapping module 40 is connected to the constraint construction module 30, and the Ising mapping module 40 includes a coupling loading controller 41. The coupling loading controller 41 adopts an SPI high-speed interface design and is responsible for loading the coupling parameters to the Ising solver module 50; it has a built-in loading timing control circuit to match the working timing of the coherent Ising machine to ensure the synchronization and accuracy of the coupling parameter loading; it also has a parameter loading verification function, which automatically checks the parameter integrity after loading to avoid parameter loss or transmission errors and ensure the stability of the subsequent Ising solver process.

[0077] In some embodiments, the Ising solver module 50 is connected to the Ising mapping module 40. The Ising solver module 50 has a built-in coherent Ising machine based on an optical parametric oscillator. The Ising solver module 50 includes an optical parametric oscillator array 51, an optocoupler network 52, a pump control circuit 53, and an environmental stabilization unit 54.

[0078] In some embodiments, the optical parametric oscillator array 51 includes multiple periodically polarized lithium niobate optical parametric oscillator ring cavities. The physical cavity length of the optical parametric oscillator ring cavity is a preset length. The optical parametric oscillator array 51 generates multiple coherent pulses. The pulse phase and amplitude of the coherent pulses correspond to the Ising spin state, i.e., the phase sign of the coherent optical pulse. This corresponds to the Ising spin state. Periodically polarized lithium niobate is a special optoelectronic material capable of generating coherent light pulses under laser irradiation. An optical parametric oscillator ring cavity allows light to circulate and form a ring optical path, thus generating stable coherent pulses.

[0079] In some embodiments, the optocoupler network 52 includes a photodetector, an analog-to-digital converter (ADC), a digital signal processor (DSP), a digital-to-analog converter (DAC), and an electro-optic modulator. The optocoupler network 52 is used for programmable coupling of the coupling parameter set of the Ising model, enabling preset coupling coefficients. The photodetector detects the intensity or phase of each light pulse; the ADC converts the analog signal output from the photodetector into a digital signal for the DSP to perform calculations; the DSP calculates the coupling coefficients based on the input parameters. The coupling relationship between pulses is calculated; the digital-to-analog converter converts the calculated coupling relationship into an analog signal and sends it to the electro-optic modulator; the electro-optic modulator applies the coupling relationship to the optical path, so that the pulses interact with each other according to the rules.

[0080] In some embodiments, the pump control circuit 53 includes a tunable laser source and a power control circuit. The pump control circuit 53 is used to perform hardware-level adjustment of the pump power, cavity loss, and injection noise of the optical parametric oscillator array 51 to achieve physical annealing evolution. The tunable laser source refers to a pump laser with adjustable power, which can provide energy to the optical parametric oscillator ring cavity and excite optical pulse oscillations. The power control circuit is used to control the intensity of the laser, which can gradually reduce the pump power to achieve physical annealing and allow the system to converge to the optimal solution.

[0081] In some embodiments, the environmental stabilization unit 54 includes a temperature control circuit and an active vibration isolation platform for optical path and phase locking, temperature control, and vibration suppression. The temperature control circuit, as a temperature control module, can maintain the optical path temperature stability and prevent phase drift; the active vibration isolation platform, as a vibration damping platform, can avoid vibration interference with the optical path, ensuring phase stability and accurate calculation.

[0082] In the embodiments of the present invention, the Ising solver module 50 adopts a physical coupling structure of optical parametric oscillator combined with measurement feedback, which enables the minimization of the energy function of the Ising model to be driven by natural laws, provides an internal structure different from that of a general-purpose CPU, and can significantly improve performance.

[0083] Understandably, the optical parametric oscillator array 51 in this invention can generate and maintain phase-coherent OPO pulses by relying on the optical feedback characteristics of the ring cavity. The interaction between these pulses simulates the coupling relationship of spin variables in the Ising model, providing a physical basis for the fast energy minimization solution of the Ising Hamiltonian. In conjunction with embodiments of this invention, the optical parametric oscillator array 51, combined with PLL phase locking and TEC temperature control functions, can ensure the phase consistency of the output pulses from each chamber (ensuring solution stability). Simultaneously, through its parallel array architecture, it adapts to the parallel solution requirements after the coupling matrix is ​​partitioned, enabling fast solutions of ≤100μs, thus improving the real-time performance of the entire verification process.

[0084] In the embodiments of this invention, the optical parametric oscillator array 51 can generate N coherent optical pulses (corresponding to the number of spins), and the phase or amplitude of the coherent optical pulses corresponds to the Ising spin state. The coherent optical pulses generated by each optical parametric oscillator ring cavity of the optical parametric oscillator array 51 have a phase of 0° or 180°, corresponding to the spin state, respectively. and The pulse amplitude is adjusted by the pump power, resulting in high amplitude stability. The optocoupler network 52 can detect the pulse amplitude in real time using a photodetector. After sampling by an analog-to-digital converter, the digital processor calculates the feedback signal, which is then applied to the optical path via a digital-to-analog converter and an electro-optic modulator, achieving arbitrary coupling matrices. The programmable photodetector detects the amplitude of each OPO pulse in real time, samples it after sampling by an analog-to-digital converter, and sends it to a digital feedback processor. The processor then processes the pulses according to the coupling matrix. The calculated feedback signal is converted into an analog signal by a digital-to-analog converter, which controls the electro-optic modulator to adjust the optical path coupling strength, thereby achieving an arbitrary coupling matrix. The pump control circuit 53 adjusts the pump power, cavity loss, and injected noise of the optical parametric oscillator array 51, enabling physical annealing evolution. This promotes high-gain excitation of multimode oscillations, and the gain decreases over time to induce the system to converge to the ground state. For example, the annealing process includes the following three stages: In the initial stage, the pump power is adjusted to 10W, the gain is high, the injected noise intensity is maximum, and multimode oscillations are excited; In the intermediate stage, the pump power is linearly reduced to 5W, the noise intensity gradually decreases, and the cavity loss remains unchanged; In the stable stage, the pump power is reduced to 3W, the noise intensity is 0, and the cavity loss is adjusted to 5dB, causing the system to converge to the minimum energy state. In this way, the Ising solver module 50 can evolve to a stable state under the pump control circuit 53 and the noise injection strategy. After the annealing process, the environmental stabilization unit 54 performs optical path and PLL phase locking, TEC temperature control, and vibration suppression on the optical signal path of the optical parametric oscillator array 51 and the optocoupler network 52 to ensure solution accuracy. By monitoring the temperature of the optical parametric oscillator array 51 and the measurement feedback optical path in real time, it maintains temperature stability based on PID control to avoid optical path deviation caused by temperature changes. It locks the phase of the OPO pulse to ensure coherence. The active vibration isolation platform suppresses the influence of external vibration on the optical path and avoids coupling strength fluctuations.

[0085] In some embodiments, the result decoding module 60 is connected to the Ising solver module 50, and the result decoding module 60 obtains the spin variable by performing phase detection on the pulses output by the optical parametric oscillator array 51. In the embodiments of this invention, the result decoding module 60 employs a phase detector, which corresponds one-to-one with the optical parametric oscillator array 51. The phase detector can detect the phase of each pulse in real time; if the phase is 0°, the spin variable is determined. If the phase is 180°, then determine the spin variable. Furthermore, the result decoding module 60 will determine the spin variable. Convert to original binary variable The system retrieves quantized candidate values ​​of continuous variables; reconstructs the input perturbation based on the original binary variables and candidate values ​​of continuous variables, and performs forward propagation to verify whether classification flip exists; if classification flip exists, the system outputs the reconstructed input perturbation and returns a verification failure signal to the host interface module 10; if classification flip does not exist, the system returns a verification success signal to the host interface module 10 when the stopping condition is met.

[0086] In some embodiments, the piecewise linear neural network robustness verification device further includes a control scheduling module 70, which is connected to the host interface module 10, the constraint construction module 30, the Ising mapping module 40, the Ising solution module 50, and the result decoding module 60. The control scheduling module 70 is used to control the neural network model to perform robustness verification tasks, such as initiating boundary propagation, generating a quadratic unconstrained binary optimization model, and solving the coherent Ising machine. The control scheduling module 70 also transmits the verification conclusions (including counterexamples or proof information) back to the host computer 80 or the test platform through the host interface module 10.

[0087] In some embodiments, the control scheduling module 70 includes a monitoring stop criterion unit 71, which is used to determine whether to terminate the current verification task.

[0088] In some embodiments, equivalent substitutions can be made to the circuit structure, memory organization, quantization bit width, and CIM physical implementation (such as using superconducting qubits to replace optical parametric oscillators) of each module of the piecewise linear neural network robustness verification device in the embodiments of the present invention, and these substitutions are all considered to fall within the scope of protection of this patent.

[0089] The piecewise linear neural network robustness verification device provided by the embodiments of this invention, due to its hardware structure, generates an accurate mixed-integer constraint problem description through the constraint construction module 30, completes the efficient conversion from the constraint problem to the Ising model through the Ising mapping module 40, and achieves the rapid minimization of the Ising energy by utilizing the physical parallel evolution characteristics of the Ising solution module 50. Combined with the result decoding module 60, it completes spin inverse transformation and classification verification. The entire process is achieved through hardware collaboration without software iterative computation, which significantly reduces the computational complexity of piecewise linear neural network robustness verification, greatly improves verification efficiency, and reduces a large number of instruction-level operations and memory accesses, significantly reducing energy consumption and heat dissipation pressure during the verification process. Furthermore, the device adopts a modular and configurable design, which can adapt to piecewise linear neural networks with different structures and different numbers of segments, enabling repeated verification, calibration, and engineering deployment, expanding the types of neural networks that can be equivalently verified, and improving the accuracy and stability of robustness verification through accurate constraint construction and solving, effectively solving the technical pain points of low verification efficiency, high energy consumption, and poor versatility in related technologies.

[0090] As can be seen from the above description, the embodiments of the present invention achieve the following technical effects: (1) Adaptive bit-width fixed-point quantization and effective segmented interval pruning are adopted to significantly reduce the variable size and hardware computation overhead while ensuring verification accuracy, and improve the efficiency of constraint construction and Ising mapping. (2) The physical parallel evolution of the coherent Ising machine based on optical parametric oscillator is used to solve the Ising energy minimization problem, replacing the traditional branch and bound and integer programming of CPU / GPU, which significantly reduces computational complexity, memory access and instruction overhead, and achieves high-speed and low-power verification. (3) By accelerating the interval propagation, linearization and decoding process through FPGA hardware pipeline and parallel architecture, the overall system has low latency and high throughput, which can meet the real-time requirements of autonomous driving, safety detection and other scenarios. (4) The optoelectronic heterogeneous integrated architecture is adopted, the modules are specialized and the process is hardware-based, which has strong versatility and high stability, and is easy to deploy and verify repeatedly in engineering. It effectively solves the problems of high complexity, low efficiency and high energy consumption of traditional verification methods.

[0091] It should be noted that the term "comprising" and its variations used in the embodiments of this invention are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "a plurality" mentioned in the embodiments of this invention are illustrative and not restrictive, and those skilled in the art should understand that unless explicitly indicated otherwise in the context, they should be understood as "one or more".

[0092] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0093] The steps described in the method embodiments provided by the present invention can be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0094] The term "embodiment" in this specification refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily imply the same embodiment, nor does it imply independence or alternativeity from other embodiments. The various embodiments in this specification are described in a related manner, with reference to each other for similar or identical parts. In particular, for apparatus, device, and system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, and relevant details are referred to in the description of the method embodiments.

[0095] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for verifying the robustness of a piecewise linear neural network, characterized in that, include: Receive robustness verification tasks for neural network models to be verified; A hybrid integer constraint problem description is constructed based on the robustness verification task described above; Adaptive bit-width fixed-point quantization is performed on the continuous variables described in the mixed integer constraint problem to generate the Ising model and extract the corresponding coupling parameters. Load the coupling parameters and solve the Ising energy minimization problem of the Ising model, and obtain the corresponding spin value based on the solution result; The spin values ​​are inversely transformed to obtain the robustness verification results of the neural network model; The Ising model is solved by minimizing the Ising energy, and the corresponding spin values ​​are obtained based on the solution results, including: The Ising model and the corresponding coupling parameters are loaded into the preset Ising solver; The Ising solver performs a physical evolution of the Ising model based on the coupling parameters, thereby achieving the solution of minimizing the Ising energy of the Ising model; Phase detection is performed on the output pulse of the Ising solver to obtain the spin value.

2. The robustness verification method for piecewise linear neural networks according to claim 1, characterized in that, Based on the robustness verification task, a description of the mixed integer constraint problem is constructed, including: Based on the robustness verification task, robustness verification constraints of the neural network model are extracted, including network computation constraints and perturbation domain constraints. The network computation constraints and perturbation domain constraints are converted into a set of linear inequalities; Selection and auxiliary variables are introduced into the piecewise linear activation function of the neural network model; A standardized description of the mixed integer constraint problem is constructed based on the set of linear inequalities, the selection variable, and the auxiliary variable.

3. The robustness verification method for piecewise linear neural networks according to claim 1, characterized in that, Adaptive bit-width fixed-point quantization is performed on the continuous variables described in the mixed integer constraint problem to generate the Ising model and extract the corresponding coupling parameters, including: The continuous variables in the description of the mixed integer constraint problem are estimated by the fixed-point interval boundary propagation algorithm to obtain the quantization range, and the quantization range is subjected to adaptive bit-width fixed-point quantization processing. The quantized continuous variables are then encoded based on multiple binary variables. The inequality constraints in the description of the mixed integer constraint problem are transformed into an equality structure by using non-negative relaxation variables. Construct a unified binary variable vector, which includes all original binary variables, quantized bits of continuous variables, and relaxation bits corresponding to non-negative relaxation variables; Based on the robustness verification task, the initial optimization objective and constraint violation penalty term of the neural network model are determined, and a quadratic unconstrained binary optimization model is constructed based on the initial optimization objective and the constraint violation penalty term. The quadratic unconstrained binary optimization model is transformed into the Ising Hamiltonian form using matrix transformation, resulting in the Ising model. The weight matrix and bias vector of the Ising model are optimized to obtain the corresponding coupling parameters.

4. The robustness verification method for piecewise linear neural networks according to claim 3, characterized in that, The adaptive bit width is determined by the number of effective segment intervals of the linear segmented activation function of the neural network model.

5. The robustness verification method for piecewise linear neural networks according to claim 1, characterized in that, Performing an inverse transform on the spin values ​​to obtain the robustness verification results of the neural network model includes: The spin values ​​are inversely transformed into candidate values ​​of the original binary variable and the quantized continuous variable; The input perturbation is reconstructed based on the original binary variables and the candidate values ​​of the continuous variables, and forward propagation is performed to verify whether the neural network model has classification flipping. If classification flip exists, output the corresponding adversarial example and return a verification failure signal; If no category flipping occurs, a verification pass signal is returned when the stopping condition is met.

6. A robustness verification device for piecewise linear neural networks, used to execute the robustness verification method for piecewise linear neural networks according to any one of claims 1 to 5, characterized in that, include: The host interface module is used to receive robustness verification tasks for the neural network model to be verified. A constraint construction module is used to construct a hybrid integer constraint problem description based on the robustness verification task; The Ising mapping module is used to perform adaptive bit-width fixed-point quantization on the continuous variables described by the mixed integer constraint problem, generate the Ising model, and extract the corresponding coupling parameters. The Ising solver module is used to load the coupling parameters and perform Ising energy minimization on the Ising model, obtaining the corresponding spin value based on the solution result. The process of performing Ising energy minimization on the Ising model and obtaining the corresponding spin value based on the solution result includes: loading the Ising model and the corresponding coupling parameters into a preset Ising solver; the Ising solver performing physical evolution on the Ising model based on the coupling parameters to achieve Ising energy minimization; and performing phase detection on the output pulse of the Ising solver to obtain the spin value. The result decoding module is used to perform an inverse transformation on the spin values ​​to obtain the robustness verification results of the neural network model.

7. The piecewise linear neural network robustness verification device according to claim 6, characterized in that, The Ising solution module uses a coherent Ising machine with a built-in optical parametric oscillator to perform Ising energy minimization on the Ising model.

8. The piecewise linear neural network robustness verification device according to claim 7, characterized in that, The device adopts a heterogeneous integrated architecture. The constraint construction module and the Ising mapping module are integrated on the same FPGA chip. The optical parametric oscillator array of the Ising solving module is opto-interconnected with the FPGA chip through an optical fiber array. The phase detector of the result decoding module is electrically connected to the FPGA chip through a high-speed ADC.

9. The piecewise linear neural network robustness verification device according to claim 8, characterized in that, The FPGA chip includes: The interval arithmetic hard core unit is used to perform fixed-point interval boundary propagation operations; The Big M linearization circuit is used to generate auxiliary variables and constraint coefficients; The coupling parameter generation unit is used to convert the description of a mixed integer constraint problem into a coupling matrix and a bias vector.

10. The robustness verification device for piecewise linear neural networks according to claim 9, characterized in that, The constraint construction module includes: The boundary propagation submodule is used to generate the upper and lower bounds of the pre-activation values ​​of each neuron in the neural network model using a fixed-point interval boundary propagation algorithm. The segmented selection encoding submodule is used to prune infeasible segments based on the upper and lower bounds of the pre-activation value, and generate binary variables corresponding to the effective segmented intervals of each neuron in the neural network model. The auxiliary variable generation submodule is used to convert the product of the binary variable and the continuous variable into the auxiliary variables and constraint coefficients required for equivalent linear constraints using a hardware parallel architecture.

11. The robustness verification device for piecewise linear neural networks according to claim 8, characterized in that, The Ising solver module includes: An optical parametric oscillator array includes multiple periodically polarized lithium niobate optical parametric oscillator ring cavities, the physical cavity length of each optical parametric oscillator ring cavity being a preset length, and the optical parametric oscillator array being used to generate multiple coherent pulses, the pulse phase and amplitude of which correspond to the Ising spin state. An optocoupler network includes a photodetector, an analog-to-digital converter, a digital signal processor, a digital-to-analog converter, and an electro-optic modulator. The optocoupler network is used for programmable coupling of the coupling parameter set of the Ising model. The pump control circuit includes a tunable laser source and a power control circuit. The pump control circuit is used to perform hardware-level adjustment of the pump power, cavity loss and injection noise of the optical parametric oscillator array to achieve physical annealing evolution. An environmental stabilization unit, including a temperature control circuit and an active vibration isolation platform, is used for optical path and phase locking, temperature control, and vibration suppression.

12. The robustness verification device for piecewise linear neural networks according to any one of claims 7 to 11, characterized in that, The device further includes a model storage module, the model storage module comprising: A weight memory, wherein the weight memory is used to store the weight matrix of the neural network model; Bias memory, the bias memory being used to store the bias vector of the neural network model; Piecewise linear parameter memory, wherein the piecewise linear parameter memory is used to store the interval parameters of the piecewise linear activation function of the neural network model.

13. The robustness verification device for piecewise linear neural networks according to any one of claims 7 to 11, characterized in that, The device further includes a control and scheduling module, which is connected to the host interface module, the constraint construction module, the Ising mapping module, the Ising solving module, and the result decoding module, respectively. The control and scheduling module is used to control the neural network model to perform a robustness verification task.

Images