Picture annotation neural network model construction method and system
By converting the traditional image annotation neural network model into a pulsed neural network, using the pulsed Gaussian linear error unit to improve robustness and adaptability, the problem of insufficient robustness of traditional models when capturing transient features of images is solved, and the more High labeling accuracy and efficiency.
Patent Information
- Application Number
- CN202411213463.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-06
AI Technical Summary
Traditional image annotation neural network models are difficult to capture transient or burst features when processing complex image features, resulting in poor robustness.
Replace the Gaussian linear error unit in the original picture annotation neural network model with a pulsed Gaussian linear error unit and convert it into a pulsed neural network to improve the robustness and adaptability of the model.
By simulating the pulse distribution characteristics of neurons, the accuracy of picture annotation and the generalization ability of the model are significantly improved, while reducing the computational complexity and memory usage, and improving the labeling efficiency.
Smart Images

Figure CN119089964A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing neural network construction, and specifically to a method and system for constructing an image annotation neural network model. Background Art
[0002] In the field of image processing and computer vision, image annotation is a crucial task, which involves automatically identifying and classifying image content and assigning corresponding labels or descriptions. The traditional way of image annotation is pure manual annotation, which consumes a lot of manpower and is inefficient. With the rapid development of deep learning technology, especially the widespread application of neural network models in image recognition and understanding, it has become possible to annotate images using neural network models, and the accuracy and efficiency of image annotation have been significantly improved. However, traditional neural network models for image annotation mostly rely on continuous activation functions. When these models process complex image features, they often have problems such as difficulty in effectively capturing transient or burst features in images, making the robustness of image annotation neural networks poor. Summary of the invention
[0003] To solve the above problems, the present invention provides a method and system for constructing a picture annotation neural network model, which uses a pulse Gaussian linear error unit to convert the original picture annotation neural network model into a pulse neural network, and utilizes the nonlinearity of the pulse Gaussian linear error unit to improve the robustness of the picture annotation neural network.
[0004] In a first aspect, the technical solution of the present invention provides a method for constructing a neural network model for image annotation, comprising the following steps: Construct an original image annotation neural network model, in which the activation function uniformly uses Gaussian linear error unit; Train the neural network model for original image annotation; The trained original image annotation neural network model is converted into a spiking neural network using a spiking Gaussian linear error unit, which is the final constructed image annotation neural network model.
[0005] In an optional implementation, a pulsed Gaussian linear error unit is used to convert the trained original image annotation neural network model into a pulsed neural network, specifically including: Construct a pulsed Gaussian linear error unit; The Gaussian linear error unit of the original image annotation neural network model is replaced with the constructed pulse Gaussian linear error unit to obtain a pulse neural network.
[0006] In an optional embodiment, the pulsed Gaussian linear error unit is a pulsed Gaussian linear error unit with a scaling parameter; Construct a pulse Gaussian linear error unit, specifically including: Configuring Scaling Parameters ; The energy accumulation of the pulse Gaussian linear error unit is calculated by the following formula,
[0007] In the formula, represents the energy accumulated by the pulse Gaussian linear error unit at the nth time step, represents the energy accumulated by the pulse Gaussian linear error unit at the previous time step, represents the input of the nth time step, express Cumulative distribution function of the standard normal distribution; The pulse value is obtained by the following formula based on energy accumulation , which means if the energy is positive, it is rounded down, otherwise it is rounded up.
[0008]
[0009] According to the scaling parameters The pulse value is calculated by the following formula Scaling to get the output value of the nth time step ,
[0010] The energy attenuation of the pulsed Gaussian linear error unit is calculated by the following formula, .
[0011] In an optional embodiment, the trained original image annotation neural network model is converted into a spiking neural network using a spiking Gaussian linear error unit, further comprising: The pulse sparsity of the spiking neural network at the nth time step is calculated by the following formula ,
[0012]
[0013] In the formula, Represents the output mask of the spiking neural network at the nth time step , Represents the output mask of the neural network at the n-1th time step , Indicates the output mask The number of computational units after expansion; Calculate the average of the spike sparsity over all time steps ; Determining pulse sparsity Is it greater than the threshold? If so, reduce the scaling parameter by the preset step size ; Otherwise, no adjustment is made; Use the new scaling parameters Reconstruct the pulse Gaussian linear error unit and obtain a new pulse neural network until the pulse sparsity The requirements are met and the final spiking neural network is obtained.
[0014] In an optional embodiment, the input of the spiking neural network at each time step remains unchanged, and its final output result is the average value of the output masks of all time steps.
[0015] In an optional implementation, training the original image annotation neural network model specifically includes: Use Gaussian process as a surrogate model to approximate the objective function, and use the surrogate model to guide the next sampling to gradually approach the optimal solution; Adopting the expected improvement strategy, minimizing the loss function is used as maximizing the expected improvement; After each sampling, the value of the loss function is calculated according to the new sampling point, and the proxy model is updated; Hyperparameter tuning is performed using the Bayesian optimization algorithm. In each iteration, the Bayesian optimizer selects the next hyperparameter combination for evaluation based on the current surrogate model and the expected improvement strategy, and updates the surrogate model until the optimal hyperparameter combination is found.
[0016] In an optional embodiment, the original image annotation neural network model is a Transformer model, including an image encoder, a hint encoder and a mask decoder; The image encoder is encoded through self-attention mechanism, multi-head attention mechanism, and feedforward neural network respectively; The prompt encoder accepts point prompt information, box prompt information and text prompt information input by the user; The mask decoder consists of deconvolution layers, upsampling layers, and convolution layers; The original image annotation neural network model uses a combination of binary cross entropy loss and boundary loss to construct the loss function; During the training process of the original image annotation neural network model, a coordination mechanism between modules is added, including adding parameter-learnable coordination matrices to the attention heads in the image encoder, the feedforward neural network, and each convolutional layer in the mask decoder.
[0017] In an optional implementation, after the original image annotation neural network model is trained, a step of compressing the trained image annotation neural network model is further included, specifically including: (1) Pruning the model: During the forward propagation of the model, collect the attention weight matrix of each attention head, average the attention weight matrix of each attention head, obtain the average attention weight of each attention head, and discard several attention heads with low average attention weight; (2) Perform low-rank decomposition on the convolutional layers in the image encoder and mask decoder: Perform singular value decomposition on the weight matrix of each target convolutional layer, truncate the calculated singular values, retain the largest singular values, reconstruct the low-rank matrix using the truncated singular value matrix, and replace the weight matrix of the original convolutional layer with the reconstructed low-rank matrix; (3) Convert the model's floating-point weights to integer weights: (a) Calculate the weight of each floating point number Minimum value of and maximum value , determine the quantization range,
[0018] (b) Calculate the scaling factor based on the range , maps 32-bit floating point numbers to 8-bit integers, and for 8-bit integers, q max and q min 128 and -127 respectively,
[0019] (c) Using scaling factors The floating point weight Convert to integer weights ,
[0020] (d) Integer weights for storage.
[0021] In a second aspect, the technical solution of the present invention provides a system for constructing a neural network model for image annotation, comprising: Original model building module: builds an original image annotation neural network model. The activation function in the original image annotation neural network model uniformly uses Gaussian linear error units; Original model training module: train the original image annotation neural network model; Spiking neural network conversion module: Use the pulsing Gaussian linear error unit to convert the trained original image annotation neural network model into a pulsing neural network, which is the final constructed image annotation neural network model.
[0022] In an optional embodiment, the system further includes a model compression module: compressing the original image annotation neural network model; The model compression module includes: Model pruning unit: prunes the model, including collecting the attention weight matrix of each attention head during the forward propagation of the model, averaging the attention weight matrix of each attention head to obtain the average attention weight of each attention head, and discarding several attention heads with low average attention weight; Low-rank decomposition unit: performs low-rank decomposition on the convolutional layers in the image encoder and mask decoder, including performing singular value decomposition on the weight matrix of each target convolutional layer, truncating the calculated singular values, retaining the largest singular values, reconstructing the low-rank matrix using the truncated singular value matrix, and replacing the weight matrix of the original convolutional layer with the reconstructed low-rank matrix; Computational precision compression unit: Converts the model's floating-point weights into integer weights, including: (a) Calculate the weight of each floating point number Minimum value of and maximum value , determine the quantization range,
[0023] (b) Calculate the scaling factor based on the range , maps 32-bit floating point numbers to 8-bit integers, and for 8-bit integers, q max and q min 128 and -127 respectively,
[0024] (c) Using scaling factors The floating point weight Convert to integer weights ,
[0025] (d) Integer weights for storage.
[0026] The method and system for constructing a neural network model for image annotation provided by the present invention have the following beneficial effects compared to the prior art: by replacing the continuous activation function in the traditional neural network with a pulsed Gaussian linear error unit, the model can more effectively simulate the pulse emission characteristics of neurons when processing image information, thereby more accurately capturing the details and dynamic changes in the image and improving the accuracy of image annotation. The use of the pulsed Gaussian linear error unit enables the model to exhibit stronger adaptability and robustness when processing unseen images or complex scenes, thereby improving the generalization ability of the model. At the same time, the pulsed neural network, with its event-driven and sparsely activated characteristics, can significantly reduce the computational complexity and memory usage compared to the traditional continuous neural network, thereby accelerating the model operation process and improving the efficiency of image annotation. BRIEF DESCRIPTION OF THE DRAWINGS In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0027] Figure 1 It is a flow chart of a method for constructing a neural network model for image annotation provided by an embodiment of the present invention.
[0028] Figure 2 It is a schematic block diagram of the structure of a picture annotation neural network model construction system provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.
[0031] The key terms appearing in the present invention are explained below.
[0032] GELU: Gaussian error Linear Unit, Gaussian linear error unit.
[0033] SNN: Spiking Neural Network.
[0034] SGELU: Spiking Gaussian error Linear Unit.
[0035] Figure 1 This is a flow chart of a method for constructing a neural network model for image annotation provided by an embodiment of the present invention. Figure 1 The execution subject may be a picture annotation neural network model construction system. According to different requirements, the order of the steps in the flowchart may be changed, and some may be omitted. The picture annotation neural network model construction method provided in the embodiment of the present invention is executed by a computer device, and accordingly, the picture annotation neural network model construction system runs in the computer device.
[0036] like Figure 1 As shown, the method includes the following steps.
[0037] S1, build the original image annotation neural network model, the activation function in the original image annotation neural network model uniformly uses the Gaussian linear error unit.
[0038] S2, train the neural network model for original image annotation.
[0039] S3, uses a pulsed Gaussian linear error unit to convert the trained original image annotation neural network model into a pulsed neural network, which is the final constructed image annotation neural network model.
[0040] To further understand the present invention, a specific embodiment is provided below to further explain the present invention in detail. The specific embodiment first constructs an interactive segmentation model with an inter-block coordination mechanism, uses GELU to train the model, and then uses SGELU to convert the network into SNN. The SGELU nonlinearity can usually make the network more robust. At the same time, the specific embodiment uses model compression technologies such as model pruning and calculation accuracy compression, as well as Bayesian optimization automatic machine learning methods, to lightweight the interactive model and improve the model operation efficiency.
[0041] 1. Build a neural network model for original image annotation.
[0042] The original picture annotation neural network model of this specific embodiment is a Transformer model, including a picture encoder, a hint encoder and a mask decoder, and adopts an inter-block coordination mechanism during training, and uses a combination of binary cross entropy loss and boundary loss to construct a loss function. The activation function of the original picture annotation neural network model uniformly uses GELU.
[0043] (a) Encode an image using an image encoder.
[0044] 1. Downsample the images uploaded by the user and use them as input for the algorithm.
[0045] 2. Divide the image into a series of image blocks , embed the image blocks into high-dimensional space and form a feature matrix of all embedded image blocks , and input into the Transformer structure for processing. The input of the Transformer structure is expressed as follows:
[0046] 3. Encode through self-attention mechanism, multi-head attention mechanism and feedforward neural network respectively. The encoded image feature is F_image.
[0047] Step 1: Get Q (query matrix), K (key matrix) and V (value matrix) based on the input feature matrix, W Q , W K , W V are the corresponding learnable weight matrices respectively.
[0048]
[0049] Step 2: Calculate the i-th head self-attention matrix A i With the self-attention output Z,d k The dimensions for each head.
[0050]
[0051]
[0052] Step 3, use h attention heads to perform a multi-head attention mechanism to obtain the attention distribution, which is expressed by the following formula, is the linear transformation matrix of multi-head attention, and Concat concatenates the outputs of all attention heads along the feature dimension (i.e., the dimension of each vector).
[0053]
[0054] The calculation method of each attention head is the above A i
[0055] Step 4: The output of the multi-head attention mechanism is represented as X, which is processed by a feedforward neural network as shown below. The network uses the GeLU activation function.
[0056]
[0057] In the formula, , , , is a learnable bias.
[0058] (ii) The prompt encoder receives prompt information input by the user.
[0059] The prompt encoder accepts the point prompt information, box prompt information and text prompt information input by the user, and embeds them into high-dimensional space respectively. For point prompts and box prompt information, it supports key point, rectangular box and polygon annotation forms, corresponding to the annotation of one key point, four key points and any number of key points respectively.
[0060] 1. The position prompt information is mapped to the high-dimensional space through the sinusoidal position mapping function.
[0061]
[0062] is a set of points representing a region, is the weight matrix, For bias.
[0063] 2. The text prompt information is embedded in the high-dimensional space in the form of Word2Vec. It is a text prompt information.
[0064]
[0065] 3. Combine the location prompt information and text prompt information to generate F_prompt.
[0066]
[0067] (iii) The mask decoder performs feature fusion and outputs the mask.
[0068] 1. Use linear feature fusion to combine F_image and F_prompt to form F_fusion feature, where the coefficient and Take 0.7 and 0.3 respectively.
[0069]
[0070] 2. Use the mask decoder to process the F_fusion features. The decoder consists of a deconvolution layer, an upsampling layer, and a convolution layer, which are expressed as the following formulas.
[0071]
[0072]
[0073]
[0074] The final output As a mask, the mask result is stored in the database and returned to the front end to be drawn in the picture uploaded by the user.
[0075] (IV) Inter-block coordination mechanism.
[0076] Since the model structure is relatively complex, a coordination mechanism between modules can be added during training to improve the training efficiency of specific scenarios. This specific embodiment mainly refers to the attention head in the image encoder, the feedforward neural network, and each convolutional layer in the decoder, adding a series of parameter-learnable coordination matrices .
[0077] Assume that the output of each collaborative module (referring to the above encoder attention head, etc.) is The corresponding coordination matrix is , through the coordination matrix adjustment, the original output changes from for .
[0078]
[0079] In the loss function, the Frobenius norm is used to regularize the relationship matrix. is the Frobenius norm of the i-th relationship matrix, is the coefficient in the total loss function and is set to 0.1.
[0080]
[0081] (V) Construct the loss function using a combination of binary cross entropy loss and boundary loss.
[0082] Binary Cross Entropy Loss It is expressed as:
[0083] In the formula, is the true label, is the predicted value, and N is the number of samples. The coefficient of the cross entropy loss function is 0.9. The total loss function is
[0084] 2. Train the neural network model for annotating original images.
[0085] This specific embodiment combines the Bayesian optimization algorithm to perform model training to improve the model effect, by building a proxy model to approximate the objective function, and using the proxy model to guide the next step of sampling, thereby gradually approaching the optimal solution. Combined with automatic machine learning, Bayesian optimization can automatically adjust the model's hyperparameters, thereby significantly improving the model's performance.
[0086] The process of using Bayesian optimization on the original image annotation neural network model is as follows.
[0087] 1. Select the surrogate model: Use Gaussian Process (GP) as the surrogate model.
[0088] 2. Sampling strategy: adopt the expected improvement strategy, which will select the point that can maximize the expected improvement. In this specific embodiment, minimizing the above loss function is used as maximizing the expected improvement.
[0089] 3. Update the proxy model: After each sampling, the value of the loss function is calculated according to the new sampling point, and the Gaussian process proxy model is updated to more accurately reflect the distribution of the objective function.
[0090] 4. Hyperparameter search space setting: Define the hyperparameter search space of the Transformer model, which in this specific embodiment includes learning rate, batch size, number of attention heads, number of layers, hidden layer size, etc.
[0091] 5. Bayesian optimization training: Use the Bayesian optimization algorithm to adjust hyperparameters. In each iteration, the Bayesian optimizer selects the next hyperparameter combination for evaluation based on the current proxy model and sampling strategy, and updates the proxy model until the optimal hyperparameter combination is found.
[0092] 3. Compress the trained image annotation neural network model.
[0093] After model training, a model compression process is performed, wherein when pruning multi-head attention, the average contribution of each attention head has been calculated during the training process. This specific embodiment uses a series of model compression methods for the trained image annotation neural network model to improve the efficiency of the model operation, and then replaces the activation function to achieve the conversion from the neural network model to the spiking neural network.
[0094] (a) Pruning the model.
[0095] Prune the attention heads in the multi-layer Transformer encoder. In the multi-head self-attention mechanism, each attention head generates an attention weight matrix, which represents the correlation between each position in the input sequence. By calculating the average attention weight of each attention head on the entire training set, its contribution to the model performance can be evaluated. The calculation process is as follows.
[0096] 1. Collect attention weights: During the forward propagation of the model, collect the attention weight matrix of each attention head.
[0097] 2. Calculate the average value: For each attention head, calculate its average on the training set, which is expressed here as , through the above A i Perform calculations.
[0098]
[0099] Assuming that the training set has N samples, the attention weight matrix calculated for each sample is , then the average attention weight matrix is , which is calculated as shown above.
[0100] 3. Evaluate importance: Calculate the Frobenius norm of the matrix (i.e., the square root of the sum of the squares of each element of the matrix), and evaluate the importance of each attention head based on the size of the calculation result. Heads with lower weights are considered unimportant. In this specific embodiment, the weights are sorted and the 20% attention heads with lower weights are discarded.
[0101] (ii) Perform low-rank decomposition on the convolutional layers in the image encoder and mask decoder.
[0102] Low-rank decomposition is an effective technique that significantly reduces the number of model parameters and computation by decomposing a high-dimensional weight matrix into the product of multiple low-dimensional matrices. The following are the steps for low-rank decomposition of the convolutional layers in the image encoder and mask decoder, using singular value decomposition (SVD) to decompose the high-dimensional convolution kernel into a combination of low-dimensional kernels.
[0103] 1. Perform singular value decomposition on the weight matrix of each target convolutional layer.
[0104] Perform singular value decomposition on the weight matrix of each target convolution layer. Assuming that the weight matrix of the convolution kernel is W∈Rm×n, it can be decomposed into the product of three matrices: W=UΣVT. Among them, U∈Rm×r and V∈Rn×r are orthogonal matrices, Σ∈Rr×r is a diagonal matrix, and r is the rank of the matrix.
[0105] 2. Truncate the calculated singular values and retain the largest singular values.
[0106] In order to further reduce the amount of calculation, the singular values can be truncated to keep only the largest k singular values.
[0107] The truncated matrix is ,in is the i-th singular value 3. Use the truncated singular value matrix to reconstruct the low-rank matrix and replace the weight matrix of the original convolutional layer with the reconstructed low-rank matrix.
[0108] Reconstruct the low-rank matrix using the truncated singular value matrix: .
[0109] , are all truncated orthogonal matrices. After reconstruction, the weight matrix of the original convolutional layer is replaced with the matrix after low-rank decomposition.
[0110] (iii) Convert the model’s floating-point weights into integer weights to compress computational accuracy.
[0111] Reduce computation and storage requirements by converting model weights and activations from high precision to low precision. The weights in the image encoder, hint encoder, and mask decoder are the main targets of this step. These modules occupy a lot of computational resources and storage space in the model. Convert the 32-bit floating point weights in the model to 8-bit integers, which significantly reduces the computational cost of the model.
[0112] (a) Calculate the weight of each floating point number Minimum value of and maximum value , determine the quantization range,
[0113] (b) Calculate the scaling factor based on the range , maps 32-bit floating point numbers to 8-bit integers, and for 8-bit integers, q max and q min 128 and -127 respectively,
[0114] (c) Using scaling factors The floating point weight Convert to integer weights ,
[0115] (d) Integer weights for storage.
[0116] 4. Convert the trained original image annotation neural network model into a spiking neural network.
[0117] This specific embodiment uses a pulse Gaussian linear error unit to convert the trained original image annotation neural network model into a pulse neural network. It should be noted that after the trained original image annotation neural network model is compressed, the compressed model is converted into a pulse neural network. The pulse Gaussian linear error unit is first constructed, and then the network is converted into an SNN.
[0118] (A) Structural design of the SGELU unit with scaling parameters.
[0119] 1. SGELU unit energy accumulation.
[0120] SNN has time domain characteristics, so its SGELU design also has time domain characteristics. The energy accumulation of the pulse Gaussian linear error unit is calculated by the following formula.
[0121]
[0122] In the formula, represents the energy accumulated by the pulse Gaussian linear error unit at the nth time step, represents the energy accumulated by the pulse Gaussian linear error unit at the previous time step, represents the input of the nth time step, express Cumulative distribution function of the standard normal distribution.
[0123]
[0124] 2. The SGELU unit releases a pulse.
[0125] The pulse value is obtained by the following formula based on energy accumulation , the following formula indicates that if the energy is positive, it is rounded down, otherwise it is rounded up.
[0126]
[0127]
[0128] 3. Scale the pulse value to get the output value.
[0129] According to the scaling parameters The pulse value is calculated by the following formula Scaling to get the output value of the nth time step , and this output value is the input of other connected neurons in the next layer of the network.
[0130]
[0131] 4. Energy decay of SGELU unit.
[0132] The energy attenuation of the pulsed Gaussian linear error unit is calculated by the following formula. Through the process of energy accumulation and attenuation, the behavior of this calculation unit will be similar to that of GELU.
[0133]
[0134] (B) Measurement of pulse sparsity.
[0135] For spiking neural networks, due to their dynamic characteristics in the time domain, the output should vary between different time steps. Therefore, the change in the output layer is used to calculate a spike sparsity index of the network, denoted as " In this specific embodiment, the indicator is mainly controlled by changing the above-mentioned s_rate scaling parameter.
[0136] The output layer at time step n is recorded as " Since the prediction uses the average output of the last few time steps, the sparsity index value is also averaged between these time steps, which can be regarded as the average percentage change in the output of each computing unit in the output layer between consecutive time steps. In this specific embodiment, That is the output mask corresponding to time step n ,therefore for The number of computational units after flattening.
[0137] The spike sparsity of the spike neural network is calculated by the following formula ,
[0138]
[0139] In the formula, Represents the output mask of the spiking neural network at the nth time step , Represents the output mask of the neural network at the n-1th time step .
[0140] When the scaling ratio is very small, most of the computational units are often in a non-output state, which results in a small indicator value. If the scaling factor is increased slightly, the computational units start to fire pulses more frequently and the indicator value may also become larger. If the scaling factor is increased further, the pulse density becomes very high. In this case, the indicator value becomes smaller again and the network looks more like a non-spiking network. This indicator is used to control the sparsity of the network in the subsequent conversion SNN step. <0.3, then reduce the scaling factor and gradually increase .
[0141] (iii) Convert the network to SNN.
[0142] The Gaussian linear error unit of the original image annotation neural network model is replaced with the constructed pulse Gaussian linear error unit to obtain a pulse neural network.
[0143] The activation function in the original network is GELU, and it is replaced by the above SGELU unit to complete the SNN conversion. After completing the SNN conversion, the pulse sparsity is measured to ensure that the sparsity of the network is not too low, including calculating the pulse sparsity of the spiking neural network and determining whether the pulse sparsity is greater than the threshold. If so, the scaling parameter is reduced according to the preset step size. ; Then, use the new scaling parameters Reconstruct the pulse Gaussian linear error unit and obtain a new pulse neural network until the pulse sparsity The requirements are met and the final spiking neural network is obtained.
[0144] For example, the total time step of network simulation is 50 steps, and the initial setting is = 5, calculate the average I of all time steps sparsity , and evaluate it on one-tenth of the training set, taking the average value calculated for each sample. <0.3, then continue to decrease in steps of 0.1 , so as to ensure that the sparseness of the network is not too low. If I sparsity >0.3, no additional adjustment is required.
[0145] Since the network has time domain characteristics, it is necessary to consider the input and output calculations of each step. In this invention, the input of the network remains unchanged at each time step. In order to reduce the computational cost, the total number of time steps is taken as 50 steps. The final output result is Take the average of the outputs over all time steps.
[0146] This specific embodiment designs a pulsed Gaussian error linear unit model SGELU with a scaling parameter, and uses the computing unit to convert an interactive segmentation neural network with an inter-block collaboration mechanism into a corresponding pulsed neural network, thereby improving the robustness of the network. At the same time, the network is optimized by combining Bayesian optimization with model pruning, low-rank decomposition and computational accuracy compression technology, thereby improving the network's reasoning efficiency.
[0147] An embodiment of a method for constructing a neural network model for image annotation is described in detail above. Based on the method for constructing a neural network model for image annotation described in the above embodiment, an embodiment of the present invention also provides a system for constructing a neural network model for image annotation corresponding to the method.
[0148] Figure 2 This is a schematic block diagram of the structure of a picture annotation neural network model construction system provided by an embodiment of the present invention. In this embodiment, the picture annotation neural network model construction system can be divided into multiple functional modules according to the functions it performs. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory.
[0149] Original model building module: builds an original image annotation neural network model. The activation function in the original image annotation neural network model uniformly uses Gaussian linear error units.
[0150] Original model training module: train the original image annotation neural network model.
[0151] Spiking neural network conversion module: Use the pulsing Gaussian linear error unit to convert the trained original image annotation neural network model into a pulsing neural network, which is the final constructed image annotation neural network model.
[0152] In an optional embodiment, the system further includes a model compression module: compressing the original image annotation neural network model.
[0153] The model compression module includes a model pruning unit, a low-rank decomposition unit, and a computational accuracy compression unit.
[0154] Model pruning unit: prunes the model, including collecting the attention weight matrix of each attention head during the forward propagation of the model, averaging the attention weight matrix of each attention head to obtain the average attention weight of each attention head, and discarding several attention heads with low average attention weights.
[0155] Low-rank decomposition unit: performs low-rank decomposition on the convolutional layers in the image encoder and mask decoder, including singular value decomposition of the weight matrix of each target convolutional layer, truncating the calculated singular values, retaining the largest singular values, reconstructing the low-rank matrix using the truncated singular value matrix, and replacing the weight matrix of the original convolutional layer with the reconstructed low-rank matrix.
[0156] Computational precision compression unit: Converts the model's floating-point weights into integer weights, including: (a) Calculate the weight of each floating point number Minimum value of and maximum value , determine the quantization range,
[0157] (b) Calculate the scaling factor based on the range , maps 32-bit floating point numbers to 8-bit integers, and for 8-bit integers, q max and q min 128 and -127 respectively,
[0158] (c) Using scaling factors The floating point weight Convert to integer weights ,
[0159] (d) Integer weights for storage.
[0160] The image labeling neural network model construction system of this embodiment is used to implement the aforementioned image labeling neural network model construction method. Therefore, the specific implementation method of the system can be seen in the embodiment part of the image labeling neural network model construction method in the previous text. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part, which will not be introduced in detail here.
[0161] In addition, since the picture labeling neural network model construction system of this embodiment is used to implement the aforementioned picture labeling neural network model construction method, its function corresponds to that of the aforementioned method and will not be repeated here.
[0162] The above disclosure is only a preferred embodiment of the present invention, but the present invention is not limited thereto. Any non-creative changes that can be thought of by a person skilled in the art, as well as several improvements and modifications made without departing from the principle of the present invention, should fall within the protection scope of the present invention.
Claims
1. A method for constructing a neural network model for image annotation. It is characterized in that The following steps are involved: Construct an original image annotation neural network model, in which the activation function uniformly uses Gaussian linear error unit; Train the neural network model for original image annotation; The trained original image annotation neural network model is converted into a spiking neural network using a spiking Gaussian linear error unit, which is the final constructed image annotation neural network model.
2. The method for constructing a neural network model for image annotation according to claim 1, It is characterized in that The trained original image annotation neural network model is converted into a spiking neural network using a spiking Gaussian linear error unit, including: Construct a pulsed Gaussian linear error unit; The Gaussian linear error unit of the original image annotation neural network model is replaced with the constructed pulse Gaussian linear error unit to obtain a pulse neural network.
3. The method for constructing a neural network model for image annotation according to claim 2, It is characterized in that The pulsed Gaussian linear error unit is a pulsed Gaussian linear error unit with a scaling parameter; Construct a pulse Gaussian linear error unit, specifically including: Configuring Scaling Parameters ; The energy accumulation of the pulse Gaussian linear error unit is calculated by the following formula, In the formula, represents the energy accumulated by the pulse Gaussian linear error unit at the nth time step, represents the energy accumulated by the pulse Gaussian linear error unit at the previous time step, represents the input of the nth time step, express Cumulative distribution function of the standard normal distribution; The pulse value is obtained by the following formula based on energy accumulation , which means if the energy is positive, it is rounded down, otherwise it is rounded up. According to the scaling parameters The pulse value is calculated by the following formula Scaling to get the output value of the nth time step , The energy attenuation of the pulsed Gaussian linear error unit is calculated by the following formula, 。 4. The method for constructing a neural network model for image annotation according to claim 3, It is characterized in that The trained original image annotation neural network model is converted to a spiking neural network using a spiking Gaussian linear error unit, and also includes: The pulse sparsity of the spiking neural network at the nth time step is calculated by the following formula , In the formula, Represents the output mask of the spiking neural network at the nth time step , Represents the output mask of the neural network at the n-1th time step , Indicates the output mask The number of computational units after expansion; Calculate the average of the spike sparsity over all time steps ; Determining pulse sparsity Is it greater than the threshold? If so, reduce the scaling parameter by the preset step size ; Otherwise, no adjustment is made; Use the new scaling parameters Reconstruct the pulse Gaussian linear error unit and obtain a new pulse neural network until the pulse sparsity The requirements are met and the final spiking neural network is obtained.
5. The method for constructing a neural network model for image annotation according to any one of claims 2 to 4, It is characterized in that The input of the spiking neural network remains unchanged at each time step, and its final output is the average of the output masks of all time steps.
6. The method for constructing a neural network model for image annotation according to any one of claims 1 to 4, It is characterized in that The original image annotation neural network model is trained, including: Use Gaussian process as a surrogate model to approximate the objective function, and use the surrogate model to guide the next sampling to gradually approach the optimal solution; Adopting the expected improvement strategy, minimizing the loss function is used as maximizing the expected improvement; After each sampling, the value of the loss function is calculated according to the new sampling point, and the proxy model is updated; Hyperparameter tuning is performed using the Bayesian optimization algorithm. In each iteration, the Bayesian optimizer selects the next hyperparameter combination for evaluation based on the current surrogate model and the expected improvement strategy, and updates the surrogate model until the optimal hyperparameter combination is found.
7. The method for constructing a neural network model for image annotation according to any one of claims 1 to 4, It is characterized in that The original image annotation neural network model is a Transformer model, which includes an image encoder, a hint encoder, and a mask decoder; The image encoder is encoded through self-attention mechanism, multi-head attention mechanism, and feedforward neural network respectively; The prompt encoder accepts point prompt information, box prompt information and text prompt information input by the user; The mask decoder consists of deconvolution layers, upsampling layers, and convolution layers; The original image annotation neural network model uses a combination of binary cross entropy loss and boundary loss to construct the loss function; During the training process of the original image annotation neural network model, a coordination mechanism between modules is added, including adding parameter-learnable coordination matrices to the attention heads in the image encoder, the feedforward neural network, and each convolutional layer in the mask decoder.
8. The method for constructing a neural network model for image annotation according to claim 7, It is characterized in that After the original image annotation neural network model is trained, the step of compressing the trained image annotation neural network model is also included, specifically including: (1) Pruning the model: During the forward propagation of the model, collect the attention weight matrix of each attention head, average the attention weight matrix of each attention head, obtain the average attention weight of each attention head, and discard several attention heads with low average attention weight; (2) Perform low-rank decomposition on the convolutional layers in the image encoder and mask decoder: Perform singular value decomposition on the weight matrix of each target convolutional layer, truncate the calculated singular values, retain the largest singular values, reconstruct the low-rank matrix using the truncated singular value matrix, and replace the weight matrix of the original convolutional layer with the reconstructed low-rank matrix; (3) Convert the model's floating-point weights to integer weights: (a) Calculate the weight of each floating point number Minimum value of and maximum value , determine the quantization range, (b) Calculate the scaling factor based on the range , maps 32-bit floating point numbers to 8-bit integers, and for 8-bit integers, q max and q min 128 and -127 respectively, (c) Using scaling factors The floating point weight Convert to integer weights , (d) Integer weights for storage.
9. A neural network model construction system for image annotation, It is characterized in that include, Original model building module: builds an original image annotation neural network model. The activation function in the original image annotation neural network model uniformly uses Gaussian linear error units; Original model training module: train the original image annotation neural network model; Spiking neural network conversion module: Use the pulsing Gaussian linear error unit to convert the trained original image annotation neural network model into a pulsing neural network, which is the final constructed image annotation neural network model.
10. The image annotation neural network model construction system according to claim 9, It is characterized in that The system also includes a model compression module: compressing the original image annotation neural network model; The model compression module includes: Model pruning unit: prunes the model, including collecting the attention weight matrix of each attention head during the forward propagation of the model, averaging the attention weight matrix of each attention head to obtain the average attention weight of each attention head, and discarding several attention heads with low average attention weight; Low-rank decomposition unit: performs low-rank decomposition on the convolutional layers in the image encoder and mask decoder, including performing singular value decomposition on the weight matrix of each target convolutional layer, truncating the calculated singular values, retaining the largest singular values, reconstructing the low-rank matrix using the truncated singular value matrix, and replacing the weight matrix of the original convolutional layer with the reconstructed low-rank matrix; Computational precision compression unit: Converts the model's floating-point weights into integer weights, including: (a) Calculate the weight of each floating point number Minimum value of and maximum value , determine the quantization range, (b) Calculate the scaling factor based on the range , maps 32-bit floating point numbers to 8-bit integers, and for 8-bit integers, q max and q min 128 and -127 respectively, (c) Using scaling factors The floating point weight Convert to integer weights , (d) Integer weights for storage.
Citation Information
Patent Citations
Training method of convolutional pulse neural network based on reweighted membrane voltage
CN112633497A
Fine-grained image classification method based on segmentation mask and self-attention neural network
CN114119979A
Spiking neural network-based short-range tracking method and system
WO2021012752A1
Cited By
Visual image classification method based on pulse Transform
CN119942244A
