Data processing method and apparatus, device, and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA MOBILE COMM LTD RES INST
- Filing Date
- 2022-01-06
- Publication Date
- 2026-06-05
Smart Images
Figure CN116467585B_ABST
Abstract
Description
Technical Field
[0001] This application relates to artificial intelligence technology, including but not limited to data processing methods, apparatus, devices, and storage media. Background Technology
[0002] Convolutional Neural Networks (CNNs) are a class of feedforward neural networks (FNNs) that incorporate convolutional computations and have a deep structure. They are one of the representative algorithms of Deep Learning (DL). CNNs are widely used in anomaly detection applications. Therefore, improving the real-time performance of anomaly detection algorithms based on CNNs is of significant importance for anomaly detection applications. Summary of the Invention
[0003] In view of this, the data processing method, apparatus, equipment, and storage medium provided in this application can greatly reduce the amount of multiplication calculations required for convolution operations in convolutional neural networks, thereby effectively improving the real-time performance of data detection.
[0004] According to one aspect of the embodiments of this application, a data processing method is provided, comprising: performing inference on data to be identified through a trained target convolutional neural network to obtain a first inference result; wherein the convolution kernel of the target convolutional neural network includes a hyperparameter matrix and a target scale value; the absolute value of the elements in the hyperparameter matrix is 1, and the hyperparameter matrix is used to perform bit operations on the elements of the corresponding position region of the data to be identified and then fuse them to obtain a first convolution result; the target scale value is used to fuse with the first convolution result to obtain a second convolution result; and anomaly determination is performed based on the first inference result.
[0005] Since the absolute values of all elements in the hyperparameter matrix are 1, the first convolution result can be obtained by fusing them after bitwise operations, which greatly reduces the amount of multiplication required for convolution operations and thus improves the implementation efficiency of the data processing method.
[0006] According to one aspect of the embodiments of this application, a data processing apparatus is provided, comprising: an inference module, configured to infer from data to be identified using a trained target convolutional neural network to obtain a first inference result; wherein the convolution kernel of the target convolutional neural network includes a hyperparameter matrix and a target scale value; the absolute value of the elements in the hyperparameter matrix is 1, and the hyperparameter matrix is used to perform bitwise operations on the elements of the corresponding position region of the data to be identified and then fuse them to obtain a first convolution result; the target scale value is used to fuse with the first convolution result to obtain a second convolution result; and a determination module, configured to determine anomalies based on the first inference result.
[0007] According to one aspect of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor executes the program to implement the method described in the embodiments of the present application.
[0008] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the methods provided in the embodiments of this application.
[0009] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.
[0011] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.
[0012] Figure 1 A schematic diagram illustrating the implementation flow of the data processing method provided in the embodiments of this application;
[0013] Figure 2 This is a schematic diagram of the convolution operation process provided in an embodiment of this application;
[0014] Figure 3 A schematic diagram illustrating the implementation flow of the method for obtaining the target convolutional neural network provided in this application embodiment;
[0015] Figure 4 This is a schematic diagram illustrating the implementation process of the method for determining the first convolutional neural network to be trained, as provided in an embodiment of this application.
[0016] Figure 5 A schematic diagram illustrating the implementation process of the method for selecting alternative models according to waterfall rules provided in this application embodiment;
[0017] Figure 6 This is a schematic diagram of the security detection architecture provided in an embodiment of this application;
[0018] Figure 7 This is a schematic diagram illustrating another method for selecting alternative models according to a waterfall rule, provided in an embodiment of this application.
[0019] Figure 8 A schematic diagram of the composition structure of a convolutional neural network provided in an embodiment of this application;
[0020] Figure 9 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;
[0021] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.
[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0024] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0025] This application provides a data processing method applied to an electronic device. This electronic device can be of various types with information processing capabilities, such as end-side devices in a security detection system (e.g., surveillance cameras, smart doorbells), servers, mobile phones, laptops, personal computers, etc. The functions implemented by this method can be achieved by a processor in the electronic device calling program code. Of course, the program code can be stored in a computer storage medium. Therefore, the electronic device includes at least a processor and a storage medium.
[0026] Figure 1 This is a schematic diagram illustrating the implementation flow of the data processing method provided in the embodiments of this application, as shown below. Figure 1 As shown, the method may include the following steps 101 to 102:
[0027] Step 101: Inferring the data to be identified using a trained target convolutional neural network to obtain a first inference result; wherein, the convolution kernel of the target convolutional neural network includes a hyperparameter matrix and a target scale value; the absolute value of the elements in the hyperparameter matrix is 1, and the hyperparameter matrix is used to perform bit operations on the elements of the corresponding position region of the data to be identified and then fuse them to obtain the first convolution result; the target scale value is used to fuse with the first convolution result to obtain a second convolution result.
[0028] In this application, there are no restrictions on the type of data to be identified; it can be various types of data, such as image data, text data, or speech data. Of course, for the text data and speech data to be identified, they need to be converted into two-dimensional matrices before being input into the target convolutional neural network.
[0029] In this application, the method of fusing bitwise operation results is not limited. For example, the hyperparameter matrix is used to perform bitwise operations on the elements of the corresponding positional region of the data to be identified, and then the average value is taken to obtain the first convolution result. Alternatively, the sum of the bitwise operation results can be used as the first convolution result, and so on.
[0030] There are no restrictions on how the target scale value is fused with the first convolution result. For example, the product of the target scale value and the first convolution result can be used as the second convolution result.
[0031] Step 102: Determine an anomaly based on the first reasoning result.
[0032] In this application, the application scenarios of the data processing method are not limited, and it can be applied to a variety of intelligent security scenarios or other anomaly detection scenarios.
[0033] For example, in smart home scenarios, this method can be applied to smart doorbell cameras, enabling them to identify whether people in captured images are strangers through a target convolutional neural network, thus making anomaly detection. Similarly, in industrial parks where cameras are installed on walls, applying this method to these cameras allows them to identify whether anyone is climbing over walls in captured images, thus making anomaly detection. Furthermore, on construction sites with high-risk operations, cameras are installed at entrances or other locations; applying this method to these cameras allows them to identify which people in captured images are not wearing safety helmets, thus making anomaly detection.
[0034] In this embodiment, the target convolutional neural network used includes a hyperparameter matrix and a target scale value in its convolution kernel. The absolute value of each element in the hyperparameter matrix is 1. It is used to perform bitwise operations on the elements of the corresponding position region of the data to be identified and then fuse them to obtain a first convolution result. The target scale value is used to fuse the first convolution result to obtain a second convolution result. Thus, since the absolute value of each element in the hyperparameter matrix is 1, the first convolution result can be obtained by fusing after bitwise operations, thereby greatly reducing the amount of multiplication calculation required for convolution operations and improving the implementation efficiency of the data processing method.
[0035] Understandably, the convolutional kernels in the target convolutional neural network and the convolutional kernels in common convolutional neural networks have the same function: feature extraction. The only difference is their structure. For example, as shown in equation (1), the structure of the convolutional kernel in a common convolutional neural network is a W matrix, while the structure of the convolutional kernel in the target convolutional neural network is a B matrix multiplied by the scale parameter α.
[0036]
[0037] Assuming W is the convolution kernel and I is the input tensor, the convolution calculation can be expressed as: in This represents convolution calculation. Therefore, in this embodiment, the common convolution calculation is performed by... Replace with and This can be achieved by using bitwise operations and fusing the results of bitwise operations, without having to multiply each element in B with its corresponding element in I before fusing them; thus, the amount of multiplication computation required for convolution operations is greatly reduced.
[0038] For example, Figure 2 As shown, assuming the input tensor is 201, the convolution kernel includes a hyperparameter matrix 202 and a scale parameter α (assumed to be 1.1), the convolution stride is 1, and the bitwise operation-based fusion algorithm is summation, then the calculation process is as follows: Figure 2 As shown, the input tensor 201 is convolved by the hyperparameter matrix 202 (first bitwise operations and then summation) to obtain the feature map 203; each element in the feature map 203 is multiplied by the scale parameter α = 1.1 to obtain the final feature map 204. Compared with using the full-precision convolution kernel W as shown in Equation (1), the number of multiplication operations is reduced by 72.
[0039] This application embodiment further provides a method for obtaining a target convolutional neural network. Figure 3 This is a schematic diagram illustrating the implementation flow of the method for obtaining the target convolutional neural network provided in the embodiments of this application, as shown below. Figure 3 As shown, the method may include the following steps 301 to 303:
[0040] Step 301: Determine the first convolutional neural network to be trained; wherein the convolution kernel in the first convolutional neural network is a full-precision convolution kernel.
[0041] Understandably, a full-precision convolution kernel refers to the original convolution kernel that has not been decomposed, and the absolute values of the elements in its matrix are not all 1. As shown in equation (1) above, the matrix W on the left is the full-precision convolution kernel, while the part on the right is the convolution kernel obtained by decomposing the full-precision convolution kernel, and the absolute values of the elements in the hyperparameter matrix are all 1.
[0042] Step 302: Based on the sign of the elements in the full-precision convolution kernel of the first convolutional neural network to be trained, the full-precision convolution kernel is decomposed into the hyperparameter matrix and the initial scale value to obtain the second convolutional neural network.
[0043] In some embodiments, before decomposing the full-precision convolutional kernels of the first convolutional neural network to be trained, the method further includes: performing lightweight processing on the architecture of the first convolutional neural network to be trained according to the performance index requirements of the application device of the data processing method, thereby obtaining a new first convolutional neural network, and then performing step 302 based on this.
[0044] For example, the application device of the data processing method requires one of the model's metrics to be a model size of less than or equal to 8Mb. However, if the size of the first convolutional neural network to be trained is greater than 8Mb, then the first convolutional neural network to be trained needs to be further designed to be lightweight so that it meets the metric requirement.
[0045] Step 303: Using the sample dataset, train the initial scale value and other weight parameters (excluding the hyperparameter matrix) in the second convolutional neural network to obtain the target convolutional neural network; wherein, the hyperparameter matrix is used to perform bit operations on the elements of the corresponding position region of the sample data and then fuse them to obtain the third convolution result; the initial scale value is used to fuse with the third convolution result to obtain the fourth convolution result.
[0046] In this embodiment, the full-precision convolutional kernel of the first convolutional neural network to be trained is first decomposed into a hyperparameter matrix and an initial scale value to obtain a second convolutional neural network. Then, the initial scale value and other weight parameters (excluding the hyperparameter matrix) in the second convolutional neural network are trained using a sample dataset. Thus, the amount of multiplication computation is greatly reduced in the forward propagation operation. In the backpropagation operation, since the hyperparameter matrix does not participate in model training, the backpropagation does not need to calculate the gradient of each element of the hyperparameter matrix, thus greatly reducing the amount of multiplication computation. In summary, the above method can greatly reduce the amount of computation in the model training process, allowing the training process to be deployed on the edge. The edge can then use a sample dataset suitable for its own scenario to train the second convolutional neural network, thereby obtaining a personalized and customized target convolutional neural network that suits its own scenario, and improving inference accuracy.
[0047] The following sections will describe further optional implementation methods for each of the above steps, as well as related terms.
[0048] In step 301, a first convolutional neural network to be trained is determined; wherein the convolutional kernel in the first convolutional neural network is a full-precision convolutional kernel.
[0049] In this application, the method for determining the first convolutional neural network to be trained is not limited. In some embodiments, step 301 can be implemented by the following steps 3011 and 3012; in other embodiments, it can also be implemented by the following step 3014.
[0050] Among them, steps 3011 and 3012 are as follows Figure 4 As shown:
[0051] Step 3011: Select candidate models from multiple pre-configured convolutional neural networks based on various constraints imposed on the convolutional neural network by the target application scenario; wherein the convolutional kernels in the pre-configured convolutional neural network are full-precision convolutional kernels.
[0052] Understandably, different constraints can be pre-configured for different target application scenarios. The constraints configured for different target application scenarios may also be different; thus, since the first convolutional neural network is selected based on various constraints imposed on the convolutional neural network according to the target application scenario, the final selected first convolutional neural network can meet the customized requirements of the target application scenario.
[0053] It should be noted that the various restrictions mentioned are pre-configured conditions, and developers can customize them according to the indicator requirements of the application scenario.
[0054] Let L1, L2, ..., L KThe scenario imposes K types of constraints on the algorithm, for example:
[0055] L1: The model's single-inference accuracy is greater than or equal to 90%.
[0056] L2: The model's single inference time is less than or equal to 30ms.
[0057] ...
[0058] L K Model size is less than or equal to 10Mb.
[0059] Furthermore, in some embodiments, such as Figure 5 As shown, step 3011 can be achieved through steps 501 to 504, namely, selecting candidate models according to the waterfall rule:
[0060] Step 501: Determine the i-th pre-configured convolutional neural network (denoted as M). i ) Does the first priority constraint condition meet? If yes, proceed to step 502; otherwise, proceed to step 504; where i is greater than 0 and less than or equal to the number of pre-configured convolutional neural networks.
[0061] Understandably, if model M i If the first priority constraint is not met, the model is discarded, and the next model (i.e., the (i+1)th pre-configured convolutional neural network) is determined according to the waterfall rule to determine whether the first priority constraint is met.
[0062] Step 502, determine M i Does the second priority constraint condition meet? If yes, proceed to step 503; otherwise, proceed to step 504. Wherein, the first priority is higher than the second priority.
[0063] Similarly, if model M i If the second priority constraint is not met, the model is discarded, and the waterfall rule is continued to determine whether the next model (i.e., the (i+1)th pre-configured convolutional neural network) meets the first priority constraint.
[0064] Understandably, if M i If the second priority constraint is met and there is no next constraint, then model M is... i As one of the alternative models, the method is continued to determine the (i+1)th pre-configured convolutional neural network that satisfies the first priority constraint condition;
[0065] If M i If the second priority constraint is not met, the model is discarded. Of course, if M... iIf the second priority constraint is met, and there is also a lower priority constraint, then further judgment is required.
[0066] Step 503, determine M i Does it meet the restrictions of the third priority? The third priority is lower than the second priority.
[0067] Understandably, if M i If the third priority constraint is met and there is no next constraint, then model M is... i As one of the alternative models, the method is continued to determine the (i+1)th pre-configured convolutional neural network that satisfies the first priority constraint condition;
[0068] If M i If the third priority constraint is not met, the model is discarded. Of course, if M... i If the third priority constraint is met, and there is also a lower priority constraint, then further judgment is required.
[0069] Step 504: Determine the (i+1)th pre-configured convolutional neural network (denoted as M). i+1 Whether the first priority constraint is met; thus, the constraints are judged in descending order of priority to select the pre-configured convolutional neural network that meets each constraint as the candidate model.
[0070] Understandably, selecting candidate models through this waterfall-style rule method can not only select candidate models with better performance, but also improve the screening efficiency; that is, it can select better-performing models more efficiently, thereby improving the efficiency of obtaining the target convolutional neural network.
[0071] Step 3012: Determine the first convolutional neural network based on the candidate models.
[0072] Understandably, the candidate model selected in step 3011 may be one or more. Therefore, any one of the candidate models can be used as the first convolutional neural network, or a model that meets the conditions can be further selected as the first convolutional neural network.
[0073] For example, in some embodiments, step 3012 can be implemented as follows: if the number of candidate models is 1, the candidate model is used as the first convolutional neural network; if the number of candidate models is greater than 1, the model whose inference accuracy satisfies the first condition is selected from the selected candidate models and used as the first convolutional neural network. This can further improve the inference accuracy of the target convolutional neural network.
[0074] In other embodiments, determining the first convolutional neural network to be trained in step 301 can also be achieved through the following step 3014 (not shown in the figures):
[0075] Step 3014: Based on the predefined loss function and the sample dataset, select the submodule combination whose loss function value satisfies the second condition from multiple different submodule combinations as the first convolutional neural network; wherein, the predefined loss function includes the loss function of the inference result and the loss functions corresponding to various constraints of the target application scenario on the convolutional neural network.
[0076] In other words, the convolutional neural network is divided into n sub-modules, with at least one sub-module corresponding to multiple candidate modules. Various constraints are quantified into corresponding loss functions, allowing the best-performing combination of sub-modules to be selected as the first convolutional neural network based on the sample dataset; this improves the detection accuracy of the first convolutional neural network. For example, the second condition is minimizing the value of the loss function.
[0077] In step 303, the initial scale value and other weight parameters (excluding the hyperparameter matrix) in the second convolutional neural network are trained using the sample dataset to obtain the target convolutional neural network.
[0078] In some embodiments, the second convolutional neural network includes at least one convolutional layer and a convolution result processing unit, wherein the convolutional layer includes at least one convolutional kernel having an initial scale value and a hyperparameter matrix; step 303 (not shown in the figures) can be implemented by the following steps 3031 to 3035:
[0079] Step 3031: Perform a convolution operation on the input tensor using the hyperparameter matrix of the first convolution kernel in the convolutional layer to obtain the first feature map;
[0080] Step 3032: Fuse each element of the first feature map with the initial scale value of the first convolution kernel (e.g., multiply them) to obtain a second feature map corresponding to the first convolution kernel.
[0081] Step 3033: If the hyperparameter matrix of the second convolutional kernel in the convolutional layer satisfies the same condition as the hyperparameter matrix of the first convolutional kernel, the first feature map is reused, and each element of the first feature map is fused with the initial scale value of the second convolutional kernel to obtain a third feature map corresponding to the second convolutional kernel.
[0082] In some embodiments, the similarity condition is that the number of corresponding identical elements is greater than or equal to a specific threshold. For example, suppose the hyperparameter matrix of the second convolution kernel is matrix B1 of equation (2) below, the hyperparameter matrix of the first convolution kernel is matrix B2 of equation (3) below, and the specific threshold in the similarity condition is set to 7;
[0083]
[0084]
[0085] It can be seen that the number of identical elements in matrices B1 and B2 is 8, which is greater than the specific threshold of 7. Therefore, these two matrices are similar matrices, and matrices B1 and B2 satisfy the similarity condition.
[0086] Understandably, by using step 3033, which involves reusing the convolution results, the computational cost of addition can be greatly reduced, thereby improving the overall training efficiency.
[0087] Step 3034: The input feature map is processed by the convolution result processing unit to obtain the second inference result;
[0088] Step 3035: Perform backpropagation based on the second inference result to update the initial scale value and other weight parameters in the second convolutional neural network; iterate in this way until the cutoff condition is reached to obtain the target convolutional neural network.
[0089] For example, the cutoff condition is that the number of iterations reaches a threshold; another example is that the value of the loss function tends to converge.
[0090] It should be noted that, in this application, the data processing method and the method for obtaining the target convolutional neural network provided in the above embodiments can be executed by the same entity or by different entities. When they are the same entity, for example, both are implemented on the edge (e.g., a smart doorbell or camera); when they are different entities, for example, the data processing method is implemented on the edge, and the method for obtaining the target convolutional neural network is implemented on a cloud platform.
[0091] The following describes an exemplary application of the embodiments of this application in a real-world application scenario.
[0092] This application provides a customized edge-side home intelligent security detection system that can perform real-time and efficient detection using video detection data without relying on multiple sensor devices, and has advantages such as convenient deployment. The customized edge-side deployment can meet the anomaly detection needs of different scenarios, so the system has strong generalization ability. Combined with an efficient detection model, it balances detection speed and accuracy, thereby improving security detection efficiency.
[0093] The customized home smart security detection system provided in this application embodiment has the following architecture: Figure 6 As shown, it includes: a device-side data acquisition module 601, a data storage module 602, a device-side model customization module 603, a device-side model training module 604, a device-side model deployment module 605, a device-side model inference module 606, and an anomaly detection module 607; wherein,
[0094] The specific implementation steps are as follows:
[0095] Step 1. Acquire training data (i.e., an example of a sample dataset, such as an image dataset) for the current application scenario through the edge data acquisition module 601 and store it in the data storage module 602; after the training data is labeled, select the most suitable edge model training hyperparameters, i.e. the first convolutional neural network to be trained, through the edge model customization module 603 and various constraints of the application scenario on the algorithm.
[0096] Step 3. The edge model training module 604 designs a lightweight anomaly detection model (i.e., a new first convolutional neural network) based on the model training hyperparameters obtained in Step 1, and combines it with the edge training acceleration architecture to realize the training of the customized lightweight model on the edge device.
[0097] Step 3. The edge model deployment module 605 implements the edge deployment of the lightweight model, and the edge model inference module 606 implements the edge rapid inference of the test data, and feeds the inference results back to the anomaly judgment module, and finally realizes the anomaly judgment of the application scenario.
[0098] To provide a more detailed description of the above steps, the main functions of the steps are further subdivided below, and a specific embodiment is provided.
[0099] The manual and automatic edge-side customization schemes provided in this application embodiment can effectively realize the process of edge-side model customization module 603 selecting the most suitable model training hyperparameters in step 1. The specific selection can be determined based on the actual situation. The functions of the manual edge-side customization scheme are detailed as described in steps 1.1 to 1.3 below:
[0100] Step 1.11: Based on human experience, initially determine the pool of full-precision neural network models to be screened (i.e., multiple pre-configured convolutional neural networks), denoted as M1, M2, ...;
[0101] Step 1.12, let the various constraints of the anomaly detection scenario on the model be L1, L2, ..., first select candidate models from the model pool that meet the various constraints and have an accuracy greater than the threshold;
[0102] Step 1.13: If the number of candidate models is greater than 1, the optimal model is further selected based on the topological characteristics of the models and used as hyperparameters for subsequent model training.
[0103] The detailed functions of the automated end-side customization solution are as follows:
[0104] Step 1.21: Divide the neural network into multiple sub-modules, with each sub-module having multiple alternatives for model training;
[0105] Step 1.22: Quantify various constraints in the anomaly detection scenario into loss functions during model training, and select the best-performing sub-module combination based on the training dataset for use as hyperparameters in lightweight model training.
[0106] The lightweight neural network technology and edge-side algorithm training architecture provided in this application can effectively realize the design and training of the edge-side lightweight anomaly detection model in step 3. The specific functions are detailed as follows:
[0107] Step 3.1: Combining the model structure and training model hyperparameters obtained in Step 1, select a suitable combination of lightweight model strategies for edge devices to achieve the final edge model architecture design.
[0108] Step 3.2: Train the lightweight model on the edge device by fixing and reusing model hyperparameters.
[0109] The edge computing acceleration framework provided in this application embodiment can effectively realize edge model inference in step 3. Specifically, it greatly reduces the amount of computation during edge model inference by reusing model parameters, reusing convolution results, and using low-precision calculations.
[0110] In some embodiments,
[0111] (I) Manual end-side customization solution in step 1
[0112] For specific anomaly detection scenarios, corresponding full-precision neural network models can be pre-trained, denoted as M1, M2, ..., and L1, L2, ... can be denoted as K types of constraints imposed by the scenario on the algorithm, for example:
[0113] L1: Model size less than or equal to 10Mb
[0114] L2: The model's single inference time is less than or equal to 30ms.
[0115] ...
[0116] The subscripts reflect the priority of each constraint; the smaller the subscript, the higher the priority. This can be based on... Figure 7 The waterfall-style rule selection of alternative models is shown, such as Figure 7 As shown, first determine the full-precision model M.i Does it satisfy constraint L1? If not, do not retain this model and continue to evaluate the next model M. i+1 If satisfied, then determine model M. i Does it meet constraint L2? If not, do not retain this model and continue to evaluate the next model M. i+1 If satisfied, continue to evaluate model M. i Whether constraint L3 is satisfied, etc.; ultimately, if model M i If all constraints are met, the model is retained.
[0117] Furthermore, when multiple candidate models exist, the model with the most robust topological structure can be selected as the training hyperparameter by extracting the topological properties of the activation values of each node output of the candidate models.
[0118] (II) The automated end-side customization solution in step 1
[0119] like Figure 8 As shown, the neural network is divided into n parts, each of which corresponds to multiple candidate modules for model training. For various constraints in the scenario, L1, L2, ..., these constraints can be quantified and incorporated into the loss function of the final neural network. The total loss is shown in the following formula (4):
[0120] Total loss = loss0 + loss1 + loss2 + ... (4);
[0121] Here, loss0 represents the loss quantified from the task objective in the application scenario. i This refers to loss functions quantified for various constraints. For example... Here num(block) i The parameter ) represents the number of parameters in the i-th part of the neural network. Automatic model selection is achieved by optimizing the total loss. The inventors verified on a public dataset that the model obtained using this method showed a significant improvement in prediction accuracy, effectively demonstrating the feasibility of the customized deployment scheme.
[0122] (III) Lightweight Neural Network Techniques in Step 3
[0123] As shown in equation (5) below, W on the left is a full-precision model convolution kernel example; B on the right is the model hyperparameter obtained in step one. The precision of the hyperparameter can be dynamically adjusted according to different scenarios. In the figure above, the hyperparameter B is simplified to 1 bit. α is the scale parameter, and its precision can also be simplified and compressed according to the needs of the scenario. At the same time, different convolution kernels can be matched with different precision α. After adopting the above lightweight design, the size of the model can be compressed to at least one-tenth of the original size.
[0124]
[0125] (iv) Edge-side algorithm training architecture in step 3
[0126] For a given set of input and output channels with numbers C respectively... in C out A full-precision convolutional layer with kernel size s×s and output size w×h requires a total of C calculations during backpropagation. in ×C out The gradient of the ×s×s parameters is of size w×h. In this embodiment, the parameters of the convolution kernel in the model are decomposed into B and α, and B is fixed and not involved in model training. In this way, the backpropagation of the edge algorithm only needs to consider C during training. in ×C out The gradient of each parameter is calculated, and the computational cost of backpropagation is reduced to the original s. 2 One-third.
[0127] (v) The end-side computing acceleration framework in step 3
[0128] Let W be the convolution kernel weights and I be the input tensor, then the convolution calculation can be written as... in This represents convolution calculation. In the embodiments of this application, the general convolution calculation is performed by... Replace with
[0129] Because B contains only low-precision parameters, and the order of parameter α calculation has been changed, in the calculation... In this application, the convolutional computation architecture provided in the embodiments can reuse already computed convolutions to assist in the computation of new convolutions. Combined with data experiments, it is shown that when the number of output channels of the convolutional layer is sufficiently large, the amount of multiplication computation required can be reduced by convolution decomposition. 2 The number of addition calculations required for the calculation is reduced to nearly s. 2 / 2 times; where s is the kernel size.
[0130] In this application embodiment, a customized home intelligent security detection system for the edge is provided, which has the following key technical points: 1) The customized deployment scheme for the edge can be combined with the customized scenario to realize algorithm design; 2) The lightweight neural network technology aims to sink the algorithm in the cloud platform to the edge device; 3) The edge algorithm training architecture combines the lightweight technology and computing acceleration method proposed above to realize the edge training of the algorithm; 4) The lightweight neural network model builds the edge computing acceleration framework for edge algorithm computing acceleration.
[0131] In the embodiments of this application, the provided edge-side customized home smart security detection system can solve the following problems: 1) It brings deep learning technology for smart security applications down from the cloud to the terminal side; 2) It does not rely on multiple sensors, only image data collected by cameras, making deployment convenient; 3) Lightweight model technology, edge-side inference acceleration technology, and edge-side model training architecture make edge-side algorithms more flexible and efficient in deployment; 4) Edge-side customized deployment can meet the anomaly detection needs of different scenarios, and the system has strong generalization ability.
[0132] It should be noted that although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps; or steps from different embodiments may be combined into a new technical solution.
[0133] Based on the foregoing embodiments, this application provides a data processing device, which includes various modules and units included in each module. It can be implemented by a processor or by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA), etc.
[0134] Figure 9 This is a schematic diagram of the structure of the data processing device according to an embodiment of this application, as shown below. Figure 9 As shown, the device 900 includes:
[0135] The inference module 901 is used to infer the data to be identified using a trained target convolutional neural network to obtain a first inference result; wherein, the convolution kernel of the target convolutional neural network includes a hyperparameter matrix and a target scale value; the absolute value of the elements in the hyperparameter matrix is 1, and the hyperparameter matrix is used to perform bitwise operations on the elements of the corresponding position region of the data to be identified and then fuse them to obtain the first convolution result; the target scale value is used to fuse with the first convolution result to obtain a second convolution result;
[0136] The determination module 902 is used to determine anomalies based on the first reasoning result.
[0137] In some embodiments, the apparatus 900 further includes a determining module, a decomposition module, and a training module; wherein, the determining module is used to determine a first convolutional neural network to be trained; wherein the convolutional kernel in the first convolutional neural network is a full-precision convolutional kernel; the decomposition module is used to decompose the full-precision convolutional kernel of the first convolutional neural network into the hyperparameter matrix and an initial scale value according to the positive and negative status of the elements in the full-precision convolutional kernel of the first convolutional neural network, to obtain a second convolutional neural network; the training module is used to train the second convolutional neural network using the initial scale value and other weight parameters excluding the hyperparameter matrix using a sample dataset, to obtain the target convolutional neural network; wherein, the hyperparameter matrix is used to perform bitwise operations on the elements of the corresponding position region of the sample data and then fuse them to obtain a third convolution result; the initial scale value is used to fuse with the third convolution result to obtain a fourth convolution result.
[0138] In some embodiments, the second convolutional neural network includes at least one convolutional layer and a convolution result processing unit. The convolutional layer includes at least one convolutional kernel having an initial scale value and a hyperparameter matrix. The training module includes a convolutional unit, a fusion unit, a convolution result processing unit, and a backpropagation unit. The convolutional unit is used to perform a convolution operation on the input tensor using the hyperparameter matrix of the first convolutional kernel in the convolutional layer to obtain a first feature map. The fusion unit is used to fuse each element of the first feature map with the initial scale value of the first convolutional kernel to obtain a second feature map corresponding to the first convolutional kernel, and to process the convolution result in the convolutional layer. If the hyperparameter matrix of the second convolutional kernel in the multiplicative layer satisfies the same condition as the hyperparameter matrix of the first convolutional kernel, the first feature map is reused, and each element of the first feature map is fused with the initial scale value of the second convolutional kernel to obtain a third feature map corresponding to the second convolutional kernel. The convolution result processing unit is used to process the input feature map to obtain a second inference result. The backpropagation unit is used to perform backpropagation based on the second inference result to update the initial scale value and other weight parameters in the second convolutional neural network. This process is iterated until the cutoff condition is reached to obtain the target convolutional neural network.
[0139] In some embodiments, the determining module includes a filtering unit and a determining unit; wherein the filtering unit is used to select candidate models from multiple pre-configured convolutional neural networks according to various constraints on the convolutional neural network in the target application scenario; wherein the convolution kernel in the pre-configured convolutional neural network is a full-precision convolutional kernel; the determining unit is used to determine the first convolutional neural network according to the candidate models.
[0140] In some embodiments, the determining unit is configured to: if the number of candidate models is greater than 1, select a model whose inference accuracy satisfies a first condition from the selected candidate models as the first convolutional neural network; if the number of candidate models is 1, use the candidate model as the first convolutional neural network.
[0141] In some embodiments, the filtering unit is configured to: determine whether the i-th pre-configured convolutional neural network satisfies a first priority constraint; wherein i is greater than 0 and less than or equal to the number of pre-configured convolutional neural networks; if the i-th pre-configured convolutional neural network satisfies the first priority constraint, determine whether the i-th pre-configured convolutional neural network satisfies a second priority constraint; wherein the first priority is higher than the second priority; if the i-th pre-configured convolutional neural network does not satisfy the second priority constraint, determine whether the (i+1)-th pre-configured convolutional neural network satisfies the first priority constraint; and so on, the judgments are made sequentially according to the priority of each constraint from high to low, thereby selecting the pre-configured convolutional neural network that satisfies each constraint as the candidate model.
[0142] In some embodiments, the determining module is configured to select, based on a predefined loss function and the sample dataset, a combination of submodules whose loss function values satisfy a second condition from multiple different submodule combinations as the first convolutional neural network; wherein, the predefined loss function includes a loss function for the inference result and loss functions corresponding to various constraints imposed on the convolutional neural network by the target application scenario.
[0143] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.
[0144] It should be noted that, in the embodiments of this application... Figure 9 The module division of the data processing device shown is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical units, or be integrated into one unit with two or more units. The integrated units can be implemented in hardware, as software functional units, or a combination of both.
[0145] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.
[0146] This application provides an electronic device. Figure 10 This is a schematic diagram of the hardware entity of the electronic device according to an embodiment of this application, such as... Figure 10 As shown, the electronic device 100 includes a memory 1001 and a processor 1002. The memory 1001 stores a computer program that can run on the processor 1002. When the processor 1002 executes the program, it implements the steps in the method provided in the above embodiments.
[0147] It should be noted that the memory 1001 is configured to store instructions and applications executable by the processor 1002, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data and video communication data) in the processor 1002 and various modules in the electronic device 100. It can be implemented by flash memory or random access memory (RAM).
[0148] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method provided in the above embodiments.
[0149] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the method provided in the above-described method embodiments.
[0150] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.
[0151] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.
[0152] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.
[0153] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0154] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.
[0155] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.
[0156] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.
[0157] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.
[0158] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.
[0159] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.
[0160] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.
[0161] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.
[0162] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A data processing method, characterized in that, The method includes: A first convolutional neural network to be trained is determined; wherein the convolutional kernels in the first convolutional neural network are full-precision convolutional kernels; Based on the sign of the elements in the full-precision convolution kernel of the first convolutional neural network, the full-precision convolution kernel is decomposed into a hyperparameter matrix and an initial scale value to obtain a second convolutional neural network; wherein, the hyperparameter matrix is used to perform bitwise operations on the elements of the corresponding position region of the sample data and then fuse them to obtain a third convolution result; the initial scale value is used to fuse with the third convolution result to obtain a fourth convolution result; the second convolutional neural network includes at least one convolutional layer and a convolution result processing unit, wherein the convolutional layer includes at least one convolution kernel with an initial scale value and a hyperparameter matrix; The input tensor is convolved using the hyperparameter matrix of the first convolution kernel in the convolutional layer to obtain the first feature map; Each element of the first feature map is fused with the initial scale value of the first convolution kernel to obtain a second feature map corresponding to the first convolution kernel. If the hyperparameter matrix of the second convolutional kernel in the convolutional layer satisfies the similarity condition with the hyperparameter matrix of the first convolutional kernel, the first feature map is reused, and each element of the first feature map is fused with the initial scale value of the second convolutional kernel to obtain a third feature map corresponding to the second convolutional kernel; the similarity condition is that the number of corresponding identical elements is greater than or equal to a specific threshold. The input feature map is processed by the convolution result processing unit to obtain the second inference result; Backpropagation is performed based on the second inference result to update the initial scale value and other weight parameters (excluding the hyperparameter matrix) in the second convolutional neural network; this process is iterated until the cutoff condition is met to obtain the target convolutional neural network. The trained target convolutional neural network performs inference on the data to be identified to obtain a first inference result; wherein the data to be identified is image data, text data, or speech data; the convolution kernel of the target convolutional neural network includes the hyperparameter matrix and a target scale value; the absolute value of the elements in the hyperparameter matrix is 1, and the hyperparameter matrix is used to perform bitwise operations on the elements of the corresponding position region of the data to be identified and then fuse them to obtain the first convolution result; the target scale value is used to fuse with the first convolution result to obtain a second convolution result; Anomaly determination is made based on the first reasoning result.
2. The method according to claim 1, characterized in that, The determination of the first convolutional neural network to be trained includes: Based on the various constraints imposed on the convolutional neural network by the target application scenario, candidate models are selected from multiple pre-configured convolutional neural networks; wherein, the convolutional kernels in the pre-configured convolutional neural networks are full-precision convolutional kernels; The first convolutional neural network is determined based on the candidate models.
3. The method according to claim 2, characterized in that, The step of determining the first convolutional neural network based on the candidate models includes: If the number of candidate models is greater than 1, the model whose inference accuracy satisfies the first condition is selected from the selected candidate models as the first convolutional neural network. If the number of candidate models is 1, the candidate model is used as the first convolutional neural network.
4. The method according to claim 2, characterized in that, The step of selecting candidate models from multiple pre-configured convolutional neural networks based on various constraints imposed on the convolutional neural network by the target application scenario includes: Determine whether the i-th pre-configured convolutional neural network satisfies the first priority constraint condition; where i is greater than 0 and less than or equal to the number of pre-configured convolutional neural networks; If the i-th pre-configured convolutional neural network satisfies the first priority constraint, determine whether the i-th pre-configured convolutional neural network satisfies the second priority constraint; wherein, the first priority is higher than the second priority; If the i-th pre-configured convolutional neural network does not meet the second priority constraint, determine whether the (i+1)-th pre-configured convolutional neural network meets the first priority constraint. Thus, the various constraints are judged in descending order of priority to select the pre-configured convolutional neural network that satisfies each constraint as the candidate model.
5. The method according to claim 1, characterized in that, The determination of the first convolutional neural network to be trained includes: Based on a predefined loss function and a sample dataset, the combination of sub-modules whose loss function values satisfy the second condition is selected from multiple different sub-module combinations as the first convolutional neural network; The predefined loss function includes the loss function of the inference result and the loss function corresponding to various constraints of the target application scenario on the convolutional neural network.
6. A data processing apparatus, characterized in that, include: A determination module is used to determine the first convolutional neural network to be trained; wherein the convolutional kernels in the first convolutional neural network are full-precision convolutional kernels; A decomposition module is used to decompose the full-precision convolution kernel of the first convolutional neural network into a hyperparameter matrix and an initial scale value based on the sign of the elements in the full-precision convolution kernel, thereby obtaining a second convolutional neural network; wherein, the hyperparameter matrix is used to perform bitwise operations on the elements of the corresponding position region of the sample data and then fuse them to obtain a third convolution result; the initial scale value is used to fuse with the third convolution result to obtain a fourth convolution result; the second convolutional neural network includes at least one convolutional layer and a convolution result processing unit, wherein the convolutional layer includes at least one convolution kernel with an initial scale value and a hyperparameter matrix; A convolutional unit is used to perform a convolution operation on the input tensor through the hyperparameter matrix of the first convolutional kernel in the convolutional layer to obtain a first feature map; A fusion unit is used to fuse each element of the first feature map with the initial scale value of the first convolutional kernel to obtain a second feature map corresponding to the first convolutional kernel; if the hyperparameter matrix of the second convolutional kernel in the convolutional layer satisfies a similarity condition with the hyperparameter matrix of the first convolutional kernel, the first feature map is reused, and each element of the first feature map is fused with the initial scale value of the second convolutional kernel to obtain a third feature map corresponding to the second convolutional kernel; the similarity condition is that the number of corresponding identical elements is greater than or equal to a specific threshold. The convolution result processing unit is used to process the input feature map to obtain a second inference result; A backpropagation unit is used to perform backpropagation based on the second inference result to update the initial scale value and other weight parameters (excluding the hyperparameter matrix) in the second convolutional neural network; this process is iterated until the cutoff condition is met to obtain the target convolutional neural network. An inference module is used to infer from the trained target convolutional neural network onto the data to be identified, to obtain a first inference result; wherein the data to be identified is image data, text data, or speech data; the convolution kernel of the target convolutional neural network includes the hyperparameter matrix and a target scale value; the absolute value of the elements in the hyperparameter matrix is 1, and the hyperparameter matrix is used to perform bitwise operations on the elements of the corresponding position region of the data to be identified and then fuse them to obtain the first convolution result; the target scale value is used to fuse with the first convolution result to obtain a second convolution result; The determination module is used to determine anomalies based on the first reasoning result.
7. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1 to 5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.