A method and system for image data recognition model construction

Abstract

The application provides a method and system for image data recognition model construction. The method comprises: obtaining an image data set comprising true value labels; based on the image data set and a teacher network model, using an FSP method to initialize the weights of each layer of a student network to obtain at least two student network models with the same network structure; based on the image data set comprising true value labels, using a DML distillation method to train each student network model, and when the trained student network models all meet preset conditions, taking the trained student network models as an image data recognition model. Through the method, at least two student network models with the same structure and completed weight initialization of each layer are obtained by transferring learning from a teacher network model, which can avoid problems such as the inability to well initialize a neural network due to insufficient data, and can reduce the training difficulty. The DML distillation method is used to obtain an image data recognition model with guaranteed recognition ability, and the training cycle is short, the speed is fast, the deployment is easy, and the applicability is wide.

Description

A method and system for building image data recognition models Technical Field

[0001] This application relates to the field of image data processing technology, and in particular to a technique for constructing image data recognition models. Background Technology

[0002] With the continuous advancement of urbanization, the installation of cameras at intersections and key or important locations on urban roads to collect video footage for daily urban management, such as road traffic safety management, public security, and fire prevention, has played an irreplaceable role in current urban governance.

[0003] However, typically only the location, identification number, and / or a combination thereof of the cameras are available. The specific content of the video footage captured by these cameras needs to be identified before it can be clearly understood. Furthermore, the identified content can be categorized to achieve differentiated monitoring, prevention, and early warning. For example, for fire prevention purposes, the focus is on the content captured by cameras, primarily from gas stations and oil depots; for public security purposes, the focus is on the content captured by cameras from stations with large numbers of people and high traffic. Because these scenarios are prone to danger, the identified content should be monitored and managed differently according to category. If all this is handled manually by back-end staff, it requires a huge investment of manpower and resources, and staff fatigue can easily lead to errors.

[0004] Currently, image recognition models based on deep neural networks have been applied to urban management. However, these models typically employ deep neural networks with very large depths or widths, containing numerous parameters and requiring the processing of large amounts of data. This not only places high demands on hardware but also results in long processing times. Furthermore, these models are trained on a large number of samples, requiring a large dataset for training. For application scenarios where data samples are typically limited and the deployed hardware's computing and storage capabilities are limited, the training time is long, the training model struggles to converge, and the performance is poor. Therefore, existing image recognition models based on deep neural networks are not well-suited for these applications. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for constructing image data recognition models, which can be used to construct image data recognition models with few parameters and high performance in application scenarios where image datasets are small and computing and storage resources are limited, in order to recognize image data.

[0006] According to one aspect of this application, a method for constructing an image data recognition model is provided, characterized in that the method comprises:

[0007] Obtain an image dataset and perform ground truth annotation on each image data in the image dataset to obtain an image dataset including ground truth annotations;

[0008] Based on the image dataset and the teacher network model, the FSP method is used to initialize the weights of each layer of the student network to obtain at least two student network models with the same network structure. The teacher network model is a pre-trained neural network with the weights of each layer already initialized. The teacher network model and the student network include the same number of stages, and each stage includes a number of residual blocks. The number of residual blocks in each stage of the teacher network model is greater than or equal to the number of residual blocks in the corresponding stage of the student network.

[0009] Based on an image dataset including ground truth annotations, each student network model is trained using the DML distillation method. Once the trained student network models meet the preset conditions, they are used as image data recognition models.

[0010] Optionally, after ground truth labeling is performed on each image data in the image dataset, a GAN network is used to augment the image data in the image dataset.

[0011] Optionally, the step of initializing the weights of each layer of the student network using the FSP method based on the image dataset and the teacher network model to obtain at least two student network models with the same network structure includes:

[0012] Based on the image dataset and the teacher network model, the student network is trained synchronously to transfer the knowledge of the teacher network model;

[0013] Calculate the L2 loss of the FSP matrix for each stage of the teacher network model and the student network respectively, and calculate the total L2 loss based on all L2 losses, which is used as the first loss;

[0014] If the first loss satisfies a preset first threshold, then training is completed, and at least two student network models with the same network structure are obtained, wherein the weight values ​​of each layer in each student network model are not exactly the same.

[0015] Optionally, if the first loss does not meet a preset first threshold, the method further includes:

[0016] Based on the formula for calculating the first loss, backpropagation is performed to update the weight values ​​of each layer of the student network.

[0017] Based on the image dataset and the teacher network model, the student network with updated weight values ​​at each layer is trained synchronously.

[0018] Optionally, the FSP matrix is ​​determined by calculating the inner product of the input feature map and the output feature map of the corresponding stage.

[0019] Optionally, the step of training each student network model using the DML distillation method based on an image dataset including ground truth annotations, and using the trained student network models as image data recognition models when all trained student network models meet preset conditions, includes:

[0020] Based on an image dataset including ground truth annotations, each student's network model is trained synchronously with the others.

[0021] The metrics of each student network model are evaluated. If the metrics of the trained student network model meet the preset conditions, the training is completed, and the trained student network model is used as the image data recognition model.

[0022] Optionally, the method further includes, before determining the metrics for each student's network model:

[0023] Calculate the KL divergence and CE loss for each student network model. Based on the KL divergence and CE loss of each student network model, determine the second loss for each student network model.

[0024] Once the second loss of each student network model meets the preset second threshold, the metrics of each student network model are then evaluated.

[0025] Optionally, the calculation of the KL divergence for each student network model includes:

[0026] Based on the image dataset, the prediction results of each student network model corresponding to the image dataset are obtained;

[0027] Based on the prediction results of the student network models, the KL divergence of each student network model is calculated.

[0028] Optionally, if the metrics of the trained student network model do not meet the preset conditions, the method further includes:

[0029] Based on the calculation formula of the second loss, backpropagation is performed to update the weight values ​​of each layer of the student network model;

[0030] The student network model, with updated weight values ​​for each layer, is trained synchronously based on an image dataset including ground truth annotations.

[0031] Optionally, the metrics include at least one of the following: precision, recall, accuracy, and F-score.

[0032] Optionally, the method for constructing an image data recognition model further includes:

[0033] The image data to be identified is input into the image data recognition model to determine the category of the image data to be identified.

[0034] Optionally, the method for constructing an image data recognition model further includes:

[0035] The category of the image data to be identified is used as its ground truth label to expand the image dataset including ground truth labels;

[0036] Based on the expanded image dataset including ground truth annotations, the teacher network model, and the student network, the model is retrained to obtain an upgraded and iterative image data recognition model.

[0037] Optionally, the method for constructing an image data recognition model further includes:

[0038] The image data to be identified is differentiated according to its category.

[0039] According to another aspect of this application, a system for constructing an image data recognition model is provided, characterized in that the system comprises:

[0040] The first module is used to acquire an image dataset and perform ground truth annotation on each image data in the image dataset to obtain an image dataset including ground truth annotations;

[0041] The second module is used to initialize the weights of each layer of the student network based on the image dataset and the teacher network model using the FSP method, so as to obtain at least two student network models with the same network structure. The teacher network model is a pre-trained neural network with the weights of each layer already initialized. The teacher network model and the student network include the same number of stages, and each stage includes a number of residual blocks. The number of residual blocks included in each stage of the teacher network model is greater than or equal to the number of residual blocks included in the corresponding stage of the student network.

[0042] The third module is used to train each student network model using the DML distillation method based on an image dataset including ground truth annotations. When all the trained student network models meet the preset conditions, the trained student network models are used as image data recognition models.

[0043] Compared with existing technologies, this application provides a method and system for constructing an image data recognition model. The method includes: acquiring an image dataset and labeling each image data in the image dataset with ground truth values ​​to obtain an image dataset including ground truth labels; based on the image dataset and a teacher network model, initializing the weights of each layer of a student network using the FSP method to obtain at least two student network models with the same network structure, wherein the teacher network model is a pre-trained neural network with pre-initialized weights for each layer, and the teacher network model and the student network include the same number of stages, each stage including a number of residual blocks, wherein the number of residual blocks in each stage of the teacher network model is greater than or equal to the number of residual blocks in the corresponding stage of the student network; training each student network model using the DML distillation method based on the image dataset including ground truth labels, and when all trained student network models meet preset conditions, using the trained student network models as image data recognition models. This method employs the FSP approach to transfer learning from a teacher network model (large model) with the same stage but deeper layers and / or larger layer sizes. This yields at least two student network models (small models) with identical structures and fully initialized layer weights. These student models possess basic feature extraction capabilities, avoiding issues such as insufficient image data for neural network initialization and reducing training difficulty. Each student network model is then trained using the DML distillation method, resulting in an image data recognition model with performance indicators suitable for image data recognition within a short training cycle. This approach features short training cycles, fast inference speed, and easy deployment, making it applicable to various image data classification and recognition application scenarios. Attached Figure Description

[0044] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0045] Figure 1 shows a schematic flowchart of a method for constructing an image data recognition model according to one aspect of this application;

[0046] Figure 2 shows a schematic diagram of an optional embodiment of the FSP method according to this application;

[0047] Figure 3 shows a schematic diagram of an optional embodiment of the present application using a DML distillation method;

[0048] Figure 4 shows a schematic diagram of a system for constructing an image data recognition model according to another aspect of this application;

[0049] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0050] This application provides a method and system for constructing image data recognition models, which is particularly applicable to scenarios where: In the absence of a suitable pre-trained teacher network model (or even if a pre-trained teacher network model exists, it was trained using a dataset with different class divisions than the image dataset of this application), an existing teacher network model trained on a dataset with different class divisions than the image dataset of this application can be directly used. Through the FSP method, the student network transfers and learns from the existing teacher network model, initializing the weights of the student network to obtain at least two student network models with pre-initialized weights at each layer, rather than the conventional FSP distillation training of the student network using the teacher network model (because the existing teacher network model originally...). The network structure described above is for weights used in classification tasks with different class divisions than the image dataset in this application. If the student network is to be trained using FSP distillation, the teacher network must first be trained to a preset metric to obtain a teacher network model. Then, the teacher network model is used to train the student network using FSP distillation, transferring the knowledge from the teacher network model to the student network to obtain a lightweight student network model with the same classification and recognition capabilities as the teacher network model. The teacher network itself is a large model relative to the student network, making training quite time-consuming and labor-intensive. In this application, however, the teacher network does not need to be trained to the preset metric, saving training time and resources. Simultaneously, transfer learning can be achieved through the FSP method, ensuring that the weights of each layer of the student network are not chaotic and possess preliminary feature extraction capabilities. Then, different student network models are trained using DML distillation to obtain an image data recognition model with the ability to recognize image data categories.

[0051] The present invention will now be described in further detail with reference to the accompanying drawings.

[0052] In a typical configuration of various embodiments of this application, each trusted party and / or module of the device, system, and / or apparatus may include one or more processors (CPU, GPU), input / output interfaces, network interfaces, and memory.

[0053] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0054] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0055] To further illustrate the technical means adopted and the effects achieved in this application, the technical solution of this application will be clearly and completely described below in conjunction with the accompanying drawings and preferred embodiments.

[0056] Figure 1 shows a schematic flowchart of a method for constructing an image data recognition model according to one aspect of this application, wherein the method of one embodiment includes:

[0057] S101 acquires an image dataset and performs ground truth annotation on each image data in the image dataset to obtain an image dataset including ground truth annotation;

[0058] S102 Based on the image dataset and the teacher network model, the FSP method is used to initialize the weights of each layer of the student network to obtain at least two student network models with the same network structure. The teacher network model is a pre-trained neural network with the weights of each layer already initialized. The teacher network model and the student network include the same number of stages, and each stage includes a number of residual blocks. The number of residual blocks in each stage of the teacher network model is greater than or equal to the number of residual blocks in the corresponding stage of the student network.

[0059] S103 uses an image dataset with ground truth annotations and employs the DML distillation method to train each student network model. When all trained student network models meet the preset conditions, the trained student network models are used as image data recognition models.

[0060] The various method embodiments of this application can be implemented or performed by device 100, which can be a computer device that can acquire image data via a network, such as a server in a central computer room. The computer device includes, but is not limited to, personal computers, laptops, industrial computers, servers, network hosts, single network servers, or network server clusters. Here, the computer device is merely an example; other existing or future devices and / or resource platforms that are applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.

[0061] In this embodiment, in step S101, device 100 can acquire an image dataset and perform truth value annotation on each image data in the image dataset to obtain an image dataset including truth value annotations (labels).

[0062] For example, device 100 can acquire image data captured by several cameras managed by the smart city management platform in the early stage, and manually identify each image data. By referring to a pre-designed category table or truth labeling rules, each image data is labeled with a truth value to obtain an image dataset including truth value labels.

[0063] If the initial amount of image data is small, the training iteration cycle may be long and the results may not be ideal. Optionally, after the device 100 performs ground truth annotation on each image in the acquired image dataset, it uses a GAN (Generative Adversarial Network) network to augment the image dataset, generating similar image data to supplement the image dataset. This increases the amount of image data available for subsequent training and obtains a more diverse image dataset including ground truth annotations. Alternatively, image data augmentation can be performed by geometric transformations, such as flipping, rotating, cropping, deforming, and scaling, or by color transformations, such as blurring, color changing, erasing, and filling, to augment the image dataset.

[0064] Continuing in this embodiment, in step S102, device 100 initializes the weights of each layer of the student network using the FSP (Flow of Solution Procedure) method based on the image dataset and the teacher network model, to obtain at least two student network models with the same network structure. The teacher network model is a neural network that has been pre-trained and has completed the initialization of the weights of each layer. The teacher network model and the student network include the same number of stages, and each stage includes a number of residual blocks. The number of residual blocks included in each stage of the teacher network model is greater than or equal to the number of residual blocks included in the corresponding stage of the student network.

[0065] The dataset used by the teacher network model in pre-training can have different category classification features than the image dataset of this application. For example, the image dataset of this application is classified into categories such as stations, gas stations, parking lots, and shopping streets, and the teacher network model can use a dataset that distinguishes vehicle categories. Alternatively, it can have the same category classification features as the image dataset of this application, and similarly, be classified into categories such as stations, gas stations, parking lots, and shopping streets.

[0066] In cases where the amount of acquired image data is small, in addition to expanding the image data through GAN networks, this application considers using the FSP method based on the acquired image dataset. Through transfer learning, the lightweight student network can learn the knowledge of the teacher network model. After transfer learning, the weight values ​​of each layer of the student network are initialized to obtain at least two student network models with the same network structure. The teacher network model is a pre-trained neural network with all layers' weights initialized. The student network has the same number of stages as the teacher network model. Each stage of both the teacher and student networks includes a number of residual blocks. The number of residual blocks in each stage of the teacher network model and the student network may not be the same. However, the number of residual blocks in each stage of the teacher network model is greater than or equal to the number of residual blocks in the corresponding stage of the student network. This reduces the initial training difficulty and training cycle, and allows each layer of the learned student network model to have preliminary feature extraction capabilities, which is beneficial for convergence optimization and can improve the subsequent training speed. Moreover, the image recognition model obtained after subsequent training can be applied to application scenarios with limited computing and storage capabilities.

[0067] In practical application scenarios with limited image data, if there are high requirements for image recognition metrics, fast image recognition speed, and easy deployment, the FSP method adopted in this application can be used to train a small model with a certain structure (as a student network) using a large model (as a teacher network model) that has been pre-trained on a large dataset and has completed the weight initialization of each layer. This can effectively achieve transfer learning, and the resulting small model has good convergence and generalization effects. Moreover, each layer has preliminary feature extraction capabilities, which is beneficial for further training of the small model in the later stages, resulting in an image data recognition model for use in image data recognition scenarios with high recognition capability requirements.

[0068] Optionally, the step of initializing the weights of each layer of the student network using the FSP method based on the image dataset and the teacher network model to obtain at least two student network models with the same network structure includes:

[0069] Based on the image dataset and the teacher network model, the student network is trained synchronously to transfer the knowledge of the teacher network model;

[0070] Calculate the L2 loss of the FSP matrix for each stage of the teacher network model and the student network respectively, and calculate the total L2 loss based on all L2 losses, which is used as the first loss;

[0071] If the first loss satisfies a preset first threshold, then training is completed, and at least two student network models with the same network structure are obtained, wherein the weight values ​​of each layer in each student network model are not exactly the same.

[0072] As shown in Figure 2, an example is provided. `stem` represents the initial part of the teacher and student network models, including layers not belonging to later stages, such as convolutional layers, Batch Normalization (BN) layers, and activation function layers. `FC` represents fully connected layers. The content of `stem` and `FC` is determined based on the teacher and student network models used. Within the same teacher or student network model, the depth or number of residual blocks in later stages may not be the same. It is sufficient that the teacher and student networks include the same number of stages. The specific number of stages can be customized. Figure 2 shows that both the teacher and student network models include four stages (stage1, stage2, stage3, stage4). The configuration of the number of residual blocks in each stage of the teacher and student networks can refer to the ResNet network configuration scheme or be customized. Generally, the number of residual blocks in each stage of the teacher network model is greater than or equal to the number of residual blocks in the corresponding stage of the student network. Other network structures (such as RegNet series, Darknet series, Transformer series, etc.) can be used for the teacher network model and student network. The specific choice should be based on the actual application scenario. This application does not impose any restrictions on these structures.

[0073] The residual block structures of the teacher network model and the student network at different stages are not necessarily the same, and the number of residual blocks included in different stages is also not necessarily the same. The specific configuration should be determined based on the chosen configuration scheme in the actual application scenario. As shown in Figure 2, m∈{m1,m2,m3,m4} represents the number of residual blocks in each stage of the teacher network model, and n∈{n1,n2,n3,n4} represents the number of residual blocks in each stage of the student network. m should be greater than or equal to n. The selection of the depth and / or layer size of the student network should be determined in conjunction with the actual application scenario, the selected teacher network model, and the difficulty of transfer learning. For example, in a smart city management platform, the selected teacher network model, which has been pre-trained and has completed the weight initialization of each layer, can have a backbone network of ResNet34. However, considering that the model is generally used in edge devices with limited computing and storage capabilities, the depth of the student network can be smaller, and its backbone can be ResNet18.

[0074] Since the depth and / or layer size of the student network are usually smaller than those of the teacher network model, directly assigning the initial weights of each layer of the teacher network model to the student network will not be able to adapt them one by one. The teacher network model needs to be trimmed and adjusted first. Moreover, the convergence and generalization effect of the student network model obtained by directly copying the teacher weights is not as good as using the FSP method for transfer learning to initialize the weights of the student network (which does not have a strict distillation effect and is not for knowledge transfer).

[0075] In this process, image datasets are simultaneously input into both the teacher and student networks. The student network is trained concurrently using the pre-trained teacher network model with pre-initialized weights for each layer, enabling transfer learning of the teacher network model's knowledge. During this synchronous iterative training for transfer learning, the initial weights of each layer in the student network are determined by minimizing the distance between the FSP matrices of the corresponding stages in the student and teacher network models, thus initializing the weights for each layer. Specifically, the L2 loss of the FSP matrices for each stage in both the teacher and student networks is calculated, and the total L2 loss is calculated based on the L2 losses of all stages, serving as the first loss.

[0076] As an example, the FSP matrix can be calculated using the following formula:

[0077]

[0078] Where i and j are the channel numbers of F1 and F2, x is the input image, W represents the weights of the corresponding network, F1 and F2 represent the resized input and output feature map matrices, respectively, and h and w represent the height and width of the feature map, respectively.

[0079] The first loss can be calculated using the following formula:

[0080]

[0081] in, Let FSP be the FSP matrix for the i-th stage of the teacher network model. Let λ be the FSP matrix of the i-th stage of the student network. i Let N be the loss weight for stage i, N be the number of samples, and n be the number of stages selected.

[0082] The first loss is compared with a preset first threshold. If the first loss meets the preset first threshold, for example, if the first loss is less than or equal to the preset first threshold, then the training is completed, and at least two student network models with the same network structure and better generalization are obtained, and each layer has preliminary feature extraction capabilities. The weight values ​​of each layer in each student network model are similar or close, but not completely the same (if they are completely the same, then they are essentially the same student network model).

[0083] As shown in Figure 2, the calculation of the first loss is to first calculate the FSP matrix of the four stages of the teacher network model (stages 1-4) and the four stages of the student network (stages 1-4) respectively. Then, the distance between the FSP matrices of the corresponding stages of the teacher network model and the student network is calculated as the L2 loss of that stage. A total of four L2 losses for the four stages are obtained. Then, the first loss can be calculated according to the above formula (2).

[0084] Optionally, if the first loss does not meet a preset first threshold, the method further includes:

[0085] Based on the formula for calculating the first loss, backpropagation is performed to update the weight values ​​of each layer of the student network.

[0086] Based on the image dataset and the teacher network model, the student network with updated weight values ​​at each layer is trained synchronously.

[0087] In the synchronous iterative training process for transfer learning, if the first loss does not meet the preset first threshold, the back propagation (BP) algorithm can be used to update the weight values ​​of each layer of the student network according to the calculation formula of the first loss. Then, using an image dataset, the teacher network model and the updated student network are synchronously input, and iterative training is performed again for transfer learning. This continues until the first loss meets the preset first threshold, completing the training and obtaining at least two student network models with identical network structures, whose layers possess preliminary feature extraction capabilities.

[0088] Optionally, the FSP matrix is ​​determined by calculating the inner product of the input feature map and the output feature map of the corresponding stage.

[0089] In the student network and teacher network models, the input feature map of each stage is processed by the residual block of that stage to obtain the output feature map. The inner product of the input feature map and the output feature map of each stage can be used as the FSP matrix of that stage (the FSP matrix can essentially represent the solution process flow of that stage).

[0090] Continuing in this embodiment, in step S103, the device 100 trains each student network model using the DML (Deep Mutual Learning) distillation method based on the image dataset including ground truth annotations. When all the trained student network models meet the preset conditions, the trained student network models are used as image data recognition models.

[0091] In step S102, at least two lightweight student network models are obtained, each of which has preliminary feature extraction capabilities. In step S103, the DML distillation method is used to allow the student network models to interact and learn from each other as teacher network models based on an image dataset including ground truth annotations. When each student network model (usually there are multiple student network models) meets the preset conditions, the training is completed, and any trained student network model can be used as an image recognition model.

[0092] Since the student network model is a lightweight, small model, and it had already been trained using the FSP method before applying the DML distillation method, its layers possess preliminary feature extraction capabilities and are already approaching convergence. Therefore, training with the DML distillation method is easier to converge, has better generalization ability, and is more adaptable to lower-end computing and storage devices. The resulting image data recognition model is more suitable for the edge of distributed networks.

[0093] Optionally, step 103 includes:

[0094] Based on an image dataset including ground truth annotations, each student's network model is trained synchronously with the others.

[0095] The metrics of each student network model are evaluated. If the metrics of the trained student network model meet the preset conditions, the training is completed, and the trained student network model is used as the image data recognition model.

[0096] The image dataset is synchronously input into two student network models, allowing the student network models to train each other iteratively as teacher network models and learn interactively. After each iteration of mutual training, the metrics of each student network model are evaluated. If the metrics of each trained student network model meet the preset conditions, the training is completed, and the trained student network model is used as the image data recognition model.

[0097] Optionally, before determining the metrics for each student network model, step S103 further includes:

[0098] Calculate the KL divergence and CE loss for each student network model. Based on the KL divergence and CE loss of each student network model, determine the second loss for each student network model.

[0099] Once the second loss of each student network model meets the preset second threshold, the metrics of each student network model are then evaluated.

[0100] To avoid judging the metrics of each student network model after each iteration of mutual training (especially in the early stages of mutual training, when metrics often do not meet preset conditions, judging the metrics of each student network model is time-consuming, memory-intensive, and slows down the training and optimization speed), the KL divergence and CE loss of each student network model can be calculated before judging the metrics of each student network model. Based on the KL divergence and CE loss of each student network model, the second loss of each student network model is determined. When the second loss of each student network model meets the preset second threshold, the metrics of each student network model are then judged.

[0101] As shown in Figure 3, an example is obtained by simultaneously inputting the image dataset into two student network models (Studentmodel1, Studentmodel2) for training, resulting in prediction results 1 (predictions1 or p1) and 2 (predictions2 or p2) respectively. Then, the Kullback-Leibler divergence (KL divergence) of each student network model is calculated. The KL divergence of Studentmodel1 can be represented as KL(p2||p1), and the KL divergence of Studentmodel2 can be represented as KL(p1||p2). Combined with the ground truth labels of the image data, the Cross Entropy Loss (CE loss) of each student network model is calculated, where Lc1 is the CE loss of Studentmodel1, and Lc2 is the CE loss of Studentmodel2. The CE loss of model2 can be determined by the KL divergence and CE loss of each student network model to determine the second loss of each student network model. Then, the second loss of each student network model is judged. If the second loss of the trained student network models all meet the preset second threshold, the index of each trained student network model is further judged. If the index also meets the preset conditions, the training is completed and the trained student network model is used as the image data recognition model.

[0102] When determining the second loss for each student network model based on its KL divergence and CE loss, the specific application scenario should be considered. Generally, weights for the KL divergence and CE loss can be omitted, and the sum of their values ​​can be used as the second loss. Alternatively, weights for the KL divergence and CE loss can be pre-set to adjust their impact on the second loss at different stages of synchronous training. For example, in the early stages of iterative training, the recognition ability of each student network model is insufficient, and the information from the ground truth labels is more trusted. Therefore, the weight of the CE loss can be increased in the second loss calculation. As the recognition ability improves during iterative training, the weight of the KL divergence can be increased in the second loss calculation. In the later stages of iterative training, both weights can be set to 0.5. The second loss of the student network model, where the weights of the pre-set KL divergence and CE loss are used, can be calculated using the following formula:

[0103] Loss = a*KL + (1-a)*CE, 0 = <a<=1, (3)

[0104] Where a is the weight of the KL divergence of the student network model, and (1-a) is the weight of the CE loss of the student network model.

[0105] For example, initially set 'a' close to 0, and later increase 'a' to 0.5. The specific setting depends on the recognition ability of the student's network model after training.

[0106] Optionally, the calculation of the KL divergence for each student network model includes:

[0107] Based on the image dataset, the prediction results of each student network model corresponding to the image dataset are obtained;

[0108] Based on the prediction results of the student network models, the KL divergence of each student network model is calculated.

[0109] As shown in Figure 3, for student network model 1, prediction result 1 corresponding to the image dataset can be obtained, and for student network model 2, prediction result 2 corresponding to the image dataset can be obtained. Then, based on prediction result 2, the KL divergence between prediction result 1 and prediction result 2 is calculated as the KL divergence of student network model 1; based on prediction result 1, the KL divergence between prediction result 2 and prediction result 1 is calculated as the KL divergence of student network model 2. Then, based on prediction result 1 of student network model 1 and the ground truth label of the corresponding image dataset, the CE loss of student network model 1 is calculated; based on prediction result 2 of student network model 2 and the ground truth label of the corresponding image dataset, the CE loss of student network model 2 is calculated. Referring to formula (3), the weighted sum of the KL divergence and CE loss of student network model 1 is used as the second loss of student network model 1; the weighted sum of the KL divergence and CE loss of student network model 2 is used as the second loss of student network model 2. Next, the second loss of each student network model is judged. If the second loss of the trained student network models all meet the preset second threshold, the index of each trained student network model is further judged. If the index also meets the preset conditions, the training is completed, and the trained student network model is used as the image data recognition model.

[0110] Optionally, if the metrics of the trained student network model do not meet the preset conditions, the method further includes:

[0111] Based on the calculation formula of the second loss, backpropagation is performed to update the weight values ​​of each layer of the student network model;

[0112] The student network model, with updated weight values ​​for each layer, is trained synchronously based on an image dataset including ground truth annotations.

[0113] In the DML distillation training of at least two identical student network models, if the second loss of the trained student network models all meet the preset second threshold, a validation dataset is used for verification. The obtained metrics are then evaluated to see if they all meet the preset conditions. The second threshold is only for reference; training can only be completed if the metrics meet the preset conditions. Usually, if the metrics meet the preset conditions, the second loss will also meet the preset second threshold, especially in the early stages of iterative training. Often, the second loss is checked first to see if it meets the preset second threshold; if so, the preset conditions are then checked. If the metrics do not meet the preset conditions, it means the trained student network model cannot yet be used as an image data recognition model. DML distillation training needs to continue. Based on the calculation formula of the second loss, the backpropagation algorithm is used to update the weight values ​​of each layer of the student network model. Then, using an image dataset including ground truth annotations, the student network model with updated weight values ​​is trained synchronously until the metrics meet the preset conditions, thus completing the training. Each trained student network model that meets the preset conditions can be used as an image data recognition model.

[0114] Optionally, the metrics include at least one of the following: precision, recall, accuracy, and F-score.

[0115] Based on optional embodiment 1 of Figure 3, the metrics include one of precision, recall, accuracy, or F-score. If the second loss of both student network model 1 and student network model 2 satisfies a preset second threshold, and the metrics of both student network model 1 and student network model 2 obtained based on the image dataset (a portion of the image data can be selected from the image dataset as a validation set and / or test set) satisfy preset conditions, wherein the preset conditions include a threshold corresponding to one of precision, recall, accuracy, or F-score, then training is complete. Each trained student network model (student network model 1 or student network model 2) whose metrics satisfy the preset conditions can be used as an image data recognition model.

[0116] Based on optional embodiment 2 of Figure 3, the metrics include two of precision, recall, accuracy, or F-score. If the second loss of student network model 1 and student network model 2 satisfies a preset second threshold, and the metrics of student network model 1 and student network model 2 obtained based on the image dataset (a portion of the image data can be selected from the image dataset as a validation set and / or test set) satisfy preset conditions, wherein the preset conditions include thresholds for the corresponding metrics including two of precision, recall, accuracy, or F-score, then training is complete. Each trained student network model (student network model 1 or student network model 2) whose metrics satisfy the preset conditions can be used as an image data recognition model.

[0117] Based on optional embodiment 3 of Figure 3, the metrics include three of precision, recall, accuracy, or F-score. If the second loss of student network model 1 and student network model 2 satisfies a preset second threshold, and the metrics of student network model 1 and student network model 2 obtained based on the image dataset (a portion of the image data can be selected from the image dataset as a validation set and / or test set) satisfy preset conditions, wherein the preset conditions include thresholds for the corresponding metrics including three of precision, recall, accuracy, or F-score, then training is complete. Each trained student network model (student network model 1 or student network model 2) whose metrics satisfy the preset conditions can be used as an image data recognition model.

[0118] Based on optional embodiment 4 of Figure 3, the metrics include four: precision, recall, accuracy, and F-score. If the second loss of student network model 1 and student network model 2 satisfies a preset second threshold, and the metrics of student network model 1 and student network model 2 obtained based on the image dataset (a portion of the image data can be selected from the image dataset as a validation set and / or test set) satisfy preset conditions, wherein the preset conditions include the corresponding four thresholds for precision, recall, accuracy, and F-score, then training is complete. The trained student network model (student network model 1 or student network model 2) whose metrics satisfy the preset conditions can be used as an image data recognition model.

[0119] The above embodiments and / or optional embodiments provide a method for constructing an image data recognition model. The method uses the FSP method to enable student networks (small models) to transfer and learn knowledge from teacher network models (large models), resulting in at least two student network models with the same structure. Each layer of these student network models has preliminary feature extraction capabilities, which can avoid problems such as the inability to properly initialize neural networks due to insufficient image data and can reduce the training difficulty. Then, based on the DML distillation method, the student network models train and learn from each other, which can obtain an image data recognition model that meets the requirements for image data recognition within a short training cycle. The method features a short training cycle, fast inference speed, and easy deployment.

[0120] Optionally, the method for constructing an image data recognition model further includes:

[0121] S104 Input the image data to be identified into the image data recognition model to determine the category of the image data to be identified.

[0122] In this embodiment and / or optional embodiment including steps S101 to S103, the construction of the image data recognition model is completed. In step S104, the image data recognition model can be used to identify the image data to be identified that has the same project task or features as the aforementioned image dataset, and to determine the category of the image data to be identified.

[0123] An exemplary image data recognition model for a smart city management platform, constructed using the above-described embodiments and / or optional embodiments including steps S101 to S103, can be deployed on the edge computing device of the smart city management platform to perform real-time online recognition of image data to be recognized collected by cameras in order to determine the category of the image data to be recognized.

[0124] Optionally, the method for constructing an image data recognition model further includes:

[0125] S105 uses the category of the image data to be identified as its ground truth label, expands the image dataset including ground truth labels, and retrains based on the expanded image dataset including ground truth labels, the teacher network model and the student network to obtain an upgraded and iterative image data recognition model.

[0126] In this process, after determining the category of the image data to be identified that has the same project task or features as the aforementioned image dataset through the above-described embodiments and / or optional embodiments including steps S101 to S104, in step S105, the category of the image data to be identified is used as its ground truth label (if necessary, the ground truth label of the image data to be identified is also manually verified and corrected to ensure accuracy), and then added to the image dataset including ground truth labels to expand the image dataset including ground truth labels. Based on the expanded image dataset including ground truth labels, the teacher network model, and the student network, the FSP method and DML distillation method are retrained to obtain an upgraded iterative image data recognition model. This process is repeated iteratively to continuously improve the recognition capability of the image data recognition model.

[0127] Moreover, small models can be trained online in real time using the DML distillation method. This not only avoids the time consumption of traditional offline distillation training based on large models, but also expands the image dataset after the real-time recognized image data is manually verified. Based on the image dataset with expanded actual image data and the teacher network model, the student network can be iteratively trained using the FSP method and the DML distillation method to achieve the iteration of the image recognition model and further improve the recognition ability.

[0128] For example, in addition to deploying the image data recognition model on the edge computing device of the smart city management platform, another image data recognition model can be deployed on the central computing device 100 of the smart city management platform. This image data recognition model is trained using the DML distillation method, and the two models have the same structure but are different student network models trained using the FSP method on the same teacher network model and student network. The image data model deployed on the edge computing device is used to perform real-time online recognition of images captured by cameras to determine the category of the image data. The determined category can be used as a ground truth label for the image data and added to the image dataset including ground truth labels to expand the image dataset. Then, based on the expanded image dataset including ground truth labels, the image recognition model deployed on the central computing device and the image recognition model deployed on the edge computing device can be trained online in real-time using DML distillation and iteratively updated to further improve recognition capabilities. If it is necessary to reinitialize the weights of each layer of the student network model, the FSP method can be used to transfer learning from the teacher network model to obtain at least two new student network models with optimized initial weights for each layer, based on the expanded image dataset including ground truth annotations. Then, the DML distillation method can be used for iterative training to obtain an image data recognition model with higher recognition capabilities.

[0129] Optionally, the method for constructing an image data recognition model further includes:

[0130] S106 Performs differential processing on the image data to be identified according to the category of the image data to be identified.

[0131] In one exemplary embodiment, through the above-described embodiments and / or optional embodiments including steps S101 to S104, the image data recognition model deployed in the edge computer device 100 of the smart city management platform performs real-time online recognition on the image data to be recognized collected by the camera to determine the category of the image data to be recognized. Then, in step S106, the image data to be recognized is further differentiated according to the category of the image data to be recognized, such as classified monitoring, early warning, etc.

[0132] Figure 4 illustrates a schematic diagram of a system for constructing an image data recognition model according to another aspect of this application, wherein, in one embodiment, the system includes:

[0133] The first module 310 is used to acquire an image dataset and perform ground truth annotation on each image data in the image dataset to obtain an image dataset including ground truth annotation;

[0134] The second module 320 is used to initialize the weights of each layer of the student network based on the image dataset and the teacher network model using the FSP method, so as to obtain at least two student network models with the same network structure. The teacher network model is a neural network that has been pre-trained and has completed the initialization of the weights of each layer. The teacher network model and the student network include the same number of stages, and each stage includes a number of residual blocks. The number of residual blocks included in each stage of the teacher network model is greater than or equal to the number of residual blocks included in the corresponding stage of the student network.

[0135] The third module 330 is used to train each student network model using the DML distillation method based on an image dataset including ground truth annotations. When the trained student network models all meet the preset conditions, the trained student network models are used as image data recognition models.

[0136] In this embodiment, the system is deployed or integrated in the device 100 that performs the foregoing method embodiments and / or optional embodiments.

[0137] In this embodiment, the first module 310 of the system can acquire an image dataset and perform truth value annotation on each image data in the image dataset to obtain an image dataset including truth value annotations (labels). For example, the device 100 can acquire image data captured by several cameras managed by a smart city management platform, and manually identify each image data, referring to a pre-designed category table to perform truth value annotation on each image data, thereby obtaining an image dataset including truth value annotations.

[0138] Continuing in this embodiment, the second module 320 of the system, based on the acquired image dataset, employs the FSP method and transfer learning to enable the lightweight student network to learn the knowledge of the teacher network model. Through transfer learning, the weight values ​​of each layer of the student network are initialized, resulting in at least two student network models with identical network structures. The teacher network model is a pre-trained neural network with pre-initialized weights for each layer. The number of stages in the selected student network is the same as the number of stages in the teacher network model. Each stage in both the teacher and student networks includes a number of residual blocks. However, the number of residual blocks in each stage of the teacher network model is greater than or equal to the number of residual blocks in the corresponding stage of the student network. This reduces the initial training difficulty and training cycle, and improves subsequent training speed. Furthermore, the image recognition model obtained after subsequent training is suitable for application scenarios with limited computing and storage capabilities.

[0139] Through the second module 320 described above, at least two lightweight student network models are obtained. In this embodiment, the third module 330 of the system uses the DML distillation method to allow the student network models to interact and learn from each other as teacher network models based on an image dataset including ground truth annotations. When each student network model (usually there are multiple student network models) meets the preset conditions, the training is completed, and any trained student network model can be used as an image recognition model.

[0140] Optionally, a system for building image data recognition models further includes:

[0141] The fourth module 340 is used to input the image data to be identified into the image data recognition model to determine the category of the image data to be identified.

[0142] In this optional embodiment, the fourth module 340 of the system can use the image data recognition model to identify image data to be identified that has the same project task or features as the aforementioned image dataset, and determine the category of the image data to be identified.

[0143] Optionally, a system for building image data recognition models further includes:

[0144] The fifth module 350 is used to expand the image dataset including the ground truth label by taking the category of the image data to be identified as its ground truth label, and to retrain based on the expanded image dataset including the ground truth label, the teacher network model and the student network to obtain an upgraded and iterative image data recognition model.

[0145] In this optional embodiment, the fifth module 350 of the system uses the category of the image data to be identified as its ground truth label (and, if necessary, manually verifies and corrects the ground truth label of the image data to be identified to ensure accuracy), and then supplements it to the image dataset including ground truth labels to expand the image dataset including ground truth labels. Based on the expanded image dataset including ground truth labels, the teacher network model, and the student network, the system retrains using the FSP method and the DML distillation method to obtain an upgraded and iterative image data recognition model. This process is repeated iteratively to continuously improve the recognition capability of the image data recognition model.

[0146] Optionally, a system for building image data recognition models further includes:

[0147] The sixth module 360 ​​is used to perform differential processing on the image data to be identified according to the category of the image data to be identified.

[0148] In this optional embodiment, the sixth module 360 ​​of the system performs differentiated processing on the image data to be identified according to the category of the image data to be identified, such as classification monitoring and early warning.

[0149] In the various embodiments and / or optional embodiments of the above system, the parts of the method steps executed by each module of the system that are not mentioned are the same as those in the aforementioned related method embodiments and / or optional embodiments, and will not be repeated here.

[0150] In various embodiments and / or optional embodiments of this application, the image dataset may also be a publicly available image dataset with completed ground truth annotation that has the same or similar characteristics as the project task in the actual application scenario.

[0151] According to another aspect of this application, a computer-readable medium is also provided, the computer-readable medium storing computer-readable instructions that can be executed by a processor to implement the foregoing method embodiments.

[0152] It should be noted that the method embodiments and / or optional embodiments in this application do not strictly limit the order of execution of each step. As long as the method embodiments and / or optional embodiments can solve the defects of the prior art, achieve the inventive purpose of this application, and obtain beneficial effects, they are acceptable. The method embodiments and / or optional embodiments in this application can be implemented in software and / or combinations of software and hardware. The software program involved in this application can be executed by a processor to implement the steps or functions of the above embodiments. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium.

[0153] Furthermore, part or all of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. The program instructions invoking the methods of this application may be stored in a fixed or removable recording medium, and / or transmitted via data streams in broadcast or other signal carrying media, and / or stored in the working memory of a computer device operating according to the program instructions.

[0154] According to another aspect of this application, an apparatus for constructing an image data recognition model is also provided. The apparatus includes: a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the apparatus is triggered to run the methods and / or technical solutions of the foregoing embodiments.

[0155] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device through software and / or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

Claims

1. A method for constructing an image data recognition model, characterized in that, The method includes: acquiring an image dataset and performing ground truth annotation on each image data in the image dataset to obtain an image dataset including ground truth annotations; initializing the weights of each layer of the student network using the FSP method based on the image dataset and the teacher network model to obtain at least two student network models with the same network structure, wherein the initializing the weights of each layer of the student network using the FSP method based on the image dataset and the teacher network model to obtain at least two student network models with the same network structure includes: synchronously training the student network based on the image dataset and the teacher network model to transfer learning of the knowledge of the teacher network model; calculating the L2 loss of the FSP matrix for each stage of the teacher network model and the student network respectively, and calculating the total L2 loss based on all L2 losses, as the first... Loss; if the first loss meets a preset first threshold, training is completed, resulting in at least two student network models with identical network structures, wherein the weight values ​​of each layer in each student network model are not completely identical, wherein the teacher network model is a pre-trained neural network with initialized weights for each layer, and the teacher network model and the student network include the same number of stages, each stage including a number of residual blocks, wherein the number of residual blocks in each stage of the teacher network model is greater than or equal to the number of residual blocks in the corresponding stage of the student network; based on an image dataset including ground truth annotations, each student network model is synchronously trained with each other, and the metrics of each student network model are judged. If the metrics of the trained student network model meet preset conditions, training is completed, and the trained student network model is used as an image data recognition model.

2. The method according to claim 1, characterized in that, After ground truth labeling for each image in the image dataset, a GAN network is used to augment the image data in the image dataset.

3. The method according to claim 1, characterized in that, If the first loss does not meet the preset first threshold, the method further includes: performing backpropagation according to the calculation formula of the first loss to update the weight values ​​of each layer of the student network; and synchronously training the student network with updated weight values ​​of each layer based on the image dataset and the teacher network model.

4. The method according to claim 1, characterized in that, The FSP matrix is ​​determined by calculating the inner product of the input feature map and the output feature map of the corresponding stage.

5. The method according to claim 1, characterized in that, Before determining the metrics of each student network model, the method further includes: calculating the KL divergence of each student network model and the CE loss of each student network model; determining the second loss of each student network model based on the KL divergence and CE loss of each student network model; and determining the metrics of each student network model only when the second loss of each student network model meets the preset second threshold.

6. The method according to claim 5, characterized in that, The calculation of the KL divergence of each student network model includes: obtaining the prediction result of each student network model for the image dataset based on the image dataset; and calculating the KL divergence of each student network model based on the prediction result of the student network model.

7. The method according to claim 5, characterized in that, If the metrics of the trained student network model do not meet the preset conditions, the method further includes: performing backpropagation according to the calculation formula of the second loss to update the weight values ​​of each layer of the student network model; and synchronously training the student network model with updated weight values ​​of each layer based on an image dataset including ground truth annotations.

8. The method according to claim 1, characterized in that, The metrics include at least one of the following: precision, recall, accuracy, and F-score.

9. The method according to claim 1, characterized in that, The method further includes: inputting the image data to be identified into the image data recognition model to determine the category of the image data to be identified.

10. The method according to claim 9, characterized in that, The method further includes: using the category of the image data to be identified as its ground truth label, expanding the image dataset including ground truth labels; and retraining based on the expanded image dataset including ground truth labels, the teacher network model, and the student network to obtain an upgraded and iterative image data recognition model.

11. The method according to claim 9, characterized in that, The method further includes: performing differential processing on the image data to be identified according to the category of the image data to be identified.

12. A system for constructing image data recognition models, characterized in that, The system includes: a first module for acquiring an image dataset and performing ground truth annotation on each image data in the image dataset to obtain an image dataset including ground truth annotations; and a second module for initializing the weights of each layer of the student network using the FSP method based on the image dataset and the teacher network model to obtain at least two student network models with the same network structure. The initialization of the weights of each layer of the student network using the FSP method based on the image dataset and the teacher network model to obtain at least two student network models with the same network structure includes: synchronously training the student network based on the image dataset and the teacher network model to transfer learning knowledge from the teacher network model; calculating the L2 loss of the FSP matrix for each stage of the teacher network model and the student network respectively; and calculating the total L2 loss based on all L2 losses. As the first loss; if the first loss meets a preset first threshold, then training is completed, and at least two student network models with the same network structure are obtained, wherein the weight values ​​of each layer between each student network model are not completely the same, wherein the teacher network model is a neural network that has been pre-trained and the weights of each layer have been initialized, the teacher network model and the student network include the same number of stages, each stage includes a number of residual blocks, wherein the number of residual blocks included in each stage of the teacher network model is greater than or equal to the number of residual blocks included in the corresponding stage of the student network; the third module is used to synchronously train each student network model based on an image dataset including ground truth annotations, judge the index of each student network model, and if the index of the trained student network model meets a preset condition, then training is completed, and the trained student network model is used as an image data recognition model.

13. A computer-readable medium, characterized in that, It stores computer-readable instructions that are executed by a processor to implement the method as claimed in any one of claims 1 to 11.

14. A device for classifying network traffic data, characterized in that, The device includes: one or more processors; and a memory storing computer-readable instructions that, when executed, cause the processor to perform the operations of the method as described in any one of claims 1 to 11.

Images