A semantic perception-based model compression method and a visual recognition method

Abstract

The application relates to the technical field of computer vision, in particular to a model compression method based on semantic perception and a visual recognition method, which are used for solving the problem of low generalization ability of a traditional model compression model, and the model compression method is as follows: image samples in a training set are input into a teacher model and a student model to obtain a knowledge imitation loss, a regularization loss and a target recognition loss of the student model, the knowledge imitation loss comprises differences between image features output by corresponding layers in the two models and differences between target recognition results output by output layers; first distillation weights and second distillation weights are determined based on semantic soft labels determined according to the image samples and a plurality of description texts, a single description text describes an image degradation type or a structural attribute of an image; model parameters of the student model are adjusted based on the first distillation weights, the knowledge imitation loss, weighted values of the second distillation weights and the regularization loss and the target recognition loss, so that a trained visual recognition model is obtained, and the generalization ability of the model is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a semantically aware model compression method and a visual recognition method. Background Technology

[0002] In industrial automation systems, visual recognition tasks such as product identification, defect detection, and equipment status determination rely heavily on deep neural network models. However, due to the complex imaging conditions in industrial settings, higher demands are placed on the robustness of these models.

[0003] To meet the deployment requirements of edge devices for low latency, low power consumption, and small memory footprint, model compression technology is widely used. Knowledge distillation is a model compression method. Knowledge distillation transfers knowledge from the original model (teacher model) to the compressed model (student model), so that although the compressed model is much smaller than the original model, it can still learn similar knowledge.

[0004] However, traditional model compression methods typically employ a uniform supervised training strategy for all training samples. This results in a significant decrease in the recognition performance of compressed models when faced with complex samples, such as those with severe image degradation or complex image structures, thus limiting the deployment of compressed models under diverse operating conditions. Summary of the Invention

[0005] This application provides a semantically aware model compression method and a visual recognition method, which enable the model to automatically perceive the content characteristics of the input image samples during the knowledge distillation training process, and dynamically adjust the distillation loss weights accordingly to improve the generalization performance of the trained visual recognition model.

[0006] The specific technical solutions provided in this application are as follows: In a first aspect, embodiments of this application provide a semantically aware model compression method, comprising: Image samples from the training set are input into the teacher model and the student model to obtain the knowledge imitation loss, regularization loss, and target recognition loss of the student model. Based on the image samples and multiple descriptive texts, semantic soft labels are determined for the image samples, and first distillation weights and second distillation weights are determined based on the semantic soft labels. A single descriptive text is used to describe an image degradation type or structural attribute of the image. Based on the weighted values ​​of the first distillation weight, the knowledge imitation loss, the second distillation weight, and the regularization loss, as well as the target recognition loss, the model parameters of the student model are adjusted to obtain the trained visual recognition model. The knowledge imitation loss includes the differences between the image features output by the corresponding intermediate layers in the teacher model and the student model, and the differences between the target recognition results output by the output layer. The regularization loss is determined based on the parameter distribution of multiple network layers in the student model, and the target recognition loss is determined based on the difference between the target recognition result output by the student model and the actual target label of the image sample.

[0007] In some possible implementations, the step of inputting image samples from the training set into the teacher model and the student model to obtain the knowledge imitation loss, regularization loss, and target recognition loss of the student model includes: The image samples in the training set are input into the teacher model to obtain the first image features output by multiple first intermediate layers and the first target recognition result output by the output layer in the teacher model. The image sample is input into the student model to obtain the second image features output by multiple second intermediate layers and the second target recognition result output by the output layer in the student model, as well as the parameter set of the multiple network layers. Each second intermediate layer corresponds to a first intermediate layer, and each network layer is any network layer in the student model. The knowledge imitation loss is determined based on the difference between each first image feature and the corresponding second image feature output by the layer, as well as the difference between the first target recognition result and the second target recognition result. The regularization loss is determined based on the parameter distribution in the parameter set of the target network layer, where the target network layer is any one of the plurality of network layers; The target recognition loss is determined based on the difference between the second target recognition result and the actual target label.

[0008] In some possible implementations, determining the semantic soft tags of the image samples based on the image samples and multiple descriptive texts includes: The image samples are image encoded to obtain the third image features; The third image features are compared with the text features of the multiple descriptive texts to obtain multiple similarity scores; The multiple similarities are normalized to obtain semantic attribute vectors, which are the semantic soft tags.

[0009] In some possible implementations, determining the first distillation weight and the second distillation weight based on the semantic soft label includes: Based on the semantic soft tags, it is determined whether the image sample is a target image sample. The semantic soft tags of the target image sample contain N types of image degradation and the probability of each type of image degradation is greater than a first preset value, and / or M types of structural attributes and the probability of each structural attribute is greater than a second preset value. N and M are both positive integers greater than 1. If so, the first distillation weight and the second distillation weight are determined based on the total number of the first type of image degradation type in the semantic soft tag that is greater than the first preset value, and / or the total number of the second type of structural attribute that is greater than the second preset value. The first distillation weight is greater than the preset first benchmark weight and is positively correlated with the sum of the total number of the first type and the total number of the second type of the image sample. The second distillation weight is less than the preset second benchmark weight and is negatively correlated with the sum of the total number of the first type and the total number of the second type of the image sample. If not, the first benchmark weight is determined as the first distillation weight, and the second benchmark weight is determined as the second distillation weight.

[0010] In some possible implementations, determining the first distillation weight and the second distillation weight based on the semantic soft label includes: The semantic soft labels are input into a preset neural network to obtain the first distillation weight and the second distillation weight; wherein, the neural network is trained on the neural network to be trained based on the semantic soft labels and supervision labels of multiple historical image samples, the semantic soft label of a single historical image sample is determined based on the similarity between the historical image sample and the multiple descriptive texts, and the supervision label of the historical image sample is determined based on the semantic soft labels of the historical image sample and the ranking result of the influence of each image degradation type or structural attribute on the model recognition.

[0011] In some possible implementations, adjusting the model parameters of the student model based on the weighted value of the first distillation weight, the knowledge imitation loss, the second distillation weight, and the regularization loss, as well as the target recognition loss, includes: Based on the first distillation weight and the knowledge imitation loss, a first distillation loss is determined, and based on the second distillation weight and the regularization loss, a second distillation loss is determined; The total loss is obtained based on the first distillation loss, the second distillation loss, and the target recognition loss; If the total loss is determined to be greater than the convergence threshold, the model parameters of the student model are adjusted based on the total loss.

[0012] In some possible implementations, the method further includes: If the total loss is determined to be no greater than the convergence threshold, the trained student model is output, and the visual recognition model is obtained.

[0013] Secondly, embodiments of this application provide a visual recognition method applied to an image acquisition device deployed in an industrial setting. The image acquisition device is equipped with a trained visual recognition model. The method includes: Acquire an original image containing a target to be identified, which may include text or an object; The original image is sized to obtain a standard image, and the standard image is input into the visual recognition model to obtain the target recognition result of the original image. The visual recognition model is trained according to the method described in any one of the first aspects.

[0014] Thirdly, embodiments of this application provide a semantically aware model compression apparatus, comprising: The loss determination unit is used to input image samples from the training set into the teacher model and the student model to obtain the knowledge imitation loss, regularization loss and target recognition loss of the student model; The weight determination unit is used to determine the semantic soft label of the image sample based on the image sample and multiple descriptive texts, and to determine the first distillation weight and the second distillation weight based on the semantic soft label. The single descriptive text is used to describe an image degradation type or structural attribute of the image. The adjustment unit is used to adjust the model parameters of the student model based on the weighted values ​​of the first distillation weight, the knowledge imitation loss, the second distillation weight and the regularization loss, and the target recognition loss, so as to obtain the trained visual recognition model. The knowledge imitation loss includes the differences between the image features output by the corresponding intermediate layers in the teacher model and the student model, and the differences between the target recognition results output by the output layer. The regularization loss is determined based on the parameter distribution of multiple network layers in the student model, and the target recognition loss is determined based on the difference between the target recognition result output by the student model and the actual target label of the image sample.

[0015] In some possible implementations, the loss determination unit is specifically used for: The image samples in the training set are input into the teacher model to obtain the first image features output by multiple first intermediate layers and the first target recognition result output by the output layer in the teacher model. The image sample is input into the student model to obtain the second image features output by multiple second intermediate layers and the second target recognition result output by the output layer in the student model, as well as the parameter set of the multiple network layers. Each second intermediate layer corresponds to a first intermediate layer, and each network layer is any network layer in the student model. The knowledge imitation loss is determined based on the difference between each first image feature and the corresponding second image feature output by the layer, as well as the difference between the first target recognition result and the second target recognition result. The regularization loss is determined based on the parameter distribution in the parameter set of the target network layer, where the target network layer is any one of the plurality of network layers; The target recognition loss is determined based on the difference between the second target recognition result and the actual target label.

[0016] In some possible implementations, the weight determination unit is specifically used for: The image samples are image encoded to obtain the third image features; The third image features are compared with the text features of the multiple descriptive texts to obtain multiple similarity scores; The multiple similarities are normalized to obtain semantic attribute vectors, which are the semantic soft tags.

[0017] In some possible implementations, the weight determination unit is specifically used for: Based on the semantic soft tags, it is determined whether the image sample is a target image sample. The semantic soft tags of the target image sample contain N types of image degradation and the probability of each type of image degradation is greater than a first preset value, and / or M types of structural attributes and the probability of each structural attribute is greater than a second preset value. N and M are both positive integers greater than 1. If so, the first distillation weight and the second distillation weight are determined based on the total number of the first type of image degradation type in the semantic soft tag that is greater than the first preset value, and / or the total number of the second type of structural attribute that is greater than the second preset value. The first distillation weight is greater than the preset first benchmark weight and is positively correlated with the sum of the total number of the first type and the total number of the second type of the image sample. The second distillation weight is less than the preset second benchmark weight and is negatively correlated with the sum of the total number of the first type and the total number of the second type of the image sample. If not, the first benchmark weight is determined as the first distillation weight, and the second benchmark weight is determined as the second distillation weight.

[0018] In some possible implementations, the weight determination unit is specifically used for: The semantic soft labels are input into a preset neural network to obtain the first distillation weight and the second distillation weight; wherein, the neural network is trained on the neural network to be trained based on the semantic soft labels and supervision labels of multiple historical image samples, the semantic soft label of a single historical image sample is determined based on the similarity between the historical image sample and the multiple descriptive texts, and the supervision label of the historical image sample is determined based on the semantic soft labels of the historical image sample and the ranking result of the influence of each image degradation type or structural attribute on the model recognition.

[0019] In some possible implementations, the adjustment unit is specifically used for: Based on the first distillation weight and the knowledge imitation loss, a first distillation loss is determined, and based on the second distillation weight and the regularization loss, a second distillation loss is determined; The total loss is obtained based on the first distillation loss, the second distillation loss, and the target recognition loss; If the total loss is determined to be greater than the convergence threshold, the model parameters of the student model are adjusted based on the total loss.

[0020] In some possible implementations, the adjustment unit is further configured to: If the total loss is determined to be no greater than the convergence threshold, the trained student model is output, and the visual recognition model is obtained.

[0021] Fourthly, embodiments of this application provide a visual recognition device applied to an image acquisition device deployed in an industrial setting. The image acquisition device is equipped with a trained visual recognition model. The device includes: An image acquisition unit is used to acquire an original image containing a target to be identified, wherein the target to be identified includes text or an object; The target recognition unit is used to standardize the size of the original image to obtain a standard image, and input the standard image into the visual recognition model to obtain the target recognition result of the original image. The visual recognition model is trained according to the method described in any one of the first aspects.

[0022] Fifthly, embodiments of this application provide an electronic device, including: Memory is used to store computer programs or instructions; A processor for executing a computer program or instructions in the memory, such that the method described in either the first or second aspect above is performed.

[0023] Sixthly, embodiments of this application provide a computer-readable storage medium that, when instructions in the storage medium are executed by a processor, enables the processor to perform the method described in either the first or second aspect above.

[0024] In a seventh aspect, embodiments of this application provide a computer program product, the computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the method described in either the first or second aspect above.

[0025] In this embodiment, by introducing semantic soft labels, the model can automatically perceive the content characteristics of the input image samples, such as which image degradation types or structural attributes the image samples contain. Based on the content characteristics of the image samples, the distillation loss weights in the knowledge distillation process are dynamically adjusted, enabling the student model to learn efficiently even when it has many image degradation types or structural attribute characteristics. This improves the generalization ability of the trained visual recognition model. At the same time, based on the knowledge imitation loss, regularization loss is introduced to guide the student model parameters to be concentrated, which facilitates subsequent model pruning and quantization, significantly reduces the amount of inference computation and memory usage, and facilitates deployment on edge devices, such as image acquisition devices deployed in industrial scenarios. Attached Figure Description

[0026] Figure 1 This is a schematic diagram illustrating an application scenario of an optional semantically aware model compression method in this application embodiment; Figure 2 This is a schematic diagram of a network architecture for a teacher / student model in an embodiment of this application; Figure 3 This is a flowchart illustrating a semantically aware model compression method in an embodiment of this application. Figure 4 This is a schematic diagram illustrating the process of determining knowledge imitation loss, regularization loss, and target recognition loss in an embodiment of this application. Figure 5 This is a schematic diagram illustrating the logic for determining knowledge imitation loss in an embodiment of this application; Figure 6 This is a schematic diagram illustrating the logic for determining regularization loss in an embodiment of this application; Figure 7 This is a schematic diagram illustrating the logic for determining target recognition loss in an embodiment of this application; Figure 8 This is a schematic diagram of the architecture of a semantic encoder according to an embodiment of this application; Figure 9 This is a schematic diagram illustrating the process of determining a semantic soft tag in an embodiment of this application; Figure 10This is a schematic diagram of a model parameter adjustment process in an embodiment of this application; Figure 11 This is a flowchart illustrating a visual recognition method in an embodiment of this application; Figure 12 This is a schematic diagram of the logical architecture of a semantically aware model compression device according to an embodiment of this application; Figure 13 This is a schematic diagram of the logical architecture of a visual recognition device according to an embodiment of this application; Figure 14 This is a schematic diagram of the physical architecture of an optional electronic device in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0028] It should be noted that the term "and / or" in the specification, claims, and drawings of this application describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the related objects before and after it are in an "or" relationship.

[0029] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0030] The data collection, dissemination, and use in this application all comply with relevant national laws and regulations.

[0031] In related technologies, model compression methods typically employ a uniform supervised training strategy for all training samples. This results in a significant decrease in the recognition performance of visual recognition models (student models) when faced with complex image samples, such as those with severe image degradation or complex image structures, thus limiting the deployment of visual recognition models under diverse working conditions.

[0032] In view of this, this application provides a semantically aware model compression method. In this application, image samples from the training set are input into a teacher model and a student model to obtain the knowledge imitation loss, regularization loss, and target recognition loss of the student model. The knowledge imitation loss includes the difference between the image features output by the corresponding intermediate layers in the teacher model and the difference between the target recognition results output by the output layer in the student model. The regularization loss is determined based on the parameter distribution of multiple network layers in the student model. The target recognition loss is determined based on the difference between the target recognition result output by the student model and the actual target label of the image sample. Then, based on the image sample and multiple descriptive texts, the semantic soft label of the image sample is determined, and the first distillation weight and the second distillation weight are determined based on the semantic soft label. A single descriptive text is used to describe an image degradation type or structural attribute of the image. Then, based on the weighted values ​​of the first distillation weight, the knowledge imitation loss, the second distillation weight, and the regularization loss, as well as the target recognition loss, the model parameters of the student model are adjusted to obtain the trained visual recognition model.

[0033] Based on the above method, by introducing semantic soft labels, the model can automatically perceive the content characteristics of the input image samples, such as which image degradation types or structural attributes the image samples contain. Based on the content characteristics of the image samples, the distillation loss weights in the knowledge distillation process are dynamically adjusted, enabling the student model to learn efficiently even when it has many image degradation types or structural attribute characteristics. This improves the generalization ability of the trained visual recognition model. At the same time, regularization loss is introduced on the basis of knowledge imitation loss to guide the distribution of student model parameters to be concentrated, which facilitates subsequent model pruning and quantization, significantly reduces the amount of inference computation and memory usage, and facilitates deployment on edge devices, such as image acquisition devices deployed in industrial scenarios.

[0034] The methods described in this application can be applied to electronic devices. In some embodiments, the electronic device may be a server, such as a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0035] Figure 1 This diagram illustrates an application scenario of an optional semantically aware model compression method as described in this application. (See also...) Figure 1As shown, the application scenario includes server 10, which is used to execute the aforementioned method. Based on the training set, teacher model and student model in the database, the student model is distilled and trained to obtain a trained visual recognition model.

[0036] In this embodiment of the application, multiple historical images are pre-acquired using image acquisition devices deployed in an industrial setting. The dimensions of the multiple historical images are standardized to obtain image samples. The aforementioned training set is then constructed based on the multiple image samples and their respective actual target labels.

[0037] Taking a historical image as an example, the size standardization is explained. In some possible implementations, the historical image is scaled proportionally to the target height to obtain an intermediate image. It is then determined whether the width of the intermediate image is equal to the target width. If so, the intermediate image is the image sample of the historical image. If not, and it is determined that the width of the intermediate image is less than the target height, a preset pixel, such as white, is filled on one side (right or left) of the intermediate image to obtain the image sample of the historical image. If it is determined that the width of the intermediate image is greater than the target height, the intermediate image is scaled to the target width, and a preset pixel, such as white, is filled at the top or bottom of the scaled image to obtain the image sample of the historical image.

[0038] Optionally, preset pixels can be filled on the right side of the intermediate image or at the bottom of the aforementioned scaled image to maintain the integrity of the image content, ensure that key information remains unchanged during processing, avoid information loss, and achieve simple and efficient adjustment of image size to match the target size.

[0039] After introducing how the training set is constructed, the teacher model and student model are described as follows: In this embodiment, the teacher model can be a pre-trained network model with strong representational capabilities, and the student model is a lightweight network model to be trained.

[0040] In this embodiment of the application, both the teacher model and the student model can be deep neural network models used in industrial applications such as Optical Character Recognition (OCR), object detection, image classification, and semantic segmentation, including but not limited to these.

[0041] The deep neural network model is a feedforward neural network with multiple trainable hidden layers, including but not limited to convolutional neural networks (CNN), recurrent neural networks (RNN), Transformers and their variants.

[0042] In the embodiments of this application, see the following: Figure 2 As shown, both the teacher model and the student model can include the following modules connected in sequence: 1) Feature extraction module In some possible implementations of this application, the feature extraction module can be constructed using a convolutional neural network (CNN) backbone, Vision Transformer, etc.

[0043] 2) Feature enhancement module

[0044] In some possible implementations of this application, the feature enhancement module can be constructed using a Feature Pyramid Network (FPN), an Atrous Spatial Pyramid Pooling (ASPP), an attention structure, a Pyramid Attention Network (PAN), or the like.

[0045] 3) Feature Decoding Module

[0046] In some possible implementations of this application, the feature decoding module can be constructed using a recurrent neural network (RNN), a connectionist temporal classification head (CTC head), a detection head, etc.

[0047] In the specific embodiments, please refer to Figure 2 As shown, image samples from the training set are input into the teacher / student model. The feature extraction module in the model extracts image features, the feature enhancement module enhances the extracted image features, and the feature decryption module decodes the enhanced image features to output the target recognition result.

[0048] In this embodiment, the structures of the above modules in the teacher model and the student model may be the same or different. When the structures are different, the outputs of the corresponding intermediate layers need to be adapted and aligned, for example, through 1×1 convolution, interpolation, etc., to support feature comparison.

[0049] It should be noted that, Figure 1 , Figure 2 This application provides an example of the application scenarios and model structure of a semantically aware model compression method provided in the embodiments of this application, but this application is not limited thereto.

[0050] After introducing the application scenarios and model structures of the embodiments of this application, the preferred embodiments of this application will be further described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only for illustration and explanation of this application and are not intended to limit this application. Furthermore, the embodiments of this application and the features in the embodiments can be combined with each other without conflict.

[0051] See Figure 3 As shown in the figure, this application provides a semantically aware model compression method, the specific process of which may include, but is not limited to, the following: Step 300: Input the image samples in the training set into the teacher model and the student model to obtain the knowledge imitation loss, regularization loss and target recognition loss of the student model.

[0052] In this embodiment, the knowledge imitation loss may include the difference between the image features output by the corresponding intermediate layers in the teacher model and the difference between the target recognition results output by the output layer in the student model and the regularization loss is determined based on the parameter distribution of multiple network layers in the student model, and the target recognition loss is determined based on the difference between the target recognition result output by the student model and the actual target label of the image sample.

[0053] For specific implementation details, please refer to... Figure 4 As shown, when executing step 300, the following process can be specifically executed: Step 3001: Input the image samples in the training set into the teacher model to obtain the first image features output by multiple first intermediate layers and the first target recognition results output by the output layer in the teacher model.

[0054] In this embodiment of the application, when performing step 3001, refer to... Figure 2 The network architecture shown inputs image samples from the training set into the feature extraction module of the teacher model. After processing by the feature extraction module, feature enhancement module, and feature decoding module, the final result obtained is the first target recognition result output by the output layer of the teacher model, and the first image features output by multiple first intermediate layers in the teacher model, denoted as F. t1 F t2 F t3 , ...

[0055] Step 3002: Input the image sample into the student model to obtain the second image features output by multiple second intermediate layers in the student model and the second target recognition result output by the output layer, as well as the parameter set of multiple network layers. Here, a single second intermediate layer corresponds to a first intermediate layer, and a single network layer is any network layer in the student model.

[0056] In this embodiment of the application, similar to step 3001, when performing step 3002, refer to [reference needed]. Figure 2The network architecture shown inputs image samples into the feature extraction module of the student model. After processing by the feature extraction module, feature enhancement module, and feature decoding module, the second target recognition result output by the output layer of the student model and the second image features output by the second intermediate layer corresponding to the multiple first intermediate layers in the student model are obtained, denoted as F. s1 F s2 F s3 ... and parameter sets of multiple network layers, where the parameter set of any network layer can reflect the parameter distribution and sparsity of that network layer, which facilitates subsequent model pruning and quantization.

[0057] Step 3003: Determine the above knowledge imitation loss based on the difference between each first image feature and the corresponding layer output second image feature, as well as the difference between the first target recognition result and the second target recognition result.

[0058] In the embodiments of this application, see the following: Figure 5 As shown, when executing step 3003, the first image features output by the first intermediate layer of the target and the second image features output by the second intermediate layer of the target corresponding to the first intermediate layer of the target are compared to obtain the difference between the two. The first intermediate layer of the target is any one of the above-mentioned multiple first intermediate layers. In this way, the difference between the image features output by the corresponding intermediate layers of the teacher model and the student model can be obtained, which serves as the knowledge imitation loss component 1 for the distillation training of the student model.

[0059] By comparing the first and second target recognition results, the difference between the target recognition results output by the output layers of the teacher model and the student model can be obtained, which serves as the knowledge imitation loss component 2 for the student model distillation training. Then, the sum of knowledge imitation loss component 1 and knowledge imitation loss component 2 is determined as the knowledge imitation loss for the student model distillation training.

[0060] In this embodiment of the application, the knowledge imitation loss is a non-negative real number with a value range of [0, +∞).

[0061] In this embodiment, the aforementioned knowledge imitation loss is used to measure the difference in feature responses between the teacher model and the student model at intermediate layers, such as the deep output of the backbone network, the output of the feature fusion module, and the output of the feature decoding module. This loss is applied simultaneously to the target recognition results of the teacher / student models and multiple intermediate layers. By calculating the differences in target recognition results and the differences between image features (feature maps) of multiple corresponding intermediate layers (e.g., calculated using L2 distance), a joint imitation supervision signal is formed, thereby achieving deep knowledge transfer.

[0062] It should be noted that the intermediate layers involved in feature comparison mentioned above can be configurable and can be set according to task requirements, model structure, or hardware resources. For example, the output of the feature extraction module, the output of the feature enhancement module, or any combination thereof can be selected to calculate the knowledge imitation loss; another example is that the output of any network layer of the feature extraction module, the input of any network layer of the feature enhancement module, the output of any network layer of the feature decoding module, or any combination thereof can be selected; yet another example is convolutional feature layers, Transformer blocks, FPN / PAN structure feature layers, etc.

[0063] Step 3004: Determine the above regularization loss based on the parameter distribution in the parameter set of the target network layer, where the target network layer is any one of multiple network layers.

[0064] In the embodiments of this application, see the following: Figure 6 As shown, when performing step 3004, the parameter distribution of each network layer is analyzed based on the parameter set obtained in step 3002, thereby obtaining the regularization loss.

[0065] The regularization loss can be the L1 norm loss, which measures the parameter distribution of the network layers in the student model. It promotes the concentration and sparsity of parameters. Sparsity means that the weights of some neurons approach zero. By introducing the regularization loss, it is easier to prune and quantize in the future. For example, in the inference stage, it can be removed by pruning strategies or optimized by hardware instruction set, thereby reducing the computational load and memory usage of the model and improving the inference speed of edge devices, such as image acquisition devices deployed in industrial scenarios.

[0066] In this embodiment of the application, the value range of the above-mentioned regularization loss is [0, +∞).

[0067] It should be noted that the above network layers can be preset, and computationally intensive network layers in the student model can be selected, including but not limited to recurrent connection layers, fully connected layers, or attention parameter matrices.

[0068] Step 3005: Determine the target recognition loss based on the difference between the second target recognition result and the actual target label.

[0069] In the embodiments of this application, see the following: Figure 7 As shown, when performing step 3005, the second target recognition result is compared with the actual target label of the image sample to determine the difference between the two, and the target recognition loss is determined based on the difference. It measures the difference between the true value (actual target label) and the target recognition result of the student model in order to maintain the performance of the student model on the target recognition task.

[0070] In this embodiment of the application, the target recognition loss is a non-negative real number, with a value range of [0, +∞).

[0071] It should be noted that the second target recognition result can be determined according to the specific industrial vision recognition task, such as including but not limited to: image category label, target bounding box coordinates, pixel-level segmentation mask, anomaly detection confidence, etc.

[0072] Step 310: Based on the image sample and multiple descriptive texts, determine the semantic soft label of the image sample, wherein a single descriptive text is used to describe an image degradation type or structural attribute of the image.

[0073] In this embodiment, a semantic encoder is introduced to generate semantic soft tags that characterize the content features of the input image sample.

[0074] See Figure 8 As shown, the semantic encoder in this embodiment includes an image encoding submodule and a text candidate pool alignment module. The image encoding submodule is used to encode (extract features) the input image to obtain global image features. The text candidate pool alignment module is used to convert a preset set of natural language statements describing the image degradation type and structural attributes into text features to obtain multiple text features. The global image features and multiple text features are compared to generate semantic soft tags for the image.

[0075] In this embodiment, the semantic soft label is represented in vector form, denoted as semantic attribute vector. Each element in the semantic attribute vector represents the response intensity of the input image in the corresponding semantic dimension, that is, the probability that the input image has the corresponding image degradation type or structural attribute. It allows multiple dimensions to have high values ​​at the same time to represent the co-occurrence of multiple image degradations or complex structures in the image.

[0076] For specific implementation details, please refer to... Figure 9 As shown, the specific process for executing step 310 is as follows: Step 3101: Encode the image sample to obtain the third image feature.

[0077] In the embodiments of this application, please refer to Figure 8 The semantic encoder architecture shown in step 3101 inputs the image sample into the image encoding submodule of the semantic encoder. The image encoding submodule encodes the image sample to obtain the third image feature and transmits the third image feature to the text candidate pool alignment module.

[0078] Step 3102: Compare the third image features with the text features of multiple descriptive texts to obtain multiple similarity scores.

[0079] In the embodiments of this application, please refer to Figure 8 In the semantic encoder architecture shown, during step 3102, the text candidate pool alignment module compares the third image features with the text features of multiple descriptive texts to determine their similarity.

[0080] In specific implementation, taking the third image feature as a vector form and the text features of multiple descriptive texts as a matrix form as an example, when executing step 3102, the text candidate pool alignment module performs matrix multiplication operation on the third image feature vector and the text feature matrix corresponding to the text features of multiple descriptive texts to obtain the similarity vector.

[0081] Step 3103: Normalize the multiple similarities to obtain a semantic attribute vector, which is the aforementioned semantic soft label.

[0082] In the embodiments of this application, please refer to Figure 8 In the semantic encoder architecture shown, when executing step 3103, the text candidate pool alignment module independently applies the Sigmoid function to each dimension of the similarity vector, mapping it to the range [0, 1] to generate a normalized semantic attribute vector, which is the aforementioned semantic soft label.

[0083] In this embodiment, the Sigmoid function is used for independent normalization, which allows multiple semantic dimensions to be activated simultaneously (such as "fuzziness + corrosion"), more realistically reflecting the complex interference superposition in industrial scenarios and improving the expressive power of semantic soft labels.

[0084] Step 320: Determine the first distillation weight and the second distillation weight based on the semantic soft label.

[0085] In this embodiment of the application, in order to enable the model to dynamically adjust the contribution ratio of different supervised training signals based on the content features of the input image during distillation training, a loss weight control module is introduced. This module can determine the first distillation weight and the second distillation weight of the image sample based on the semantic soft label. The first distillation weight is used to adjust the aforementioned knowledge imitation loss, and the second distillation weight is used to adjust the aforementioned regularization loss.

[0086] In some embodiments of this application, when performing step 320, firstly, based on the above semantic soft tags, it is determined whether the image sample is a complex image sample. Then, based on different results (whether it is a complex image sample or not), different subsequent processing methods are adopted to determine the first distillation weight and the second distillation weight corresponding to the image sample. Among them, the semantic soft tags of the complex image sample contain N types of image degradation and the probability of each type of image degradation is greater than a first preset value, and / or M types of structural attributes and the probability of each structural attribute is greater than a second preset value, where N and M are both positive integers greater than 1.

[0087] In specific implementation, when determining whether an image sample is a complex image sample based on the aforementioned semantic soft labels, the aforementioned semantic soft labels can be directly input into the loss weight control module. The loss weight control module, based on the input semantic soft labels, counts the total number of the first type of various image degradation types that is greater than the aforementioned first preset value, denoted as n, and counts the total number of the second type of various structural attributes that is greater than the aforementioned second preset value, denoted as m, and compares them with N and M of the aforementioned complex image sample. If n is not less than N, or m is not less than M, or n is not less than N and m is not less than M, then the image sample is determined to be a complex image sample; otherwise, the image sample is determined not to be a complex image sample, i.e., it is an ordinary image sample.

[0088] It should be noted that the first preset value can be the same as or different from the second preset value. Of course, different preset values ​​can also be set for each image degradation type or structural attribute, such as image degradation type 1 - 0.5 (first preset value), image degradation type 2 - 0.6 (first preset value), ..., this application does not make specific limitations.

[0089] In some embodiments, when it is determined that the image sample is a complex image sample, the first distillation weight and the second distillation weight are further determined based on the total number of first types of image degradation types in the semantic soft tags that are greater than a first preset value, and / or the total number of second types of structural attributes that are greater than a second preset value. The first distillation weight is greater than a preset first benchmark weight and is positively correlated with the sum of the total number of first types and the total number of second types of the image sample. The second distillation weight is less than a preset second benchmark weight and is negatively correlated with the sum of the total number of first types and the total number of second types of the image sample.

[0090] In this embodiment, multiple sets of weight pairs can be preset, each set of weight pairs corresponds to a sum value, and a sum value can correspond to multiple sets of weight pairs. The difference lies in the number of types of image degradation or structural attributes. When it is determined that the image sample is a complex image sample, the first distillation weight and the second distillation weight can be obtained based on the total number of first types of image degradation types that are greater than a first preset value in the semantic soft label, and / or the total number of second types of structural attributes that are greater than a second preset value, as well as multiple sets of weight pairs. That is, a set of weight pairs corresponding to the first distillation weight and the second distillation weight are selected from the multiple sets of weight pairs.

[0091] In other embodiments, when it is determined that the image sample is not a complex image sample, that is, a normal image sample, the first reference weight can be determined as the first distillation weight, and the second reference weight can be determined as the second distillation weight.

[0092] In this embodiment of the application, a unified first benchmark weight and a second benchmark weight are pre-set for ordinary image samples, which are used for knowledge imitation loss and regularization loss, respectively. In specific implementation, when it is determined that the image sample is not a complex image sample, that is, an ordinary image sample, the pre-set first benchmark weight can be directly used as the first distillation weight, and the pre-set second benchmark weight can be used as the second distillation weight.

[0093] It should be noted that the sum of the first distillation weight and the second distillation weight can be 1 or not, depending on the specific needs. When the sum of the first distillation weight and the second distillation weight is 1, the weight pair may contain only one distillation weight, and the other distillation weight can be calculated.

[0094] In this way, by identifying whether the input image sample is a complex image sample and using different first and second distillation weights, the model can dynamically adjust the distillation weights based on semantic soft labels. When the input image sample is determined to be a complex image sample, the first distillation weight is increased as the total number of image degradation types and / or structural attributes of the input image sample increases. This makes the student model pay more attention to the knowledge imitation loss of the teacher model, thereby reducing the difference in knowledge response between the student model and the teacher model and improving the generalization ability of the student model. At the same time, for ordinary image samples, i.e., non-complex image samples, a unified first and second distillation weight is used for supervised training to ensure the model's target recognition accuracy for ordinary image samples.

[0095] In other embodiments of this application, the loss weight control module may include a neural network, a lightweight learnable network, such as a multilayer perceptron (MLP), for dynamically generating first and second distillation weights during the model's distillation training. The neural network includes an input layer, at least one hidden layer, and an output layer, wherein: The input is the semantic soft label of the image, denoted as . , of which each Indicates that the image exists in the first... The probability of each image degradation type, j∈[1, n]; the image degradation types include, but are not limited to, blur, overexposure, smudges, noise, or low contrast; The output consists of two scalars: the first distillation weight. Second distillation weight The weights are between 0 and 1, and are used to weight the knowledge imitation loss term and the model sparsity regularization loss term, respectively.

[0096] In a specific embodiment, when performing step 320, the semantic soft label can be input into the neural network to obtain the first distillation weight and the second distillation weight.

[0097] To ensure the rationality of the output of the loss weight control module, before implementation, the neural network to be trained is pre-trained based on the semantic soft labels and supervision labels of multiple historical image samples to obtain the neural network used in the implementation. The semantic soft label of a single historical image sample is determined based on the similarity between the historical image sample and the aforementioned multiple descriptive texts, and the supervision label of a single historical image sample is determined based on the semantic soft label of the historical image sample and the ranking results of the influence of each image degradation type or structural attribute on the model recognition.

[0098] The following uses text recognition tasks and image degradation types as examples to illustrate the training process of the neural network to be trained: 1. Data preparation stage: Multiple historical image samples were collected, and the ranking results of multiple experts (technical professionals) on the types of image degradation according to their impact on the performance of the text recognition model were obtained. For example, experts believed that "blurring" had the greatest impact on recognition performance, followed by "overexposure," then "smudged," and so on.

[0099] 2. Image degradation comprehensive score calculation stage: First, based on the ranking results of all experts, the Borda scoring method is used to calculate the average score for each type of image degradation.

[0100] For example, for The sorting of image degradation types, the first Award Points, that is, the first place gets Points, second place gets Points, ..., the last place gets 1 point.

[0101] Assuming there is a total Based on the sorting results, the image degradation type... The average Borda score is:

[0102] in, The average Borda score represents the j-th type of image degradation; k is the index, representing the k-th sorted result; This represents the Borda score of the j-th image degradation type in the k-th sorted result.

[0103] Then, Softmax normalization is performed on all average scores to obtain the image degradation type priority weights:

[0104] in, This represents the priority weight of the j-th image degradation type.

[0105] Next, the semantic soft labels for each historical image sample are... With priority weight We perform a weighted summation to obtain the overall image degradation score S:

[0106] 3. Supervision tag generation stage: Based on the overall score Generate supervised targets (labels):

[0107] in, Supervisory labels representing the knowledge imitation loss term; Supervisory labels representing the regularization loss term; For the Sigmoid function, For example, an adjustable scaling factor. etc., used to control the steepness of the mapping.

[0108] It should be noted that the above supervision labels satisfy the following prior: the more severe the sample degradation (the larger S is), the closer the weight of the first distillation is to 1 (i.e., the knowledge imitation loss is strengthened), and the closer the weight of the second distillation is to 0 (i.e., the sparsity regularization is suppressed), so as to avoid the decrease in accuracy due to excessive sparsity on poor samples.

[0109] 4. Model training phase: The loss weight control module is trained by minimizing the mean squared error (MSE) loss between the output and the supervision label:

[0110] in, , These are the output results of the neural network to be trained; , These are the generated supervision labels.

[0111] Once training is complete, the resulting neural network can generate appropriate first and second distillation weights in real time based on the semantic soft labels of the input image, which can then be used for knowledge distillation training of subsequent models.

[0112] In the above embodiments, taking image degradation types as an example, compared with the number of image degradation types in the previous embodiment as an indicator reflecting the complexity or degree of degradation of image samples, this embodiment uses the comprehensive image degradation score, combined with the probability of each image degradation type in the image and its priority weights on the model's recognition performance, to perform a weighted summation, thereby more accurately representing the sample difficulty. Based on this, the training is carried out so that the trained neural network can distinguish the differences in the degree of influence of different image degradation types on the recognition task, and better realize that the more severe the sample degradation, the larger the weight of the first distillation and the smaller the weight of the second distillation.

[0113] In this embodiment, different first distillation weights and second distillation weights are used for different image samples, so that the model can dynamically adjust the focus in distillation training according to the different image degradation types and / or structural attributes of the input image samples. When the image samples are severely degraded and structurally complex, the student model can pay more attention to the knowledge imitation loss of the teacher model, thereby reducing the knowledge response difference between the student model and the teacher model, and improving the generalization ability of the student model while ensuring the target recognition accuracy of the model.

[0114] Step 330: Based on the weighted values ​​of the first distillation weights, knowledge imitation loss, second distillation weights, and regularization loss, as well as the target recognition loss, adjust the model parameters of the student model to obtain the trained visual recognition model.

[0115] In this embodiment of the application, when performing step 330, refer to... Figure 10 As shown, the specific process is as follows: Step 3301: Determine the first distillation loss based on the first distillation weight and the knowledge imitation loss, and determine the second distillation loss based on the second distillation weight and the regularization loss.

[0116] In this embodiment of the application, when performing step 3301, the product of the first distillation weight and the knowledge imitation loss is determined as the first distillation loss, and the product of the second distillation weight and the regularization loss is determined as the second distillation loss.

[0117] Step 3302: Based on the first distillation loss, the second distillation loss, and the target recognition loss, obtain the total loss.

[0118] In this embodiment of the application, when performing step 3302, the sum of the first distillation loss, the second distillation loss, and the target recognition loss is determined as the total loss, or the weighted sum of the first distillation loss, the second distillation loss, and the target recognition loss is determined as the total loss. The weights of the first distillation loss, the second distillation loss, and the target recognition loss can be specifically set based on actual needs, or a neural network generation method similar to the aforementioned loss weight control module can be used to specifically generate the loss based on the aforementioned semantic soft tags. This application does not impose any specific limitations.

[0119] Step 3303: If the total loss is greater than the convergence threshold, then adjust the model parameters of the student model based on the total loss.

[0120] In this embodiment of the application, when performing step 3303, the model parameters of the student model can be adjusted based on the total loss using gradient descent or other methods. This application does not impose any specific limitations on this.

[0121] In this embodiment, the total loss consists of three parts: knowledge imitation loss, regularization loss, and target recognition loss. Through steps 3301 to 3303, the three losses are weighted and summed according to the dynamic loss weights, namely the first distillation weight and the second distillation weight, to obtain the total loss. When the total loss is determined to be greater than the convergence threshold, the model parameters of the student model are adjusted based on the total loss. Then, the next image sample in the training set is input into the teacher model and the student model, and the process of steps 300 to 330 is repeated. When the total loss is determined to be no greater than the convergence threshold, the trained student model is output, thus obtaining the above-mentioned visual recognition model.

[0122] In this embodiment of the application, by introducing semantic soft labels, the model can automatically perceive the content characteristics of the input image. For example, taking the text recognition model, by introducing semantic soft labels, the model can automatically perceive the content characteristics of the input image, such as whether it is blurry or whether there are rusted characters, and dynamically adjust the loss weight in the knowledge distillation process accordingly, so that the student model can learn efficiently on complex image samples and improve generalization performance.

[0123] Furthermore, based on the knowledge imitation loss, a sparsity regularization loss is introduced, which can guide the distribution of student model parameters to be concentrated during the distillation training process, which is convenient for subsequent pruning and quantization, significantly reduces the amount of inference computation and memory usage, and facilitates the deployment of the trained visual recognition model on edge devices.

[0124] In this embodiment of the application, after obtaining the trained visual recognition model, the visual recognition model can be deployed in the image acquisition device, which is an edge device, so that it can achieve high-precision visual recognition.

[0125] This application also provides a visual recognition method, which is applied to an image acquisition device deployed in an industrial setting. The image acquisition device is equipped with a visual recognition model trained based on the aforementioned model compression method. (See attached document.) Figure 11 As shown, the method includes the following steps: Step 1100: Acquire an original image containing the target to be identified, wherein the target to be identified includes text or an object.

[0126] The objects can be people, animals, plants, or workpieces that require defect detection, chips, etc. This application does not limit them. It can be seen that the "semantic-driven dynamic weighting" logic in the embodiments of this application can realize the training and optimization of the visual recognition model that needs to be deployed on edge devices (edge ​​computing devices), and has good versatility and engineering application value.

[0127] Step 1110: Standardize the size of the original image to obtain a standard image, and input the standard image into the above visual recognition model to obtain the target recognition result of the original image.

[0128] In some possible implementations of this application, when performing step 1110, the original image is scaled proportionally to the target height, and it is determined whether the width of the scaled image is equal to the target width. If so, the standard image is obtained. If not, when it is determined that the width of the scaled image is less than the target height, a preset pixel, such as white, is filled on the right side of the scaled image to obtain the standard image. When it is determined that the width of the scaled image is greater than the target height, it is further scaled to the target width, and a preset pixel, such as white, is filled at the bottom of the scaled image to obtain the standard image.

[0129] Then, the standard image is input into the above visual recognition model, and the target to be recognized in the standard image is recognized based on the visual recognition model to obtain the target recognition result of the original image.

[0130] Based on the same inventive concept, see [reference] Figure 12 As shown in the figure, this application provides a semantically aware model compression device, including: The loss determination unit 1210 is used to input image samples from the training set into the teacher model and the student model to obtain the knowledge imitation loss, regularization loss and target recognition loss of the student model; The weight determination unit 1220 is used to determine the semantic soft label of the image sample based on the image sample and multiple descriptive texts, and to determine the first distillation weight and the second distillation weight based on the semantic soft label. The single descriptive text is used to describe an image degradation type or structural attribute of the image. The adjustment unit 1230 is used to adjust the model parameters of the student model based on the weighted values ​​of the first distillation weight, the knowledge imitation loss, the second distillation weight and the regularization loss, and the target recognition loss, so as to obtain the trained visual recognition model. The knowledge imitation loss includes the differences between the image features output by the corresponding intermediate layers in the teacher model and the student model, and the differences between the target recognition results output by the output layer. The regularization loss is determined based on the parameter distribution of multiple network layers in the student model, and the target recognition loss is determined based on the difference between the target recognition result output by the student model and the actual target label of the image sample.

[0131] In some possible implementations, the loss determination unit 1210 is specifically used for: The image samples in the training set are input into the teacher model to obtain the first image features output by multiple first intermediate layers and the first target recognition result output by the output layer in the teacher model. The image sample is input into the student model to obtain the second image features output by multiple second intermediate layers and the second target recognition result output by the output layer in the student model, as well as the parameter set of the multiple network layers. Each second intermediate layer corresponds to a first intermediate layer, and each network layer is any network layer in the student model. The knowledge imitation loss is determined based on the difference between each first image feature and the corresponding second image feature output by the layer, as well as the difference between the first target recognition result and the second target recognition result. The regularization loss is determined based on the parameter distribution in the parameter set of the target network layer, where the target network layer is any one of the plurality of network layers; The target recognition loss is determined based on the difference between the second target recognition result and the actual target label.

[0132] In some possible implementations, the weight determination unit 1220 is specifically used for: The image samples are image encoded to obtain the third image features; The third image features are compared with the text features of the multiple descriptive texts to obtain multiple similarity scores; The multiple similarities are normalized to obtain semantic attribute vectors, which are the semantic soft tags.

[0133] In some possible implementations, the weight determination unit 1220 is specifically used for: Based on the semantic soft tags, it is determined whether the image sample is a target image sample. The semantic soft tags of the target image sample contain N types of image degradation and the probability of each type of image degradation is greater than a first preset value, and / or M types of structural attributes and the probability of each structural attribute is greater than a second preset value. N and M are both positive integers greater than 1. If so, the first distillation weight and the second distillation weight are determined based on the total number of the first type of image degradation type in the semantic soft tag that is greater than the first preset value, and / or the total number of the second type of structural attribute that is greater than the second preset value. The first distillation weight is greater than the preset first benchmark weight and is positively correlated with the sum of the total number of the first type and the total number of the second type of the image sample. The second distillation weight is less than the preset second benchmark weight and is negatively correlated with the sum of the total number of the first type and the total number of the second type of the image sample. If not, the first benchmark weight is determined as the first distillation weight, and the second benchmark weight is determined as the second distillation weight.

[0134] In some possible implementations, the weight determination unit 1220 is specifically used for: The semantic soft labels are input into a preset neural network to obtain the first distillation weight and the second distillation weight; wherein, the neural network is trained on the neural network to be trained based on the semantic soft labels and supervision labels of multiple historical image samples, the semantic soft label of a single historical image sample is determined based on the similarity between the historical image sample and the multiple descriptive texts, and the supervision label of the historical image sample is determined based on the semantic soft labels of the historical image sample and the ranking result of the influence of each image degradation type or structural attribute on the model recognition.

[0135] In some possible implementations, the adjustment unit 1230 is specifically used for: Based on the first distillation weight and the knowledge imitation loss, a first distillation loss is determined, and based on the second distillation weight and the regularization loss, a second distillation loss is determined; The total loss is obtained based on the first distillation loss, the second distillation loss, and the target recognition loss; If the total loss is determined to be greater than the convergence threshold, the model parameters of the student model are adjusted based on the total loss.

[0136] In some possible implementations, the adjustment unit 1230 is further configured to: If the total loss is determined to be no greater than the convergence threshold, the trained student model is output, and the visual recognition model is obtained.

[0137] Based on the same inventive concept, see [reference] Figure 13 As shown, this application embodiment provides a visual recognition device applied to an image acquisition device deployed in an industrial setting. The image acquisition device is equipped with a trained visual recognition model. The device includes: Image acquisition unit 1310 is used to acquire an original image containing a target to be identified, the target to be identified including text or an object; The target recognition unit 1320 is used to standardize the size of the original image to obtain a standard image, and input the standard image into the visual recognition model to obtain the target recognition result of the original image. The visual recognition model is trained according to any of the aforementioned semantic-aware model compression methods.

[0138] See Figure 14 As shown, this application provides an electronic device that can implement the functions of the aforementioned semantically aware model compression method or a visual recognition method. (Refer to...) Figure 14 The electronic device includes: at least one processor 141, and a memory 142 connected to at least one processor 141.

[0139] In this embodiment, the specific connection medium between the processor 141 and the memory 142 is not limited. Figure 14 The example shown is the connection between processor 141 and memory 142 via bus 140. Bus 140 is... Figure 14 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as a limitation.

[0140] In some embodiments, bus 140 can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 14 The term is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, processor 141 may also be referred to as controller; there is no restriction on the name.

[0141] In this embodiment, memory 142 stores instructions executable by at least one processor 141. By executing the instructions stored in memory 142, at least one processor 141 can perform any of the methods described above. Processor 141 can implement the functions of any of the aforementioned devices.

[0142] In one possible design, processor 141 may include one or more processing units. Processor 141 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 141. In some embodiments, processor 141 and memory 142 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0143] Processor 141 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of any method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0144] Memory 142, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 142 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 142 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 142 may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0145] By designing and programming the processor 141, the code corresponding to any of the methods described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute the steps of the semantically aware model compression method or the visual recognition method described in the foregoing embodiments during operation. How to design and program the processor 141 is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0146] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium that, when instructions in the storage medium are executed by a processor, enables the processor to perform any of the methods described in the above embodiments.

[0147] In some possible implementations, the semantically aware model compression method or visual recognition method provided in this application may also be implemented as a program product comprising program code that, when the program product is run on a device, causes the device to perform the steps of the methods described above according to the various exemplary embodiments of this application.

[0148] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flows in a flowchart and / or one or more blocks in a block diagram.

[0151] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0152] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A semantically aware model compression method, characterized in that, include: Image samples from the training set are input into the teacher model and the student model to obtain the knowledge imitation loss, regularization loss, and target recognition loss of the student model. Based on the image samples and multiple descriptive texts, semantic soft labels are determined for the image samples, and first distillation weights and second distillation weights are determined based on the semantic soft labels. A single descriptive text is used to describe an image degradation type or structural attribute of the image. Based on the weighted values ​​of the first distillation weight, the knowledge imitation loss, the second distillation weight, and the regularization loss, as well as the target recognition loss, the model parameters of the student model are adjusted to obtain the trained visual recognition model. The knowledge imitation loss includes the differences between the image features output by the corresponding intermediate layers in the teacher model and the student model, and the differences between the target recognition results output by the output layer. The regularization loss is determined based on the parameter distribution of multiple network layers in the student model, and the target recognition loss is determined based on the difference between the target recognition result output by the student model and the actual target label of the image sample.

2. The method as described in claim 1, characterized in that, The step of inputting image samples from the training set into the teacher model and the student model to obtain the knowledge imitation loss, regularization loss, and target recognition loss of the student model includes: The image samples in the training set are input into the teacher model to obtain the first image features output by multiple first intermediate layers and the first target recognition result output by the output layer in the teacher model. The image sample is input into the student model to obtain the second image features output by multiple second intermediate layers and the second target recognition result output by the output layer in the student model, as well as the parameter set of the multiple network layers. Each second intermediate layer corresponds to a first intermediate layer, and each network layer is any network layer in the student model. The knowledge imitation loss is determined based on the difference between each first image feature and the corresponding second image feature output by the layer, as well as the difference between the first target recognition result and the second target recognition result. The regularization loss is determined based on the parameter distribution in the parameter set of the target network layer, where the target network layer is any one of the plurality of network layers; The target recognition loss is determined based on the difference between the second target recognition result and the actual target label.

3. The method as described in claim 1, characterized in that, The step of determining the semantic soft label of the image sample based on the image sample and multiple descriptive texts includes: The image samples are image encoded to obtain the third image features; The third image features are compared with the text features of the multiple descriptive texts to obtain multiple similarity scores; The multiple similarities are normalized to obtain semantic attribute vectors, which are the semantic soft tags.

4. The method as described in claim 1, characterized in that, The determination of the first distillation weight and the second distillation weight based on the semantic soft label includes: Based on the semantic soft tags, it is determined whether the image sample is a target image sample. The semantic soft tags of the target image sample contain N types of image degradation and the probability of each type of image degradation is greater than a first preset value, and / or M types of structural attributes and the probability of each structural attribute is greater than a second preset value. N and M are both positive integers greater than 1. If so, the first distillation weight and the second distillation weight are determined based on the total number of the first type of image degradation type in the semantic soft tag that is greater than the first preset value, and / or the total number of the second type of structural attribute that is greater than the second preset value. The first distillation weight is greater than the preset first benchmark weight and is positively correlated with the sum of the total number of the first type and the total number of the second type of the image sample. The second distillation weight is less than the preset second benchmark weight and is negatively correlated with the sum of the total number of the first type and the total number of the second type of the image sample. If not, the first benchmark weight is determined as the first distillation weight, and the second benchmark weight is determined as the second distillation weight.

5. The method as described in claim 1, characterized in that, The determination of the first distillation weight and the second distillation weight based on the semantic soft label includes: The semantic soft labels are input into a preset neural network to obtain the first distillation weight and the second distillation weight; wherein, the neural network is trained on the neural network to be trained based on the semantic soft labels and supervision labels of multiple historical image samples, the semantic soft label of a single historical image sample is determined based on the similarity between the historical image sample and the multiple descriptive texts, and the supervision label of the historical image sample is determined based on the semantic soft labels of the historical image sample and the ranking result of the influence of each image degradation type or structural attribute on the model recognition.

6. The method according to any one of claims 1-5, characterized in that, The step of adjusting the model parameters of the student model based on the weighted value of the first distillation weight, the knowledge imitation loss, the second distillation weight, and the regularization loss, as well as the target recognition loss, includes: Based on the first distillation weight and the knowledge imitation loss, a first distillation loss is determined, and based on the second distillation weight and the regularization loss, a second distillation loss is determined; The total loss is obtained based on the first distillation loss, the second distillation loss, and the target recognition loss; If the total loss is determined to be greater than the convergence threshold, the model parameters of the student model are adjusted based on the total loss.

7. The method as described in claim 6, characterized in that, The method further includes: If the total loss is determined to be no greater than the convergence threshold, the trained student model is output, and the visual recognition model is obtained.

8. A visual recognition method, characterized in that, An image acquisition device deployed in industrial settings, the image acquisition device being equipped with a trained visual recognition model, the method comprising: Acquire an original image containing a target to be identified, which may include text or an object; The original image is sized to obtain a standard image, and the standard image is input into the visual recognition model to obtain the target recognition result of the original image. The visual recognition model is trained according to the method described in any one of claims 1-7.

9. A semantically aware model compression device, characterized in that, include: The loss determination unit is used to input image samples from the training set into the teacher model and the student model to obtain the knowledge imitation loss, regularization loss and target recognition loss of the student model; The weight determination unit is used to determine the semantic soft label of the image sample based on the image sample and multiple descriptive texts, and to determine the first distillation weight and the second distillation weight based on the semantic soft label. The single descriptive text is used to describe an image degradation type or structural attribute of the image. The adjustment unit is used to adjust the model parameters of the student model based on the weighted values ​​of the first distillation weight, the knowledge imitation loss, the second distillation weight and the regularization loss, and the target recognition loss, so as to obtain the trained visual recognition model. The knowledge imitation loss includes the differences between the image features output by the corresponding intermediate layers in the teacher model and the student model, and the differences between the target recognition results output by the output layer. The regularization loss is determined based on the parameter distribution of multiple network layers in the student model, and the target recognition loss is determined based on the difference between the target recognition result output by the student model and the actual target label of the image sample.

10. A visual recognition device, characterized in that, An image acquisition device deployed in industrial settings, the image acquisition device being equipped with a trained visual recognition model, the device comprising: An image acquisition unit is used to acquire an original image containing a target to be identified, wherein the target to be identified includes text or an object; The target recognition unit is used to standardize the size of the original image to obtain a standard image, and input the standard image into the visual recognition model to obtain the target recognition result of the original image. The visual recognition model is trained by the method according to any one of claims 1-7.

11. An electronic device, characterized in that, include: Memory is used to store computer programs or instructions; A processor for executing a computer program or instructions in the memory such that the method described in any one of claims 1-7 is performed.

12. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor, the processor is able to perform the method as described in any one of claims 1-7.

Images