Method, apparatus, device, storage medium and program product for constructing a model

Abstract

Embodiments of this disclosure relate to methods, apparatus, devices, storage media, and program products for constructing models. The method includes: determining a first feature map generated by a reference model based on sample data and a second feature map generated by a target model based on sample data; applying an attention mechanism to the first and second feature maps to determine a first weighted feature map and a second weighted feature map, the attention mechanism being associated with a channel dimension and / or a spatial dimension; determining a first loss based on a first difference between the first and second weighted feature maps; and adjusting parameters of the target model based on the first loss. In this manner, embodiments of this disclosure enable higher-quality model distillation.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein relate generally to the field of computers, and more particularly to methods, apparatus, devices, computer-readable storage media, and computer program products for constructing models. Background Technology

[0002] Model distillation is a model compression technique designed to transfer the knowledge of a large, complex model (often called the "teacher model") to a smaller, simpler model (called the "student model"). This process can improve the efficiency of the model, enabling it to run on resource-constrained devices, while preserving as much of the performance of the original model as possible. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for constructing a model is provided. The method includes: determining a first feature map generated by a reference model based on sample data and a second feature map generated by a target model based on sample data; applying an attention mechanism to the first and second feature maps to determine a first weighted feature map and a second weighted feature map, the attention mechanism being associated with a channel dimension and / or a spatial dimension; determining a first loss based on a first difference between the first and second weighted feature maps; and adjusting parameters of the target model based on the first loss.

[0004] In a second aspect of this disclosure, an apparatus for constructing a model is provided. The apparatus includes: a first determining module configured to determine a first feature map generated by a reference model based on sample data and a second feature map generated by a target model based on the sample data; a second determining module configured to apply an attention mechanism to the first feature map and the second feature map to determine a first weighted feature map and a second weighted feature map, the attention mechanism being associated with a channel dimension and / or a spatial dimension; a third determining module configured to determine a first loss based on a first difference between the first weighted feature map and the second weighted feature map; and an adjusting module configured to adjust parameters of the target model based on the first loss.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method of the first aspect.

[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram is shown in which an example system in which embodiments of the present disclosure may be implemented;

[0011] Figure 2 A schematic diagram of a multi-task model framework according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram of a characteristic distillation framework according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic diagram of a feature map recovery algorithm according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A schematic diagram of a multi-task model distillation algorithm framework according to some embodiments of the present disclosure is shown;

[0015] Figure 6 A schematic diagram illustrating an example process for constructing a model according to some embodiments of the present disclosure is shown;

[0016] Figure 7 A schematic structural block diagram of an example apparatus for building a model according to some embodiments of the present disclosure is shown; and

[0017] Figure 8 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0018] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0019] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0020] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0021] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0022] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0023] As mentioned above, model distillation is a model compression technique designed to transfer the knowledge of a large, complex model (often called the "teacher model") to a smaller, simpler model (called the "student model"). This process can improve the efficiency of the model, enabling it to run on resource-constrained devices while preserving as much of the original model's performance as possible. However, traditional direct distillation methods only utilize the output layer information of the teacher model, ignoring the feature representations of the intermediate layers of the teacher model. This results in traditional distillation methods neglecting the relationships between tasks and having weak generality.

[0024] Embodiments of this disclosure propose a scheme for constructing a model. The scheme involves determining a first feature map generated by a reference model based on sample data and a second feature map generated by a target model based on sample data; applying an attention mechanism to the first and second feature maps to determine a first weighted feature map and a second weighted feature map, wherein the attention mechanism is associated with the channel dimension and / or spatial dimension; determining a first loss based on a first difference between the first and second weighted feature maps; and adjusting the parameters of the target model based on the first loss.

[0025] In this way, embodiments of the present disclosure can provide a distillation method that utilizes an attention mechanism.

[0026] Therefore, the embodiments of this disclosure can efficiently adjust the parameters of the target model based on the attention mechanism.

[0027] Example build model process

[0028] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0029] First see Figure 1 The illustration shows a schematic diagram of an example system 100 for multitasking identification in which embodiments of the present disclosure may be implemented.

[0030] like Figure 1 As shown, system 100 may include training device 110, teacher network model 120 and student network model 130.

[0031] Training device 110 can be any device or apparatus capable of training a model.

[0032] The teacher network model 120 may include a model with a large number of parameters, while the student network model 130 may include a model with a small number of parameters. The training device 110 can use the teacher network model 120 to train the student network model 130 to obtain a trained student network model.

[0033] The following will detail the specific implementation of the training device 110, the teacher network model 120, and the student network model 130 in system 100.

[0034] Figure 2 A schematic diagram of a classification and detection process according to some embodiments of the present disclosure is shown. This process can be implemented in system 100. References are made below. Figure 1 The process is described below. The classification and detection process described above is illustrated using the classification and detection of traffic lights as an example.

[0035] like Figure 2 As shown, Figure 2 A multi-task model framework 200 for the classification and detection of traffic lights is shown.

[0036] The input image of the multi-task model framework 200 can be, for example, an image of a traffic light on a road, which may include, for example, the traffic light portion and the digital display portion. The system can be, for example, a system for recognizing the content of the traffic light portion and the digital display portion, and simultaneously involves the tasks of identifying the position of the traffic lights and numbers, as well as panel classification. The input image can be a scene related to traffic lights, containing both traffic lights and digital displays. The goal of this system is to simultaneously recognize the state of the traffic lights and the content of the digital displays, while also involving the tasks of identifying the position of the lights and numbers, as well as panel classification.

[0037] First, the input image is expanded to generate an expanded input image. This expanded input image may remove irrelevant image elements and center the traffic light position. Then, the input image undergoes a normalization process, such as using Z-score. Finally, the normalized input image is fed into a backbone network, which may be a residual network such as ResNet-18.

[0038] The output of the residual network is then fed into the Unified Feature Pyramid Network (U-FPN), and the output of the residual network is also subjected to global-average pooling.

[0039] The outputs of the unified feature pyramid network and the residual network after global average pooling are fed into a multi-task model for prediction and classification of the input image.

[0040] The multi-task model mentioned here will be referenced below. Figure 3 and Figure 4 Provide a detailed description.

[0041] In some embodiments, this multi-task model is used to process multiple recognition tasks associated with traffic lights. Each recognition task is executed by a separate task head. Specifically, in some embodiments, taking task attribute heads as an example, the three attribute heads include: a light and number attribute head: this task head is responsible for classifying the attributes of the traffic lights and numbers, i.e., recognizing the color of the light (red, yellow, or green), the type of the light (left turn, straight, right turn, round light, etc.), the color of the number, and the content of the number (such as countdown numbers). A light and number position head: this task head is responsible for predicting the position of the traffic lights and the displayed numbers, corresponding to a detection task, outputting their specific positions in the image in the form of bounding boxes. A panel classification head: this task head is responsible for classifying the panels of the traffic lights, classifying them according to different traffic light design styles or other features.

[0042] Corresponding to the three attribute headers mentioned above, the multiple recognition tasks include: A first recognition task, namely, a task identifying the attributes of the light and the numbers, used to identify the type of traffic light, such as whether the traffic light is red and / or what the countdown number is. A second recognition task, namely, a task identifying the position of the light and the position of the numbers, used to identify the position of the traffic light, such as the position of the light and the position of the displayed numbers. A third recognition task, namely, a classification and prediction task for the traffic light panel, used to identify element attributes in the traffic light, such as classifying the style of the traffic light based on its panel.

[0043] After processing by the three attribute heads mentioned above, each attribute head outputs its respective prediction result. For example, the light / digit attribute head outputs the traffic light status (red, yellow, green) and the digit recognition result. The light / digit location head outputs the location of the traffic light and the digit (possibly represented by bounding boxes). The panel classification head outputs the category of the traffic light panel.

[0044] However, the aforementioned multi-task models face the problem of foreground-background imbalance when processing input images, which affects not only object detection but also the effectiveness of knowledge distillation. To address this, this disclosure proposes a local distillation method that distills the foreground and background separately, using the spatial and channel attention of the teacher network as weights to guide the student network, thereby calculating the local distillation loss.

[0045] Since local distillation methods lack distillation of global information, this disclosure further proposes a global distillation method that utilizes a self-attention mechanism to extract global information from the student network and the teacher network respectively, thereby calculating the global distillation loss.

[0046] The model framework proposed in this disclosure only requires obtaining the feature maps of the student network and the teacher network to complete the calculation of local and global distillation losses, simplifying its application on various types of multi-task models.

[0047] According to the solution of this disclosure, in some embodiments, firstly, the training device 110 can determine a first feature map (which can be represented as F) generated by the reference model (i.e., the teacher network model 120) based on sample data. T The second feature map (which can be represented as F) generated by the target model (i.e., student network model 130) based on the sample data and the target model (i.e., student network model 130) S ).

[0048] In some embodiments, the target model may be, for example, a student network model 130, which (may also be referred to as a lightweight network or student network)... Figure 1 The traffic light multi-task model shown is an example of a multi-task model. The training device 110 inputs the normalized image into the student network, extracts features using the backbone residual network (ResNet-18), and simultaneously passes the outputs of the four stages of the backbone network through the network neck: the Unified Feature Pyramid Network (U-FPN) and Global-average pooling, before feeding them into the multi-task head.

[0049] In some embodiments, the classification task head is a linear classification head that outputs panel classifications, and its loss may include cross-entropy loss. The location head can use the positioning head in CenterNet to output the positions of the lights and numbers in the panel, and its loss may include heatmap loss, center point offset loss, and length and width loss from CenterNet. The attribute head is a linear classification head that outputs the color and category of the lights and numbers, and its loss may include cross-entropy loss.

[0050] In some embodiments, the reference model can be a teacher network model. The teacher network model 120 used in this invention (also referred to as a network model with a larger number of parameters or a teacher network) adopts the same design as the student network model, except that the backbone network with a larger number of parameters, stronger learning ability, and deeper network is replaced. The backbone network is replaced by SEResNet101. The subsequent processing flow is the same as that of the student network (the loss functions are also the same as those of the student network). The normalized image is input into the teacher network, and the backbone network extracts features separately. The outputs of the four stages of the backbone network are simultaneously passed through the Unified Feature Pyramid Network (U-FPN) and Global-average pooling, and then fed into the multi-task head.

[0051] In some embodiments, the first feature map may be a feature map F generated by the teacher network.T The second feature map can be the feature map F generated by the student network. S .

[0052] refer to Figure 3 , Figure 3 A characteristic loss distillation process 300 according to some embodiments of the present disclosure is described.

[0053] Model 110 is trained based on feature maps F generated by the teacher network. T The feature map F generated by the student network S Perform the following operations: Set F T and F S Characteristic distillation loss (L) kd_fea )Calculation. Then, train model 110 for the feature map F generated by the student network. S Perform the following transformations and operations.

[0054] In some embodiments, the training device 110 can train the first feature map (F) T ) and second feature map (F S An attention mechanism is applied to determine the first weighted feature map (F). out_t ) and the second weighted feature map (F out_s Attention mechanisms are associated with channel and / or spatial dimensions.

[0055] In some embodiments, the attention mechanism is associated with the channel dimension, and the training device 110 can also perform global average pooling and global max pooling on the first feature map or the second feature map to determine channel weight information corresponding to the channel dimension. The training device 110 can also determine a first weighted feature map (F) based on the channel weight information. out_t ) and the second weighted feature map (F out_s ).

[0056] For example, training device 110 can generate attention mechanism feature maps from teacher and student network feature maps. Taking the generation of attention mechanism feature maps from teacher and student network feature maps by training device 110 based on teacher network feature maps as an example, let the input feature map be F∈R. C×H×W Where C is the number of channels, and H and W are the height and width of the feature map, respectively.

[0057] In some embodiments, the training device 110 uses a channel attention mechanism and a spatial attention mechanism to generate attention weight maps: the channel attention mechanism uses global average pooling (GAP) and global max pooling (GMP) to compress the feature map in the spatial dimension to obtain the channel attention weights.

[0058] A c =σ(MLP(GAP(F)+GMP(F)))

[0059] Among them: A c ∈R C σ represents the channel attention weights, σ ​​represents the activation function (such as Sigmoid), and MLP represents a fully connected layer network.

[0060] In some embodiments, the attention mechanism can also be associated with spatial dimensions.

[0061] As an example, training device 110 can perform max pooling and average pooling on multiple channels of a first feature map or a second feature map to determine spatial weight information corresponding to the spatial dimension. Training device 110 can also base its training on spatial weight information (which can be represented as A). s ), determine the first weighted feature map and the second weighted feature map.

[0062] For example, training device 110 can use a spatial attention mechanism to perform max pooling and average pooling in the channel dimension, compressing the feature map into a single-channel spatial feature map, and obtaining spatial attention weights through convolutional layers:

[0063] A s =σ(Conv([MaxPool(F),AvgPool(F)]))

[0064] Among them: A s ∈R H×W σ represents the channel attention weights, σ ​​represents the activation function (such as Sigmoid), and Conv represents convolution.

[0065] In some embodiments, the training device 110 may perform a weighted operation between the generated attention weights and the original feature map to generate a new feature map.

[0066] In some embodiments, the training device 110 weights the channel attention:

[0067] F′=A c ·F

[0068] Among them: A c Extending to R via broadcast mechanism C×H×W .

[0069] Training device 110 weights spatial attention:

[0070] F″=A s ·F

[0071] Where A s Extending to R via broadcast mechanism C×H×W .

[0072] Training device 110 can combine channel attention and spatial attention to obtain a weighted final feature map:

[0073] F out =F′+F″

[0074] Final feature map F out Includes features emphasized by the attention mechanism (F out_t For the network attention feature map of teachers, F out_s This method generates a feature map of student network attention, which can be used in subsequent detection tasks. Through this approach, salient regions of the feature map are further enhanced by the attention mechanism.

[0075] In some embodiments, the training device 110 may also generate and recover a mask feature map based on the student network feature map.

[0076] In some embodiments, the training device 110 may be based on a preset mask (which may be represented as M∈{0,1}). H×W At least a portion of the second feature map is set to a preset value (i.e., set to 0 or 1) to obtain an intermediate feature map (which can be represented as F). masked ).

[0077] In some examples, let the input feature map be F∈R C×H×W Where C is the number of channels, and H and W are the height and width of the feature map, respectively.

[0078] In some embodiments, the training device 110 generates a random occlusion mask. Specifically, a random occlusion mask M∈{0,1} H×W It is applied to the feature map. This mask randomly selects some regions (set to 0) for occlusion, while leaving other regions unchanged (set to 1).

[0079] Furthermore, the training device 110 can apply an occlusion mask. For example, the training device 110 applies a generated random mask to the feature map to generate an occluded feature map F. masked :

[0080] F masked =M·F

[0081] M is broadcast to the same dimension C×H×W as the feature map F, the occluded region becomes 0, and the other regions retain their original values.

[0082] refer to Figure 4 , Figure 4 An example procedure 400 for feature map recovery is illustrated. As shown, the training device 110 can acquire the target model based on intermediate feature maps (F... masked The third feature map (F) generated by ) recoveredThe training device 110 can also be based on a fourth difference between the first feature map and the third feature map (i.e., the difference between the first feature map and the point feature map) and the first difference (which can be represented as...). The training device 110 can also adjust the parameters of the target model based on the first loss, thereby determining the first loss.

[0083] In some embodiments, the training device 110 can perform convolution operations to generate recovery features.

[0084] For example, training device 110 uses two 3x3 convolutional layers and a ReLU activation function to recover occluded features. Assuming the weights of the first convolutional layer are W1 and the bias is b1, the feature map F_1 generated after the first convolution and activation operation is represented as:

[0085] F1 = ReLU(Conv(F) masked ,W1)+b1)

[0086] The training device 110 further convolutions to generate the final recovered features. Specifically, the training device 110 generates the final recovered feature map F through a second convolutional layer. recovered Assuming the weights of the second convolutional layer are W2 and the bias is b2, then the formula is:

[0087] F recovered =Conv(F1,W2)+b2

[0088] In some embodiments, the training device 110 may be based on a first weighted feature map, and a first probability distribution (P) in the spatial dimension and / or channel location. student (i,j,c)). Training device 110 can also be based on a second weighted feature map, with a second probability distribution (P) in the spatial dimension and / or channel location. teacher (i,j,c)).

[0089] The training device 110 can determine the distillation loss based on the obtained attention feature map and the student network mask recovery feature map.

[0090] F has been obtained above out_t F out_s Attention feature map, F recovered The feature map of the student network is recovered from the mask, and the original feature map of the teacher network is F. teacher ∈R C×H×W .

[0091] In some embodiments, KL divergence is used to measure the similarity between two probability distributions. To compute KL Loss, the feature map must first be converted into a probability distribution. Typically, the feature map is represented as a probability distribution by a softmax operation.

[0092] To represent the feature map as a probability distribution, a softmax operation is performed at each location of the feature map. For the feature map F recovered by the student model... recovered Feature graph F of the teacher network teacher Softmax is defined as (F out_t F out_s The calculations are similar, corresponding to F respectively. teacher and F recovered (I will not go into details here):

[0093]

[0094]

[0095] Among them, P student (i,j,c) represents the probability distribution of the student model at the c-th channel at position (i,j), P teacher (i,j,c) represents the probability distribution of the teacher network in the same location and channel.

[0096] In some embodiments, the training device 110 may determine the first difference (which may be represented as) based on the difference between the first probability distribution and the second probability. ).

[0097] For example, the KL divergence loss measures the distance between two probability distributions, as shown in the following formula:

[0098]

[0099] This formula represents the calculation of the KL divergence between the probability distributions of the teacher network and the student network at each position (i,j) and each channel c in the feature map. The KL Loss can be minimized through backpropagation, allowing the feature map generated by the student model to gradually approach the feature map of the teacher network.

[0100] Training device 110 can also train student networks based on a multi-task distillation algorithm, the training process of which will be described in detail below. Figure 5 Describe it.

[0101] refer to Figure 5 , Figure 5 The process 500 of training a student network using the multi-task distillation algorithm of the present invention is shown.

[0102] In some embodiments, the training device 110 can load a trained teacher network model (with fixed parameters that do not participate in training). The training set is simultaneously processed through the teacher network backbone, the student network backbone, and the network neck, resulting in the feature map generated in the final stage of U-FPN. The teacher network can generate feature map F. TStudent networks can generate feature maps F S .

[0103] In some embodiments, the training device 110 may be based on F T and F S Determine the first loss, which includes the characteristic distillation loss (L). kd_fe For example, F T and F S Characteristic distillation loss (L) kd_fea )calculate.

[0104] In some embodiments, the training device 110 can determine the first classification information (T) generated by the reference model (i.e., the teacher network) based on the sample data. i The target model (i.e., the student network) generates second classification information (S) based on sample data. i For example, let T be the classification output of the teacher network and the student network respectively. i S i (represents the predicted output for class i, and the first classification information can be represented as T) i The second category information can be represented as S i ).

[0105] In some embodiments, the training device 110 may be based on the first classification information (T) i ) and the second classification information (S) i The second difference, determining the second loss (L) kd_cls The training device 110 can also adjust the parameters of the target model based on the first loss and the second loss. For example, T... i and S i Classification of distillation losses (L) kd_cls ) Calculation. Then, the output losses (L) of each item in the student network multi-task head are calculated. multi The calculation is then performed. Finally, the various losses are summed to train the backpropagation optimization.

[0106] Training device 110 can determine a third loss (L) based on a third difference between the second classification information and reference classification information (which may correspond to labeled information). multi The training device 110 can also adjust the parameters of the target model based on the first loss, the second loss, and the third loss.

[0107] The total training loss can be expressed as L = L multi +L kd_fea +L kd_cls The training device 110 can also adjust the parameters of the target model based on the first loss, the second loss, and the third loss.

[0108] Based on the process described above, embodiments of this disclosure can provide a distillation method that utilizes an attention mechanism.

[0109] Therefore, the embodiments of this disclosure can efficiently and accurately adjust the parameters of the target model based on an attention mechanism and by referencing differences in intermediate feature maps. This improves the recognition efficiency and accuracy of the target model.

[0110] Example process

[0111] Figure 6 A flowchart of an example process 600 for building a model according to some embodiments of the present disclosure is shown. Process 600 can be implemented at training device 110. Reference is made below. Figure 1 To describe process 600.

[0112] As shown in the figure, in box 610, training device 110 determines a first feature map generated by the reference model based on sample data and a second feature map generated by the target model based on sample data.

[0113] In box 620, training device 110 applies an attention mechanism to the first feature map and the second feature map to determine the first weighted feature map and the second weighted feature map, the attention mechanism being associated with the channel dimension and / or spatial dimension.

[0114] In box 630, training device 110 determines a first loss based on a first difference between a first weighted feature map and a second weighted feature map.

[0115] In box 640, training device 110 adjusts the parameters of the target model based on the first loss.

[0116] In some embodiments, adjusting the parameters of the target model based on the first loss further includes: determining first classification information generated by the reference model based on sample data and second classification information generated by the target model based on sample data; determining a second loss based on a second difference between the first classification information and the second classification information; and adjusting the parameters of the target model based on the first loss and the second loss.

[0117] In some embodiments, adjusting the parameters of the target model based on the first loss and the second loss further includes: determining a third loss based on a third difference between the second classification information and the reference classification information; and adjusting the parameters of the target model based on the first loss, the second loss, and the third loss.

[0118] In some embodiments, determining the first loss based on a first difference between a first weighted feature map and a second weighted feature map includes: setting at least a portion of the second feature map to a preset value based on a preset mask to obtain an intermediate feature map; obtaining a third feature map generated by the target model based on the intermediate feature map; and determining the first loss based on a fourth difference between the first feature map and the third feature map and the first difference.

[0119] In some embodiments, process 600 further includes: a first probability distribution in spatial dimensions and / or channel locations based on a first weighted feature map; a second probability distribution in spatial dimensions and / or channel locations based on a second weighted feature map; and determining a first difference based on the difference between the first probability distribution and the second probability.

[0120] In some embodiments, the attention mechanism is associated with a spatial dimension, and applying the attention mechanism to a first feature map and a second feature map includes: performing max pooling and average pooling on multiple channels of the first feature map or the second feature map to determine spatial weight information corresponding to the spatial dimension; and determining a first weighted feature map and a second weighted feature map based on the spatial weight information.

[0121] In some embodiments, the attention mechanism is associated with the channel dimension, and applying the attention mechanism to the first feature map and the second feature map includes: performing global average pooling and global max pooling on the first feature map or the second feature map to determine channel weight information corresponding to the channel dimension; and determining the first weighted feature map and the second weighted feature map based on the channel weight information.

[0122] In some embodiments, the target model is used to process multiple recognition tasks associated with traffic lights, including a number of the following: a first recognition task for recognizing the type of traffic light; a second recognition task for recognizing the location of the traffic light; and a third recognition task for recognizing element attributes in the traffic light.

[0123] Example apparatus and devices

[0124] Figure 7 A schematic structural block diagram of an example device 700 for building a model according to some embodiments of the present disclosure is shown. Device 700 may be implemented as or included in system 100. Various modules / components in device 700 may be implemented by hardware, software, firmware, or any combination thereof.

[0125] like Figure 7As shown, the apparatus 700 includes a first determining module 710 configured to determine a first feature map generated by a reference model based on sample data and a second feature map generated by a target model based on sample data; a second determining module 720 configured to apply an attention mechanism to the first feature map and the second feature map to determine a first weighted feature map and a second weighted feature map, wherein the attention mechanism is associated with the channel dimension and / or spatial dimension; a third determining module 730 configured to determine a first loss based on a first difference between the first weighted feature map and the second weighted feature map; and an adjusting module 740 configured to adjust the parameters of the target model based on the first loss.

[0126] In some embodiments, the adjustment module 740 is further configured to: determine first classification information generated by the reference model based on sample data and second classification information generated by the target model based on sample data; determine a second loss based on a second difference between the first classification information and the second classification information; and adjust the parameters of the target model based on the first loss and the second loss.

[0127] In some embodiments, the adjustment module 740 is further configured to: determine a third loss based on a third difference between the second classification information and the reference classification information; and adjust the parameters of the target model based on the first loss, the second loss, and the third loss.

[0128] In some embodiments, the third determining module 730 is further configured to: set at least a portion of the second feature map to a preset value based on a preset mask to obtain an intermediate feature map; obtain a third feature map generated by the target model based on the intermediate feature map; and determine a first loss based on a fourth difference and a first difference between the first feature map and the third feature map.

[0129] In some embodiments, the apparatus 700 further includes a probability distribution determination module configured to: determine a first probability distribution in spatial dimensions and / or channel locations based on a first weighted feature map; determine a second probability distribution in spatial dimensions and / or channel locations based on a second weighted feature map; and determine a first difference based on the difference between the first probability distribution and the second probability.

[0130] In some embodiments, the attention mechanism is associated with a spatial dimension, and applying the attention mechanism to a first feature map and a second feature map includes: performing max pooling and average pooling on multiple channels of the first feature map or the second feature map to determine spatial weight information corresponding to the spatial dimension; and determining a first weighted feature map and a second weighted feature map based on the spatial weight information.

[0131] In some embodiments, the attention mechanism is associated with the channel dimension, and applying the attention mechanism to the first feature map and the second feature map includes: performing global average pooling and global max pooling on the first feature map or the second feature map to determine channel weight information corresponding to the channel dimension; and determining the first weighted feature map and the second weighted feature map based on the channel weight information.

[0132] In some embodiments, the target model is used to process multiple recognition tasks associated with traffic lights, including a number of the following: a first recognition task for recognizing the type of traffic light; a second recognition task for recognizing the location of the traffic light; and a third recognition task for recognizing element attributes in the traffic light.

[0133] The modules included in device 700 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the modules in device 700 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0134] Figure 8 A block diagram of an electronic device 800 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 8 The electronic device 800 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 800 can be used to implement... Figure 1 System 100.

[0135] like Figure 8 As shown, electronic device 800 is in the form of a general-purpose electronic device. Components of electronic device 800 may include, but are not limited to, one or more processors or processing units 810, memory 820, storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. Processing unit 810 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 820. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 800.

[0136] Electronic device 800 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 820 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 830 can be a removable or non-removable medium and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within electronic device 800.

[0137] Electronic device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 8 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 820 may include computer program product 825 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0138] The communication unit 840 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 800 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 800 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0139] Input device 850 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 850 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 800 can also communicate with one or more external devices (not shown) via communication unit 840 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 800, or with any device that enables electronic device 800 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0140] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores one or more computer instructions, wherein one or more computer instructions are executed by a processor to implement the methods described above.

[0141] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0142] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0143] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0144] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0145] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the implementations disclosed herein.

Claims

1. A method for constructing a model, comprising: Determine a first feature map generated by the reference model based on the sample data and a second feature map generated by the target model based on the sample data; An attention mechanism is applied to the first feature map and the second feature map to determine a first weighted feature map and a second weighted feature map, wherein the attention mechanism is associated with the channel dimension and / or the spatial dimension; A first loss is determined based on a first difference between the first weighted feature map and the second weighted feature map; as well as Based on the first loss, the parameters of the target model are adjusted.

2. The method according to claim 1, wherein adjusting the parameters of the target model based on the first loss further includes: Determine the first classification information generated by the reference model based on the sample data and the second classification information generated by the target model based on the sample data; Based on the second difference between the first classification information and the second classification information, a second loss is determined; Based on the first loss and the second loss, the parameters of the target model are adjusted.

3. The method according to claim 2, wherein adjusting the parameters of the target model based on the first loss and the second loss further includes: Based on the third difference between the second classification information and the reference classification information, the third loss is determined; as well as The parameters of the target model are adjusted based on the first loss, the second loss, and the third loss.

4. The method of claim 1, wherein determining the first loss based on the first difference between the first weighted feature map and the second weighted feature map comprises: Based on a preset mask, at least a portion of the second feature map is set to a preset value to obtain an intermediate feature map; Obtain the third feature map generated by the target model based on the intermediate feature map; as well as The first loss is determined based on the fourth difference between the first feature map and the third feature map and the first difference.

5. The method according to claim 1, further comprising: Based on the first weighted feature map, a first probability distribution is established in the spatial dimension and / or the channel location; Based on the second weighted feature map, a second probability distribution is established in the spatial dimension and / or at the channel location; and The first difference is determined based on the difference between the first probability distribution and the second probability.

6. The method of claim 1, wherein the attention mechanism is associated with the spatial dimension, and applying the attention mechanism to the first feature map and the second feature comprises: Max pooling and average pooling are performed on multiple channels of the first feature map or the second feature map to determine spatial weight information corresponding to the spatial dimension; as well as Based on the spatial weight information, the first weighted feature map and the second weighted feature map are determined.

7. The method of claim 1, wherein the attention mechanism is associated with the channel dimension, and applying the attention mechanism to the first feature map and the second feature map comprises: Perform global average pooling and global max pooling on the first feature map or the second feature map to determine the channel weight information corresponding to the channel dimension; as well as Based on the channel weight information, the first weighted feature map and the second weighted feature map are determined.

8. The method of claim 1, wherein the target model is used to process multiple recognition tasks associated with traffic lights, the multiple recognition tasks including a number of the following: The first identification task is to identify the type of the traffic light; The second identification task is to identify the position of the traffic light; The third identification task is to identify the element attributes in the traffic lights.

9. An apparatus for constructing a model, comprising: The first determining module is configured to determine a first feature map generated by the reference model based on the sample data and a second feature map generated by the target model based on the sample data. The second determining module is configured to apply an attention mechanism to the first feature map and the second feature map to determine a first weighted feature map and a second weighted feature map, wherein the attention mechanism is associated with a channel dimension and / or a spatial dimension. The third determining module is configured to determine a first loss based on a first difference between the first weighted feature map and the second weighted feature map; as well as The adjustment module is configured to adjust the parameters of the target model based on the first loss.

10. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 8.

12. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 8.

Images