Visual model compression method, device, equipment, storage medium and program product

Abstract

The present disclosure provides a visual model compression method, device, equipment, storage medium and program product, which relate to the field of artificial intelligence technology, specifically the field of computer vision and deep learning technology. A specific implementation of the method includes: inputting the input label of the visual model into the visual model; obtaining the output label of the i-th layer of the visual model; generating a first transposed label based on the output label of the i-th layer; reducing the dimension of the first transposed label to generate a reduced dimension label; inputting the reduced dimension label to the i+1th layer for transformation to generate the output label of the i+1th layer. This implementation provides a visual model compression method for reducing the dimension of the label dimension, which speeds up model training by reducing the number of labels.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technologies, specifically to the fields of computer vision and deep learning technologies. Background Art

[0002] Recently, the vision transformer has been greatly developed, and the vision transformer has achieved good model results in various vision field competitions. However, compared with CNN (Convolutional Neural Networks), the transformer generally requires huge computing power for inference and deployment. Therefore, it is urgent to miniaturize and compress the transformer.

[0003] The vision transformer is a multi-layer stacked model. When running, the picture is divided into image patches one by one (one image patch corresponds to one token). The more image patches, the higher the accuracy of the model, but it will also greatly increase the computational amount of the model. Summary of the Invention

[0004] Embodiments of the present disclosure propose a method, device, equipment, storage medium, and program product for visual model compression.

[0005] In a first aspect, embodiments of the present disclosure propose a method for visual model compression, including: inputting input tokens of a visual model into the visual model, where the number of tokens of the input tokens of the visual model is N, and the token dimension is C; obtaining output tokens of the i-th layer of the visual model, where i is a positive integer, and the number of tokens of the output tokens of the i-th layer is N, and the token dimension is C; generating a first transposed token based on the output tokens of the i-th layer, where the number of rows of the first transposed token is N, and the number of columns is C; performing dimensionality reduction on the first transposed token to generate a dimensionality-reduced token, where the number of tokens of the dimensionality-reduced token is M, and the token dimension is C, and M < N; inputting the dimensionality-reduced token into the (i + 1)-th layer for transformation to generate output tokens of the (i + 1)-th layer, where the number of tokens of the output tokens of the (i + 1)-th layer is M, and the token dimension is C.

[0006] Second aspect, an embodiment of the present disclosure provides a visual model compression device, including: an input module configured to input input tokens of a visual model into the visual model, where the number of tokens of the input tokens of the visual model is N, and the token dimension is C; an acquisition module configured to acquire output tokens of the i-th layer of the visual model, where i is a positive integer, the number of tokens of the output tokens of the i-th layer is N, and the token dimension is C; a generation module configured to generate first transposed tokens based on the output tokens of the i-th layer, where the number of rows of the first transposed tokens is N, and the number of columns is C; a dimensionality reduction module configured to perform dimensionality reduction on the first transposed tokens to generate dimensionality-reduced tokens, where the number of tokens of the dimensionality-reduced tokens is M, the token dimension is C, and M < N; a transformation module configured to input the dimensionality-reduced tokens into the (i + 1)-th layer for transformation to generate output tokens of the (i + 1)-th layer, where the number of tokens of the output tokens of the (i + 1)-th layer is M, and the token dimension is C.

[0007] Third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in any implementation manner of the first aspect.

[0008] Fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to cause a computer to execute the method described in any implementation manner of the first aspect.

[0009] Fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, where the computer program, when executed by a processor, implements the method described in any implementation manner of the first aspect.

[0010] The visual model compression method provided by the embodiments of the present disclosure provides a visual model compression method for dimensionality reduction in the token dimension, and speeds up model training by reducing the number of tokens.

[0011] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. Description of the Drawings

[0012] By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, objectives, and advantages of the present disclosure will become more obvious. The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them:

[0013] Figure 1is a flow chart of an embodiment of a visual model compression method according to the present disclosure;

[0014] Figure 2 It is a schematic diagram of the structure of the visual model;

[0015] Figure 3 It is a schematic diagram of the structure of each layer of the visual model;

[0016] Figure 4 is a flowchart of another embodiment of the visual model compression method according to the present disclosure;

[0017] Figure 5 It is a process diagram of aggregation operation;

[0018] Figure 6 It is a schematic diagram of the process of dimensionality reduction operation;

[0019] Figure 7 It is a schematic diagram of the structure of visual model compression;

[0020] Figure 8 is a structural schematic diagram of an embodiment of a visual model compression device according to the present disclosure;

[0021] Fig. 9 It is a block diagram of an electronic device used to implement the visual model compression method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] Figure 1 A process 100 of an embodiment of a visual model compression method according to the present disclosure is shown. The visual model compression method comprises the following steps:

[0025] Step 101 , inputting the input label [N, C] of the visual model into the visual model.

[0026] In this embodiment, the execution subject of the visual model compression method may input the input label [N, C] of the visual model into the visual model, where N is the number of labels and C is the label dimension.

[0027] Generally, the visual model can be various vision transformers, which are multi-layer stacked models with multiple layers stacked together. Each tag can correspond to an input position of the network. The tag can be a vector. N tags with a dimension of C can form a matrix tag [N, C] with N rows and C columns. After the tag [N, C] is input into the visual model, multiple layers of the visual model will process the tag [N, C] in turn. For each layer of the visual model, its input is the tag output by the previous layer, and its output is the tag input by the next layer. The tag output by the previous layer has more semantic information than the tag output by the next layer.

[0028] Generally, a visual model can be used to process images. An image is divided into N image blocks, and based on the N image blocks, a label [N, C] of the visual model can be generated. The image block can be a matrix, the label can be a vector generated based on the image block, and the image block and the label correspond one to one. The number N of image blocks divided from the image can be determined by the computational complexity of the model and the accuracy of the model. The larger N is, the higher the accuracy of the model is, but the larger the computational complexity of the model is, so the value of N can be determined by balancing the computational complexity of the model and the accuracy of the model. In some embodiments, linear mapping or leveling operations are performed on the N image blocks respectively to generate a label [N, C] of the visual model.

[0029] For ease of understanding, Figure 2 Figure 2 shows a schematic diagram of the structure of the visual model. Figure 2 As shown in the figure, the visual model can be a variety of vision transformers, including n stacked layers: layer 1, ..., layer n. The image is divided into N image blocks, each of which corresponds to an input position of the network. The N image blocks are linearly mapped or flattened to generate a label [N, C]. The label [N, C] is input to the n stacked layers for processing.

[0030] For ease of understanding, Figure 3 The schematic diagram of the structure of each layer of the visual model is shown. Figure 3As shown, the layer may include two normalization modules, an MHA (multi head self-attention) module and an MLP (Multilayer Perceptron) module. Among them, the normalization module can keep the input in the same distribution. The MHA module can be used to extract semantic information. The MLP module can be a two-layer fully connected layer for information induction and integration. The tag [N, C] is input to the first normalization module for normalization. The processing results are respectively input to the MHA module through the three heads of the MHA module for semantic information extraction. The extraction result is matrix-added with the tag [N, C], and the addition result is input to the second normalization module for normalization. The processing result is input to the MLP module for information induction and integration. The integration result is matrix-added with the result of the first matrix addition, and the tag [N, C] is output. The output tag [N, C] has more semantic information than the input tag [N, C].

[0031] For an image of size 224*224, a 12-layer vision transformer is used. The transformer encoding calculation process in the vision transformer is as follows:

[0032] Y=X+FC out (Attention(FC query (X), FC key (X), FC value (X)) (1)

[0033]

[0034] Z=Y+FC2(FC1(Y)) (3)

[0035] Among them, formula (1) and formula (2) are the calculation process of MHA, and formula (2) is the calculation process of MLP. X is the label [N, C] input to the first standardization module for standardization processing, and the output processing result, Q = FC query (X), K = FC key (X), V = FC value (X) is the intermediate output result of the three heads of the MHA module. FC out (Attention(FC query (X), FC key (X), FC value(X)) is the extraction result output by the MHA module. Y is the result of matrix addition of the extraction result and X. (FC1(Y) is the processing result output by the second normalization module. FC2(FC1(Y)) is the integration result output by the MLP module. Z is the result of matrix addition of the integration result and Y, that is, the output label [N,C].

[0036] Through rough derivation, we can know that the amount of transformer encoding calculation for each layer of vision transformer is:

[0037] 4×([N, D]×[D, D])=>4ND 2 Corresponding formula (1)

[0038] [N,D]×[D,N]+[N,N]×[N,D]=>2N 2 D corresponds to formula (2)

[0039] [N,D]×[D,4D]+[N,4D]×[4D,D]=>8ND 2 Corresponding formula (3)

[0040] Total calculation amount: 12ND 2 +2N 2 D

[0041] Where N is the number of image blocks (or the number of tags, or the length of the sequence), and D is the dimension of the tags (D can generally be changed to D = H*dim, where H is the number of MHA heads, dim is the dimension of a single head, and dim is d in formula (2)). It can be seen that by reducing the dimension of the tag, a linear acceleration corresponding to the number of tags can be achieved, that is, the more the tags are reduced, the faster the model inference will be.

[0042] Step 102, obtain the output label [N, C] of the i-th layer of the visual model.

[0043] In this embodiment, the execution subject may obtain the output label [N, C] of the i-th layer of the visual model, where i is a positive integer.

[0044] It should be noted that the i-th layer can be any layer of the visual model. If the visual model needs to be reduced in dimension multiple times, i can take any number of values, representing any number of layers of the visual model.

[0045] Step 103: Generate a first transposed label [N, C] based on the output label [N, C] of the i-th layer.

[0046] In this embodiment, the execution entity may generate a first transposition tag [N, C] based on the output tag [N, C].

[0047] Typically, the execution subject may directly use the output tag [N, C] as the first transposed tag [N, C], or may generate the first transposed tag [N, C] by processing the output tag [N, C]. For example, the output tag [N, C] is first transposed to generate the second transposed tag [C, N]; the second transposed tag [C, N] is aggregated to generate the aggregated tag [C, N]; the aggregated tag [C, N] is transposed to generate the first transposed tag [N, C].

[0048] Step 104: perform dimensionality reduction on the first transposed label [N, C] to generate a dimensionality reduction label [M, C].

[0049] In this embodiment, the execution subject may perform dimensionality reduction on the first transposed label [N, C] to generate a dimensionality reduction label [M, C]. Where M is the number of labels, and M <N。

[0050] Typically, the execution subject can select the dimension reduction label [M, C] from the first transposed label [N, C], or generate the dimension reduction label [M, C] by performing information aggregation on the first transposed label [N, C]. For example, information aggregation is performed through convolution operations or other pooling operations (such as average pooling). Among them, the convolution operation is usually used in conjunction with BN (batch normalization) and gelu activation function.

[0051] Step 105, input the dimension reduction label [M, C] to the i+1th layer for transformation to generate the output label [M, C] of the i+1th layer.

[0052] In this embodiment, the execution entity may input the dimension reduction label [M, C] to the i+1th layer for transformation to generate the output label [M, C] of the i+1th layer.

[0053] Usually, the dimension-reduced label [M, C] undergoes the transformation of the i+1th layer, that is, after the normalization module, the MHA module and the MLP module, the dimension of the output label [M, C] is still [M, C].

[0054] The visual model compression method provided by the embodiment of the present disclosure provides a visual model compression method for dimensionality reduction in the label dimension. It can be seen from the calculation process of the visual model that the model acceleration is linearly related to the number of labels, so the model training is accelerated by reducing the number of labels. The visual model compression method is mainly applied to various vision transformers stacked in multiple layers. The current face recognition business model is gradually migrating to the vision transformer. The vision transformer is generally better than CNN, but the vision transformer consumes a lot of time and needs further compression. The implementation of the vision transformer compression algorithm will help to greatly improve the accuracy of face recognition and other visual tasks.

[0055] Continue to refer Figure 4 , which shows a process 400 of another embodiment of a visual model compression method according to the present disclosure. The visual model compression method comprises the following steps:

[0056] Step 401 , inputting the input label [N, C] of the visual model into the visual model.

[0057] Step 402, obtain the output label [N, C] of the i-th layer of the visual model.

[0058] In this embodiment, the specific operations of steps 401-402 are already described in Figure 1 In the illustrated embodiment, steps 101 - 102 are described in detail and will not be repeated here.

[0059] Step 403, transpose the output label [N, C] of the i-th layer to generate a second transposed label [C, N].

[0060] In this embodiment, the execution subject of the visual model compression method can transpose the output label [N, C] to generate a second transposed label [C, N], wherein the row and column of the label [N, C] are interchanged to obtain the label [C, N], and the determinant of the label [C, N] remains unchanged.

[0061] Step 404 , performing an aggregation operation on the label dimension C in the second transposed label [C, N] as a sequence dimension to generate an aggregate label [C, N].

[0062] In this embodiment, the execution subject may perform an aggregation operation using the label dimension C in the second transposed label [C, N] as a sequence dimension to generate an aggregate label [C, N]. For example, the second transposed label [C, N] is convolved using the first convolution kernel to generate an aggregate label [C, N]. The aggregation operation may include convolution, batch normalization, and activation. The convolution kernel size of the first convolution kernel is determined by the number of labels to be locally fused, and the convolution step size is 1, which is used to fuse the information of the local labels. For example, if the information of the three local labels needs to be fully fused during sliding calculation, the convolution kernel size is set to 3.

[0063] For ease of understanding, Figure 5 A schematic diagram of the aggregation process is shown. Figure 5 As shown in the figure, taking the label [N, C] as an example, the convolution kernel size of the conv 1d convolution kernel is 3, the step size is 1, the number of input channels is 6, and the number of output channels is 6. Convolution is performed in the sliding direction from left to right to generate the aggregate label [N, C]. In the sliding calculation, the information of the three local labels needs to be fully integrated.

[0064] Step 405 , transpose the aggregated label [C, N] to generate a first transposed label [N, C].

[0065] In this embodiment, the execution subject may transpose the aggregated label [C, N] to generate a first transposed label [N, C], wherein the label [C, N] is obtained by exchanging the rows and columns of the label [N, C], and the determinant of the label [N, C] remains unchanged.

[0066] Step 406: Use the label dimension C in the first transposed label [N, C] as the channel dimension to perform a dimensionality reduction operation to generate a reduced-dimensionality label [M, C].

[0067] In this embodiment, the execution subject may use the label dimension C in the first transposed label [N, C] as the channel dimension to perform a dimensionality reduction operation to generate a reduced-dimensionality label [M, C]. For example, the first transposed label [N, C] is convolved with the second convolution kernel to generate a reduced-dimensionality label [M, C]. The dimensionality reduction operation includes convolution, batch normalization, and activation. The convolution kernel size of the second convolution kernel is 1, and the convolution step size is 1, which is used to reduce the number of labels.

[0068] For ease of understanding, Figure 6 Figure 2 shows a schematic diagram of the dimensionality reduction process. Figure 6 As shown in the figure, taking the label [N, C] as an example, the convolution kernel size of the conv 1d convolution kernel is 1, the step size is 1, the number of input channels is 9, and the number of output channels is 5. Convolution is performed in the sliding direction from bottom to top to generate the aggregate label [M, C]. Among them, M <N。

[0069] Step 407, input the dimension reduction label [M, C] to the i+1th layer for transformation to generate the output label [M, C] of the i+1th layer.

[0070] In this embodiment, the specific operation of step 406 is already described in Figure 1 In the illustrated embodiment, step 104 is described in detail and will not be repeated here.

[0071] from Figure 4 It can be seen that Figure 1 Compared with the corresponding embodiment, the process 400 of the visual model compression method in this embodiment adds the step of information aggregation. Therefore, the scheme described in this embodiment provides a visual model compression method for information aggregation and dimensionality reduction in the tag dimension, which retains more valid information through information aggregation, makes full use of the valid information of each tag, and eliminates invalid information. This improves the model accuracy. At the same time, by reducing the number of tags through dimensionality reduction, the model training is accelerated.

[0072] For ease of understanding, Figure 7 Figure 2 shows a schematic diagram of the structure of visual model compression. Figure 7 As shown, the visual model can be various vision transformers, including n stacked layers: layer 1, ..., layer i, layer i+1, ..., layer j, layer j+1, ..., layer n. The image is divided into N image blocks, each of which corresponds to an input position of the network. The N image blocks are linearly mapped or flattened to generate a label [N, C]. The label [N, C] is input to n stacked layers for processing. The information aggregation and dimensionality reduction modules can be inserted between any two layers of the vision transformer to find a balance between model accuracy and speed. Figure 7 The example inserts two (or more as needed) information aggregation and dimensionality reduction modules into the vision transformer, i.e., between layer i and layer i+1, and between layer j and layer j+1. In the information aggregation and dimensionality reduction module, there are mainly two modules: the aggregation module and the dimensionality reduction module. In order to coordinate the calculation of these two modules better, the transposition process is used in combination. In the aggregation module, conv1d, BN and gelu are used to aggregate the information flow. The aggregation operation treats the label dimension as the sequence dimension of the aggregation operation, so that the information of several local labels can be fully integrated during the sliding (step size = 1) calculation. In the dimensionality reduction module, the operation process is the same as that of the aggregation module, but this operation treats the label dimension as the channel dimension. In this way, the input and output dimensions can be set to achieve dimensionality reduction of the number of labels.

[0073] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides a first embodiment of a visual model compression device. Figure 1 corresponds to the method embodiment shown, and this device can be specifically applied to various electronic devices.

[0074] As Figure 8 shown, the visual model compression device 800 in this embodiment may include: an input module 801, an acquisition module 802, a generation module 803, a dimensionality reduction module 804, and a transformation module 805. Among them, the input module 801 is configured to input the input tokens of the visual model into the visual model, where the number of tokens of the input tokens of the visual model is N, and the token dimension is C; the acquisition module 802 is configured to acquire the output tokens of the i-th layer of the visual model, where i is a positive integer, and the number of tokens of the output tokens of the i-th layer is N, and the token dimension is C; the generation module 803 is configured to generate a first transposed token based on the output tokens of the i-th layer, where the number of rows of the first transposed token is N, and the number of columns is C; the dimensionality reduction module 804 is configured to perform dimensionality reduction on the first transposed token to generate a dimensionality reduction token, where the number of tokens of the dimensionality reduction token is M, the token dimension is C, and M < N; the transformation module 805 is configured to input the dimensionality reduction token into the (i + 1)-th layer for transformation to generate the output tokens of the (i + 1)-th layer, where the number of tokens of the output tokens of the (i + 1)-th layer is M, and the token dimension is C.

[0075] In this embodiment, in the visual model compression device 800: the specific processing of the input module 801, the acquisition module 802, the generation module 803, the dimensionality reduction module 804, and the transformation module 805 and the technical effects brought by them can be respectively referred to Figure 1 the relevant descriptions of steps 101-105 in the corresponding embodiment, which will not be elaborated here.

[0076] In some optional implementation manners of this embodiment, the generation module 803 includes: a first transpose sub-module, configured to transpose the output tokens of the i-th layer to generate a second transposed token, where the number of rows of the second transposed token is C, and the number of columns is N; an aggregation sub-module, configured to perform information aggregation on the second transposed token to generate an aggregation token, where the number of rows of the aggregation token is C, and the number of columns is N; a second transpose sub-module, configured to transpose the aggregation token to generate a first transposed token.

[0077] In some optional implementation manners of this embodiment, the aggregation sub-module includes: an aggregation unit, configured to perform an aggregation operation by using the token dimension C in the second transposed token as the sequence dimension to generate an aggregation token.

[0078] In some optional implementations of this embodiment, the aggregation unit is further configured to: use the first convolution kernel to convolve the second transposed mark to generate an aggregated mark, wherein the convolution kernel size of the first convolution kernel is determined by the number of locally fused marks, and the convolution step size is 1, which is used to fuse the information of the local marks.

[0079] In some optional implementations of this embodiment, the dimension reduction module 804 includes: a dimension reduction sub-module, configured to perform a dimension reduction operation on the label dimension C in the first transposed label as a channel dimension to generate a dimension reduction label.

[0080] In some optional implementations of this embodiment, the dimensionality reduction submodule is further configured to: use a second convolution kernel to convolve the first transposed label to generate a dimensionality reduction label, wherein the convolution kernel size of the second convolution kernel is 1 and the convolution step size is 1, which is used to reduce the number of labels.

[0081] In some optional implementations of this embodiment, the aggregation operation and the dimensionality reduction operation include convolution, batch normalization, and activation.

[0082] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0083] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0084] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0085] like Fig. 9As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0086] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0087] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the visual model compression method. For example, in some embodiments, the visual model compression method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the visual model compression method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the visual model compression method in any other appropriate manner (e.g., by means of firmware).

[0088] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0089] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0090] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0091] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0092] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0093] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0094] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions provided by this disclosure can be achieved, and this document does not limit this.

[0095] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A visual model compression method, comprising: Inputting input tags of a visual model into the visual model, wherein the image is divided into N image blocks, and based on the N image blocks, generating the input tags of the visual model, the number of tags of the input tags of the visual model is N, and the tag dimension is C; Obtain output tags of the i-th layer of the visual model, where i is a positive integer, the number of tags of the output tags of the i-th layer is N, and the tag dimension is C; Generate a first transposed label based on the output label of the i-th layer, wherein the number of rows and the number of columns of the first transposed label are N and C, respectively; Perform dimensionality reduction on the first transposed label to generate a reduced-dimensional label, wherein the number of labels of the reduced-dimensional label is M, the label dimension is C, and M <N; Input the dimension reduction label to the i+1th layer for transformation to generate the output label of the i+1th layer, wherein the number of labels of the output label of the i+1th layer is M and the label dimension is C; The step of generating a first transposed label based on the output label of the i-th layer includes: Transpose the output mark of the i-th layer to generate a second transposed mark, wherein the number of rows of the second transposed mark is C and the number of columns is N; Aggregating information on the second transposed mark to generate an aggregated mark, wherein the number of rows of the aggregated mark is C and the number of columns is N; The aggregation mark is transposed to generate the first transposed mark.

2. The method according to claim 1, wherein: The aggregating information of the second transposed mark to generate an aggregated mark includes: An aggregation operation is performed on the tag dimension C in the second transposed tag as a sequence dimension to generate the aggregated tag.

3. The method according to claim 2, wherein: The step of performing an aggregation operation on the tag dimension C in the second transposed tag as a sequence dimension to generate the aggregated tag includes: The second transposed mark is convolved with a first convolution kernel to generate the aggregated mark, wherein the convolution kernel size of the first convolution kernel is determined by the number of local fused marks, and the convolution step size is 1, which is used to fuse the information of the local marks.

4. The method according to claim 1, wherein: The step of reducing the dimension of the first transposed label to generate a reduced-dimensional label includes: A dimension reduction operation is performed on the label dimension C in the first transposed label as a channel dimension to generate the reduced dimension label.

5. The method according to claim 4, wherein: The step of performing a dimensionality reduction operation on the label dimension C in the first transposed label as a channel dimension to generate the dimensionality reduction label includes: The first transposed label is convolved with a second convolution kernel to generate the reduced-dimensional label, wherein the convolution kernel size of the second convolution kernel is 1 and the convolution step size is 1, so as to reduce the number of labels.

6. The method according to claim 3 or 5, wherein: The aggregation operation and the dimension reduction operation include convolution, batch normalization and activation.

7. A visual model compression device, comprising: An input module is configured to input input tags of a visual model into the visual model, wherein the image is divided into N image blocks, and the input tags of the visual model are generated based on the N image blocks, and the number of tags of the input tags of the visual model is N and the tag dimension is C; An acquisition module is configured to acquire output tags of the i-th layer of the visual model, wherein i is a positive integer, the number of tags of the output tags of the i-th layer is N, and the tag dimension is C; A generating module, configured to generate a first transposed label based on the output label of the i-th layer, wherein the number of rows of the first transposed label is N and the number of columns is C; A dimensionality reduction module is configured to reduce the dimension of the first transposed label to generate a reduced-dimensionality label, wherein the number of labels of the reduced-dimensional label is M, the label dimension is C, and M <N; A transformation module is configured to input the dimension reduction label to the i+1th layer for transformation to generate output labels of the i+1th layer, wherein the number of labels of the output labels of the i+1th layer is M and the label dimension is C; Wherein, the generating module comprises: A first transposition submodule is configured to transpose the output mark of the i-th layer to generate a second transposition mark, wherein the second transposition mark has C rows and N columns; an aggregation submodule, configured to aggregate information on the second transposed mark to generate an aggregated mark, wherein the number of rows of the aggregated mark is C and the number of columns is N; The second transposition submodule is configured to transpose the aggregated mark to generate the first transposition mark.

8. The device according to claim 7, wherein: The aggregation submodule includes: The aggregation unit is configured to perform an aggregation operation on the tag dimension C in the second transposed tag as a sequence dimension to generate the aggregated tag.

9. The device according to claim 8, wherein: The aggregation unit is further configured to: The second transposed mark is convolved with a first convolution kernel to generate the aggregated mark, wherein the convolution kernel size of the first convolution kernel is determined by the number of local fused marks, and the convolution step size is 1, which is used to fuse the information of the local marks.

10. The device according to claim 7, wherein: The dimension reduction module comprises: The dimension reduction submodule is configured to perform a dimension reduction operation on the label dimension C in the first transposed label as a channel dimension to generate the dimension reduction label.

11. The device according to claim 10, wherein: The dimension reduction submodule is further configured to: The first transposed label is convolved with a second convolution kernel to generate the reduced-dimensional label, wherein the convolution kernel size of the second convolution kernel is 1 and the convolution step size is 1, so as to reduce the number of labels.

12. The device according to claim 9 or 11, wherein: The aggregation operation and the dimension reduction operation include convolution, batch normalization and activation.

13. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images