Data processing method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a data processing method, electronic equipment and a computer readable storage medium, and belongs to the technical field of communication. The data processing method comprises the steps that to-be-processed data is acquired; inputting the to-be-processed data into a data processing model, and outputting to obtain a data processing result; wherein the data processing model comprises a CNN (Convolutional Neural Network) model and a base model, the CNN model is connected with the base model in a residual error connection mode, and the base model is a transformer model realized based on a VIT (Virtual Information Technology) mode.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a data processing method, electronic device, and computer-readable storage medium. Background Technology

[0002] In the past few years, multimodal models based on the Vision Transformer (VIT) architecture have shone brightly in the field of data processing due to their excellent performance. For example, multimodal models based on the VIT architecture can classify and detect zero-shot data (such as image data) in some common scenarios.

[0003] However, in the aforementioned data processing scheme, the multimodal model based on the VIT architecture has limited learning effect on the local features of the data to be processed, resulting in poor data processing performance. Summary of the Invention

[0004] The purpose of this application is to provide a data processing method, electronic device, and computer-readable storage medium that can improve data processing performance.

[0005] In a first aspect, a data processing method is provided, comprising: acquiring data to be processed; inputting the data to be processed into a data processing model to output a data processing result; wherein the data processing model includes a CNN model and a base model, the CNN model and the base model are connected through a residual connection, and the base model is a transformer model implemented based on the VIT method.

[0006] Secondly, a data processing device is provided, comprising: a first processing module for acquiring data to be processed, and a second processing module for inputting the data to be processed into a data processing model to output a data processing result; wherein the data processing model includes a CNN model and a base model, the CNN model and the base model are connected by a residual connection, and the base model is a transformer model implemented based on the VIT method.

[0007] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and computer-executable instructions stored in the memory and executable on the processor, wherein the computer-executable instructions, when executed by the processor, implement the steps of the method described in the first aspect. Fourthly, embodiments of this application provide a computer-readable storage medium for storing computer-executable instructions that, when executed by a processor, implement the steps of the method described in the first aspect. Fifthly, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the steps of the method described in the first aspect.

[0008] In this embodiment, the acquired data to be processed is input into a data processing model to output the data processing result. The data processing model includes a CNN model and a base model, which are connected via residual connections. The base model is a transformer model implemented using the VIT method. By combining the local information capture capability of the CNN model and the global information capture capability of the base model in the data processing model, the model can simultaneously learn the global and local information of the data to be processed during the data processing process, thereby improving data processing performance, such as the accuracy of the data processing result. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is one of the flowcharts illustrating a data processing method provided in an exemplary embodiment of this application.

[0011] Figure 2a This is one of the structural schematic diagrams of a data processing model provided in an exemplary embodiment of this application.

[0012] Figure 2b This is a schematic diagram of the structure of a transformer block provided in an exemplary embodiment of this application.

[0013] Figure 2c This is a second schematic diagram of the structure of the data processing model provided in an exemplary embodiment of this application.

[0014] Figure 2d This is the third schematic diagram of the structure of the data processing model provided in an exemplary embodiment of this application.

[0015] Figure 3a This is one of the structural diagrams of a CNN block provided in an exemplary embodiment of this application.

[0016] Figure 3b This is the second schematic diagram of the structure of a CNN block provided in an exemplary embodiment of this application.

[0017] Figure 3c This is a schematic diagram illustrating the processing procedure of a CNN block provided in an exemplary embodiment of this application.

[0018] Figure 4 This is a second schematic flowchart of a data processing method provided in an exemplary embodiment of this application.

[0019] Figure 5 This is a schematic diagram of the configuration process of a data processing model provided in an exemplary embodiment of this application.

[0020] Figure 6 This is a schematic diagram of the structure of a data processing apparatus provided in an exemplary embodiment of this application.

[0021] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0023] The technical solutions provided in this application will be described in detail below with reference to the accompanying drawings and through some embodiments and application scenarios.

[0024] Figure 1 This diagram illustrates a flow chart of a data processing method 100 provided in an embodiment of this application. This method 100 can be executed by an electronic device, such as a terminal device or a server device. In other words, the method can be executed by software or hardware installed on the terminal device or server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. Figure 1 As shown, the method 100 may include the following steps.

[0025] S110, Obtain the data to be processed.

[0026] Depending on the data processing requirements, the data to be processed may include one or more of the following: image data, text data, audio and video data, etc.

[0027] S120, the data to be processed is input into the data processing model to output the data processing result.

[0028] The data processing model includes a Convolutional Neural Networks (CNN) model and a base model. The CNN model has excellent local feature capture capabilities and can be used to capture local information of the data to be processed.

[0029] The foundation models are general-purpose models pre-trained on large-scale data and possess broad adaptability. In this embodiment, the foundation model is a transformer model implemented using the VIT approach, which has the ability to capture local features to capture global information of the data to be processed.

[0030] Optionally, the base model may be selected from, but is not limited to, Contrastive Language–Image Pre-training (CLIP), Bootstrapping Language–Image Pre-training (BLIP), etc.

[0031] The CNN model and the base model are connected via residual connections. In this embodiment, "the CNN model and the base model are connected via residual connections" can be understood as bypassing the Transformer through residual connections to achieve parallel, non-overlapping combination between the CNN model and the base model. This integrates the local information capture capabilities of the CNN model and the global information capture capabilities of the base model into the data processing model without changing the internal structure of the base model (i.e., the data processing model possesses characteristics of both the CNN model and the base model). This allows the data processing model to simultaneously learn both global and local information of the data to be processed during data processing, thereby improving data processing performance (such as increasing the accuracy of the data processing results) while ensuring flexible and dynamic integration of local and global features. This is particularly beneficial for model fine-tuning in small datasets.

[0032] In some embodiments, the data processing model can have multiple uses, such as for business data processing like data classification and data detection. Correspondingly, depending on the purpose of the data processing model, the data processing result can also be a data classification result, a data detection result, etc., and there is no limitation here.

[0033] In some embodiments, assuming the transformer model may include at least two transformer blocks arranged in a stacked manner, then, as Figure 2a As shown, the CNN model can be connected to the target transformer block via residual connections. The target transformer block includes the first N transformer blocks from the at least two transformer blocks, where N is an integer greater than or equal to 1.

[0034] In this embodiment, the transformer model can abstract and change features block by block (also known as layer) by stacking blocks, thereby gradually enhancing the transformer model's ability to capture data characteristics globally.

[0035] Optionally, the structure of each transformer block in the transformer model can be the same. In this embodiment, the structure of the transformer block can be various, for example... Figure 2b The diagram shown is one possible schematic of the transformer block. Figure 2b The “Layernorm” shown is layer normalization, used to normalize the feature dimensions of each token (word segmentation or sample) (e.g., a 512-dimensional vector); “MHSN” is Multi-Head Self-Attention, used to learn diverse dependencies of input data (e.g., semantics, syntax, pronouns) from different subspaces; and “FFN” is Feed-Forward Network, used to enhance the representational capabilities of transformer blocks.

[0036] In some embodiments, the CNN model may include only one CNN block, or it may include at least two CNN blocks configured in parallel. In the case where the CNN model includes at least two CNN blocks configured in parallel, the convolutional kernels corresponding to each CNN block are of different sizes, thereby achieving the effect of perceiving potential targets of different sizes by performing operations on multiple receptive fields.

[0037] For example, assuming the CNN model includes two CNN blocks set in parallel, then, as Figure 2c The diagram shows the residual connection method between the CNN model and the base model.

[0038] Alternatively, assuming the CNN model comprises two CNN blocks configured in parallel, and the target transformer block comprises the first two transformer blocks of the at least two transformer blocks, then, as Figure 2d The diagram shows the residual connection method between the CNN model and the base model.

[0039] In some embodiments, there can be multiple ways to implement each CNN block included in the CNN model. For example... Figure 3a As shown, the CNN block may include, but is not limited to, a preprocessing layer, a first operation layer, a second operation layer, and a third operation layer.

[0040] The preprocessing layer transforms the input data of the first dimension into a first feature map of the second dimension, where the input data includes N tokens. In this embodiment, the preprocessing layer transforms the data dimension of the input data, enabling the input data to match the processing capabilities of the CNN model. Optionally, the input data can be understood as the input data of the data processing model, such as the data to be processed in S110, or as training data used for training the data processing model or test data used for testing the data processing model.

[0041] The first operation layer is used to perform channel attention operation on the first feature map of the second dimension using the target token to obtain the second feature map of the second dimension. The target token is the CLStoken corresponding to the base model. In this embodiment, considering that the target token can usually gather complete information from the input data, by using the target token to perform channel attention operation on the first feature map of the second dimension, it is possible to focus on what the effective information of the input data (such as image data) is.

[0042] Alternatively, if the base model initially does not have a CLS token, the target token can also be an initialized vector of all 1s. Since the target token is multiplied by the first feature map of the second dimension, the feature information of the first feature map of the second dimension remains unchanged.

[0043] The second operation layer performs a spatial attention operation on the second feature map of the second dimension to obtain a third feature map of the second dimension, and determines a fourth feature map of the second dimension based on the third feature map and the second feature map of the second dimension. This enables the CNN block to focus on the location of effective information and explicitly assigns different weights to different tokens, allowing the data processing model to focus more on tokens that are more important to the data processing task, thereby extracting more valuable information. This facilitates mapping the weighted tokens to the output of the target Transformer block, solving the problem of the Transformer treating different tokens equally and achieving effective utilization of local information.

[0044] Optionally, the spatial attention operation may include, but is not limited to, max pooling, average pooling, and convolution (such as Conv_3*3).

[0045] The third operation layer is used to convert the fourth feature map of the second dimension into N tokens of the first dimension for subsequent Transformer operations.

[0046] Optionally, depending on the data processing requirements, there can be multiple ways to combine the first dimension and the second dimension. For example, the first dimension can be 1-dimensional and the second dimension can be 2-dimensional; or the first dimension can be 2-dimensional and the second dimension can be 3-dimensional.

[0047] In some embodiments, such as Figure 3b As shown, the CNN block may further include a fourth operation layer, which includes an activation layer, a normalization layer, and a convolutional layer. The activation layer is used to perform element-wise activation on the first feature map of the second dimension using the ReLU activation function.

[0048] The normalization layer is used to normalize the first feature map of the second dimension after element-wise activation using the BatchNorm function. The BatchNorm function can also be called a batch normalization function.

[0049] The convolutional layer is used to perform convolution processing on the normalized second-dimensional first feature map using the DWConv convolution function. The DWConv convolution function can be a standard convolution function or a depthwise separable convolution function. The computational cost of the depthwise separable convolution function is lower than that of the standard convolution function.

[0050] Based on the aforementioned description of CNN blocks, the following will combine... Figure 3cThe implementation principle of the CNN block is explained as follows. It is assumed that the first dimension is 1D and the second dimension is 2D. The CNN block includes the preprocessing layer, the first operation layer, the second operation layer, the third operation layer, and the fourth operation layer.

[0051] (1) Assuming that the input data of the base model is N tokens in 1D, in order to use the CNN block, the 1D input data can be converted into a 2D feature map F1 by the preprocessing layer (where the shaded part indicates that its channel dimension > 1).

[0052] (2) The ReLU activation function in the fourth operation layer is used to activate the 2D feature map F1 element-wise. Then, the BatchNorm function in the fourth operation layer is used to normalize the element-wise activated 2D feature map F1. Finally, the DWConv convolution function in the fourth operation layer is used to convolve the normalized 2D feature map F1 to obtain the 2D feature map F2. The processing in the fourth operation layer can effectively reduce the amount of subsequent data computation and improve the data processing efficiency.

[0053] (3) Based on the first operation layer and using the target token, channel attention operation is performed on the 2D feature map F2 to obtain the 2D feature map F3, so as to focus on the effective information content in the input data (such as image data). Wherein, the target token is the CLS token of the base model.

[0054] (4) Based on the second operation layer, a spatial attention operation is performed on the 2D feature map F3 to obtain a 2D feature map F4, so as to focus on the location of effective information in the input data (such as image data), and the 2D feature map F3 and the 2D feature map F4 are multiplied to obtain a 2D feature map F5, so as to explicitly assign different weights to different tokens, so that the data processing model pays more attention to the tokens that are more important to the data processing task, thereby extracting more valuable information, so that the weighted tokens can be mapped to the output of the target Transformer block, thus solving the problem that local information cannot be effectively utilized when the Transformer model treats different tokens equally.

[0055] (5) Based on the third operation layer, the 2D feature map F5 is converted into N 1D tokens for subsequent Transformer operations.

[0056] In some embodiments, such as Figure 4 As shown, embodiment 100 of this method may further include S130 and S140, the contents of which are as follows.

[0057] S130, Obtain the training dataset.

[0058] S140, the CNN model in the data processing model is trained using the training dataset to optimize the model parameters in the CNN model in the data processing model.

[0059] In this embodiment, considering that the CNN model and the base model in the data processing model adopt a residual connection, when fine-tuning the data processing model is required, only the parameters of the lightweight CNN model in the data processing model can be fine-tuned. This not only reduces the amount of computation and significantly reduces the dependence on hardware, but also effectively avoids the model overfitting or underfitting problems that may occur during model training, since this embodiment only needs to train the lightweight CNN model for model training in some small dataset scenarios.

[0060] For example, in some vertical industrial sectors, the difference between real business data and pre-training data is significant. Therefore, when fine-tuning the data processing model using real data is necessary, the amount of real data is usually limited (i.e., a small dataset). In this case, if the VIT-based multimodal model (such as CLIP, BLIP, etc.) is directly fine-tuned using a small dataset, as in related technologies, overfitting may occur. Alternatively, if the VIT-based multimodal model is fine-tuned using a small dataset based on Low-Rank Adaptation (Lora), it requires highly skilled personnel and is prone to overfitting or underfitting. Furthermore, VIT models fine-tuned using Lora cannot utilize local features, losing some expressive power. To address these issues, this embodiment only fine-tunes the parameters of the lightweight CNN model within the data processing model, effectively avoiding overfitting or underfitting problems that may occur during model training.

[0061] In some embodiments, the CNN model can be understood as an independent component. Therefore, the base model and the CNN model can be deployed on the same physical device (or machine), or the base model and the CNN model can be deployed independently on different physical devices, thereby improving the deployment flexibility of the data processing model and reducing the hardware requirements of the physical device for the data processing model.

[0062] Based on the aforementioned description of the data processing methods, the following is as follows: Figure 5 The configuration process of the data processing model mentioned in this application is described as shown below.

[0063] (1) Business data processing The business data processing may include data annotation, dividing the annotated data into training data sets and test data sets, etc.

[0064] (2) Model structure configuration.

[0065] For example, any transformer model based on the VIT method can be selected as the base model, such as CLIP; then the CNN model is determined, and finally the CNN model and the base model are combined using a residual connection method.

[0066] In addition, if the base model does not have a [CLS] token initially, the [CLS] token can be initialized as a vector of all 1s.

[0067] (3) Model training During training, only the CNN blocks are trained, which not only reduces computational cost but also significantly lowers hardware dependence. Furthermore, once the model is trained, the versatility of the original base model is fully preserved, allowing for seamless switching between customization and general-purpose capabilities.

[0068] (4) Model evaluation For example, a test dataset can be used to evaluate the performance of the data processing model.

[0069] (5) Model Deployment The base model and the CNN model can be deployed independently on one or more physical devices. The CNN model can be understood as an independent component. Only the CNN is trained for different tasks of small dataset classes. The trained CNN is then combined with the base model to perform different tasks.

[0070] Figure 6 The diagram shows a structural schematic of a data processing apparatus 600 provided in an embodiment of this application. The apparatus 600 includes: a first processing module 610 for acquiring data to be processed; and a second processing module for inputting the data to be processed into a data processing model to output a data processing result. The data processing model includes a CNN model and a base model, wherein the CNN model and the base model are connected through a residual connection, and the base model is a transformer model implemented based on the VIT method.

[0071] In some embodiments, the transformer model includes at least two transformer blocks arranged in a stacked manner, and the CNN model is connected to the target transformer block through a residual connection. The target transformer block includes the first N transformer blocks from the at least two transformer blocks, where N is an integer greater than or equal to 1.

[0072] In some embodiments, the CNN model includes at least two CNN blocks configured in parallel, each CNN block having a different size of convolutional kernel.

[0073] In some embodiments, for each CNN block in the CNN model, the CNN block includes: a preprocessing layer for converting input data of a first dimension into a first feature map of a second dimension, the input data including N tokens; a first operation layer for performing channel attention operation on the first feature map of the second dimension using a target token to obtain a second feature map of the second dimension, the target token being the CLS token corresponding to the base model, or the target token being a vector of all 1s; a second operation layer for performing spatial attention operation on the second feature map of the second dimension to obtain a third feature map of the second dimension, and determining a fourth feature map of the second dimension based on the third feature map of the second dimension and the second feature map of the second dimension; and a third operation layer for converting the fourth feature map of the second dimension into N tokens of the first dimension.

[0074] In some embodiments, the input data includes at least one of the data to be processed and the training dataset used for training the CNN model.

[0075] In some embodiments, the CNN block further includes a fourth operation layer, which includes: an activation layer for element-wise activation of the first feature map of the second dimension using the ReLU activation function; a normalization layer for normalizing the first feature map of the second dimension after element-wise activation using the BatchNorm function; and a convolutional layer for convolutional processing of the normalized first feature map of the second dimension using the DWConv convolutional function.

[0076] In some embodiments, the DWConv convolution function is a depthwise separable convolution function.

[0077] In some embodiments, the first dimension is 1-dimensional and the second dimension is 2-dimensional; or, the first dimension is 2-dimensional and the second dimension is 3-dimensional.

[0078] In some embodiments, the apparatus 600 further includes a third processing module 630, configured to acquire a training dataset; and to train the CNN model in the data processing model using the training dataset to optimize the model parameters in the CNN model in the data processing model.

[0079] In some embodiments, the training dataset is a small dataset.

[0080] In some embodiments, the pedestal model and the CNN model are deployed independently on different physical devices.

[0081] The device 600 provided in this application embodiment can execute the methods described in the preceding method embodiments and achieve the functions and beneficial effects of the methods described in the preceding method embodiments, which will not be repeated here.

[0082] Figure 7 The diagram illustrates the hardware structure of an electronic device implementing the embodiments of this application. Referring to the diagram, at the hardware level, the electronic device includes a processor and optionally, an internal bus, a network interface, and a memory. The memory may include RAM, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk storage device. Of course, the electronic device may also include other hardware required for other services.

[0083] The processor, network interface, and memory can be interconnected via an internal bus, which can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only a single bidirectional arrow is used in this diagram, but this does not imply that there is only one bus or one type of bus.

[0084] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0085] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming a device at the logical level that is assigned to a specific user. The processor executes the program stored in memory and specifically performs the following: Figures 1-4 The methods disclosed in the embodiments shown achieve the functions and beneficial effects of the methods described in the preceding method embodiments, and will not be repeated here.

[0086] The above is as stated in this application. Figures 1-4 The methods disclosed in the illustrated embodiments can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0087] The electronic device can also execute the methods described in the preceding method embodiments and achieve the functions and beneficial effects of the methods described in the preceding method embodiments, which will not be repeated here.

[0088] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0089] This application also proposes a computer-readable storage medium that stores one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform... Figures 1-4 The methods disclosed in the embodiments shown achieve the functions and beneficial effects of the methods described in the preceding method embodiments, and will not be repeated here.

[0090] The computer-readable storage medium mentioned above includes read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc.

[0091] This application also provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0092] Alternatively, the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, implement the following process: Figure 1 The method shown in Figure 3 illustrates the functions and beneficial effects of the methods described in the preceding method embodiments, and will not be repeated here.

[0093] Computer-readable storage media include both permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media do not include transient media, such as modulated data signals and carrier waves.

[0094] In summary, the above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0095] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0096] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0097] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A data processing method, characterized in that, include: Obtain the data to be processed; The data to be processed is input into the data processing model to output the data processing result; The data processing model includes a CNN model and a base model. The CNN model and the base model are connected through a residual connection. The base model is a transformer model implemented based on the VIT method.

2. The method as described in claim 1, characterized in that, The transformer model includes at least two transformer blocks arranged in a stacked manner, and the CNN model is connected to the target transformer block through a residual connection. The target transformer block includes the first N transformer blocks from the at least two transformer blocks, where N is an integer greater than or equal to 1.

3. The method as described in claim 1, characterized in that, The CNN model includes at least two CNN blocks set in parallel, and the size of the convolution kernels corresponding to each CNN block is different.

4. The method according to any one of claims 1-3, characterized in that, For each CNN block in the CNN model, the CNN block includes: A preprocessing layer is used to convert the input data of the first dimension into a first feature map of the second dimension, wherein the input data includes N words; The first operation layer is used to perform channel attention operation on the first feature map of the second dimension using the target word segmentation to obtain the second feature map of the second dimension. The target word segmentation is the CLS word segmentation corresponding to the base model, or the target word segmentation is a vector of all 1s. The second operation layer is used to perform spatial attention operation on the second feature map of the second dimension to obtain the third feature map of the second dimension, and to determine the fourth feature map of the second dimension based on the third feature map of the second dimension and the second feature map of the second dimension. The third operation layer is used to convert the fourth feature map of the second dimension into N words of the first dimension.

5. The method as described in claim 4, characterized in that, The CNN block further includes a fourth operation layer, which includes: An activation layer is used to perform element-wise activation of the first feature map in the second dimension using the ReLU activation function. The normalization layer is used to normalize the first feature map of the second dimension after element-wise activation using the BatchNorm function; The convolutional layer is used to perform convolution processing on the first feature map of the normalized second dimension using the DWConv convolution function.

6. The method as described in claim 5, characterized in that, The DWConv convolution function is a depthwise separable convolution function.

7. The method as described in claim 4, characterized in that, The first dimension is 1-dimensional and the second dimension is 2-dimensional; or, the first dimension is 2-dimensional and the second dimension is 3-dimensional.

8. The method according to any one of claims 1-3, characterized in that, The method further includes: Obtain the training dataset; The CNN model in the data processing model is trained using the training dataset to optimize the model parameters in the CNN model within the data processing model.

9. The method as described in claim 8, characterized in that, The training dataset is a small dataset.

10. The method according to any one of claims 1-3, characterized in that, The base model and the CNN model are deployed independently on different physical devices.

11. A data processing apparatus, characterized in that, include: The first processing module is used to acquire data to be processed. The second processing module is used to input the data to be processed into the data processing model in order to output the data processing result. The data processing model includes a CNN model and a base model. The CNN model and the base model are connected through a residual connection. The base model is a transformer model implemented based on the VIT method.

12. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1-10.

14. A computer program product, characterized in that, The computer program product is stored in a storage medium, and the computer program product is executed by at least one processor to implement the steps of the method as described in any one of claims 1-10.

Images