Data processing method and device, storage medium and computer equipment

Abstract

The embodiment of the invention provides a data processing method and device, a storage medium and computer equipment, and the method comprises the steps: firstly obtaining first image features of batch images, inputting the first image features into a current processing module of a visual state space model, obtaining an initial global receptive field through four-way scanning, and then carrying out the feature adjustment and dimension transformation, thereby obtaining a visual state space model; the final dimension of the intermediate variable is the batch image number, and total output features are determined by combining the model weight parameters of the current processing module. Performing inverse transformation on the total output feature, and performing four-way scanning and integration to obtain a second image feature; and when the current processing module is the last, determining an image classification result based on the second image feature. According to the method, through reasonable dimension transformation, the situation that the TPU cannot be processed in parallel due to the fact that the final dimension is 1 is avoided, the high-performance TPU calculation potential is fully played, the TPU utilization rate is increased, and the data processing efficiency is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a data processing method, device, storage medium and computer equipment. Background Art

[0002] The Visual State Space Model (VMamba) is a novel vision backbone network model that aims to reduce computational complexity to linear time complexity while retaining the advantageous properties of Vision Transformers, such as global receptive field and dynamic weight parameters. VMamba is inspired by Mamba, a state-space language model that has demonstrated the ability to efficiently model long sequences in natural language processing (NLP) tasks. VMamba transplants this concept to the field of vision, and processes visual data with linear complexity by introducing the Visual State Space (VSS) module and the 2D Selective Sweep (SS2D) module. Extensive experiments demonstrate the excellent performance of VMamba in a variety of visual perception tasks, including image classification, object detection, and semantic segmentation, highlighting its advantage in input scaling efficiency compared to existing baseline models.

[0003] In the related art, in the SS2D module, there is a process that needs to be executed in a loop. The process needs to be executed sequentially L times, where L is the number of image blocks (Tokens) obtained by segmenting the image being processed. In the shallow Cross-Scan module, the number of image blocks L is as high as 56*56. The huge L causes the process to have a large number of loops, and the loop needs to be executed sequentially one image block at a time, making it difficult to give full play to the parallel computing advantages of the TPU architecture. In addition, the parallelism of the four-dimensional data in the tensor processing unit (TPU) architecture occurs in the second and fourth dimensions. In each actual cycle, the shape of the intermediate variables involved in the calculation of the prior art is mostly long strips, and the dimension of the last dimension is only 1, which is not friendly to batch data transmission and TPU parallel computing, and it is difficult to give full play to the computing potential of high-performance TPUs and the low utilization rate of TPUs, and the data processing efficiency is low. Therefore, the related art urgently needs to propose a data processing method to solve the above technical problems. Summary of the invention

[0004] The main purpose of this application is to provide a data processing method, apparatus, storage medium and computer equipment, which can give full play to the computing potential of high-performance TPU and improve the utilization rate of TPU, thereby improving data processing efficiency.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, including: Obtaining the first image feature of each image in the batch of images; Inputting the first image features into a current processing module in a visual state space model, performing four-directional scanning on each of the first image features to obtain an initial global receptive field; Performing different feature adjustments on the initial global receptive field to obtain an initial first intermediate variable, an initial second intermediate variable, and an initial third intermediate variable respectively; Performing dimension transformation on the initial first intermediate variable, the initial second intermediate variable, the initial third intermediate variable, and the initial global receptive field to obtain a first intermediate variable, a second intermediate variable, a third intermediate variable, and a global receptive field, wherein the last dimension of the first intermediate variable, the second intermediate variable, the third intermediate variable, and the global receptive field is the number of images in the batch of images; Determine a total output feature based on the first intermediate variable, the second intermediate variable, the third intermediate variable, the global receptive field, and a model weight parameter of a current processing module; Performing an inverse dimensionality transformation on the total output feature to obtain an inverse transformation result; Performing four-way scanning integration on the inverse transformation result to obtain a second image feature; When the current processing module is the last module, an image classification result is determined based on the second image feature.

[0006] In a second aspect, an embodiment of the present application provides a data processing device, including: An acquisition unit, used for acquiring a first image feature of each image in a batch of images; An input unit, used for inputting the first image features into a current processing module in a visual state space model, performing four-directional scanning on each of the first image features to obtain an initial global receptive field; An adjustment unit, used to perform different feature adjustments on the initial global receptive field to obtain an initial first intermediate variable, an initial second intermediate variable, and an initial third intermediate variable respectively; a transformation unit, configured to perform dimension transformation on the initial first intermediate variable, the initial second intermediate variable, the initial third intermediate variable, and the initial global receptive field to obtain a first intermediate variable, a second intermediate variable, a third intermediate variable, and a global receptive field, wherein the last dimension of the first intermediate variable, the second intermediate variable, the third intermediate variable, and the global receptive field is the number of images in the batch of images; A first determining unit, configured to determine a total output feature based on the first intermediate variable, the second intermediate variable, the third intermediate variable, the global receptive field, and a model weight parameter of a current processing module; An inverse transformation unit, used to perform a dimensional inverse transformation on the total output feature to obtain an inverse transformation result; An integration unit, used for performing four-way scanning integration on the inverse transformation result to obtain a second image feature; The second determining unit is used to determine the image classification result based on the second image feature when the current processing module is the last module.

[0007] In a third aspect, an embodiment of the present application provides a storage medium, wherein the computer-readable storage medium stores a plurality of instructions, which are suitable for loading by a processor to execute any of the above data processing methods.

[0008] In a fourth aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above data processing methods when executing the computer program.

[0009] In an embodiment of the present application, a first image feature of each image in a batch of images is obtained; the first image feature is input into a current processing module in a visual state space model, and each first image feature is scanned in four directions to obtain an initial global receptive field; different feature adjustments are made to the initial global receptive field to obtain an initial first intermediate variable, an initial second intermediate variable, and an initial third intermediate variable, respectively; the initial first intermediate variable, the initial second intermediate variable, the initial third intermediate variable, and the initial global receptive field are dimensionally transformed to obtain a first intermediate variable, a second intermediate variable, a third intermediate variable, and a global receptive field, wherein the last dimension of the first intermediate variable, the second intermediate variable, the third intermediate variable, and the global receptive field is the number of images in the batch of images; a total output feature is determined based on the first intermediate variable, the second intermediate variable, the third intermediate variable, the global receptive field, and a model weight parameter of the current processing module; a dimension inverse transformation is performed on the total output feature to obtain an inverse transformation result; the inverse transformation result is scanned and integrated in four directions to obtain a second image feature; when the current processing module is the last module, an image classification result is determined based on the second image feature corresponding to each module. Compared with the related art, in which the parallel computing capability of the TPU cannot be fully utilized due to the dimension problem of the intermediate variables, the embodiment of the present application transforms the dimensions of the intermediate variables so that the last dimension of each intermediate variable is the number of images in the batch of images, thereby avoiding the problem that the TPU cannot be processed in parallel due to the last dimension being 1, thereby exerting the computing potential of the high-performance TPU and improving the utilization rate of the TPU, thereby improving data processing efficiency.

[0010] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application.

[0013] Figure 2 A schematic diagram of data processing during four-way scanning of the Cross-Scan module provided in an embodiment of the present application.

[0014] Figure 3 A schematic diagram of transforming the dimensions of intermediate variables provided in an embodiment of the present application.

[0015] Figure 4 A schematic diagram of calculating the total output characteristics provided in an embodiment of the present application.

[0016] Figure 5 A specific schematic diagram of the inverse transformation process provided in an embodiment of the present application.

[0017] Figure 6 A schematic diagram of the four-way scanning integration process provided in an embodiment of the present application.

[0018] Figure 7 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application.

[0019] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0021] It should be noted that in some processes described in the specification, claims and the above-mentioned drawings, multiple steps appearing in a specific order are included, but it should be clearly understood that these steps may not be executed in the order in which they appear in this document or may be executed in parallel. The step numbers are only used to distinguish different steps, and the numbers themselves do not represent any execution rules. In addition, descriptions such as "first", "second" or "target" in this document are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0022] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations: Visual State Space Model (VMamba): is a new visual backbone network model that aims to reduce computational complexity to linear time complexity while retaining the advantages of vision transformers, such as global receptive field and dynamic weight parameters. Specifically, (1) in terms of state space model, VMamba is inspired by the state space language model (Mamba), which has demonstrated the ability to efficiently model long sequences in natural language processing (NLP) tasks. VMamba transplants this concept to the visual field and processes visual data with linear complexity by introducing the visual state space (VSS) module and the 2D selective scan (SS2D) module. (2) in terms of global receptive field and dynamic weight, VMamba achieves global receptive field and dynamic weight through its core visual state space (VSS) module and selective scan (SS2D) module, which helps to collect contextual information from different sources and angles and provide rich feature representation for visual tasks. (3) In terms of linear time complexity, a notable feature of VMamba is its linear time complexity, which means that as the input size increases, VMamba's computational efficiency has a significant advantage over traditional methods. Based on the current VMamba model architecture, the present invention further improves the performance of VMamba by optimizing data handling. (4) In terms of the cross-scan module (CSM), in order to solve the problem that visual signals do not have a natural order like text sequences, VMamba designs a Cross-Scan module. The Cross-Scan module adopts a four-way scanning strategy to ensure that each element in the feature integrates information from all other positions in different directions to form a global receptive field without increasing linear computational complexity. (5) In terms of experimental verification, extensive experiments demonstrate the excellent performance of VMamba in a variety of visual perception tasks, including image classification, object detection, and semantic segmentation, highlighting its advantages in input scaling efficiency compared to existing baseline models.

[0023] Tensor Processing Unit (TPU): Focusing on the research and development and promotion of computing power products such as processors, adhering to the concept of a fully open source and open ecosystem, leading the innovation of intelligent computing technology. It has created a full-scenario product matrix covering "cloud, edge, and end", which has been widely used and recognized by users in multiple scenarios such as urban operations, intelligent manufacturing, large model applications, and smart terminals. Suanneng's TPU accelerator has highly parallel computing capabilities, which can significantly improve the reasoning speed and efficiency of AI models. Provide a more stable and reliable computing power infrastructure for the global open source community and developers.

[0024] 2D Selective Scan (SS2D) module: is a functional module used for specific scanning and processing of two-dimensional data in the field of image processing. Its scanning mechanism is: it will scan two-dimensional data, such as the pixel matrix of an image, according to specific rules. This kind of scanning is not a simple row-by-row or column-by-column scanning, but is selective. It may select specific rows, columns or areas for scanning in a targeted manner based on certain characteristics of the data or pre-set algorithms to obtain more valuable information. During the scanning process, the module will perform specific calculation operations, such as numerical calculation and feature extraction of the scanned data. At the same time, data scheduling will also be performed, that is, according to the needs of calculation and subsequent processing requirements, the flow and storage of data will be reasonably arranged to ensure that the data can be processed and used efficiently.

[0025] The purpose of using the SS2D module is to help extract specific features of the image. By selectively scanning different areas of the image, it is possible to capture feature information such as edges, textures, and colors in the image, providing a basis for subsequent image analysis, recognition, and other tasks. The data can be processed based on the results of the scan to achieve data dimensionality reduction and remove some redundant information, while retaining features that are important for subsequent tasks, thereby improving the efficiency of data processing and the performance of the model. Or in some cases, data in specific areas are enhanced to highlight key information in the image. In complex image processing or deep learning models, the SS2D module usually works in conjunction with other modules. For example, in the VMamba model, it can work with the Cross-Merge module to provide support for the image feature processing and reasoning process of the entire model, and provide more suitable and valuable input data for other modules through selective scanning.

[0026] In the related art, in the 2D selective scanning (SS2D) module, there is a process that needs to be executed in a loop. This process needs to be executed sequentially L times, where L is the number of image blocks (Tokens) obtained by segmenting the image being processed. In the shallow Cross-Scan module, the number of image blocks L is as high as 56*56. The huge L causes the number of loops of this process to be very large, and the loops need to be executed sequentially, making it difficult to give full play to the parallel computing advantages of the TPU architecture. In addition, the parallelism of the TPU architecture for four-dimensional data occurs in the second and fourth dimensions. In each actual cycle, the shape of the intermediate variables involved in the calculation is mostly long strips (such as: [batch size, 1536, 1, 1]), and the dimension of the last dimension is only 1, which is not friendly to batch data transmission and TPU parallel computing, resulting in low IO efficiency and low TPU utilization, making it difficult to give full play to the computing potential of high-performance TPUs.

[0027] In order to solve the above problems, the embodiment of the present application obtains the first image feature of each image in the batch of images; inputs the first image feature into the current processing module in the visual state space model, performs four-way scanning on each first image feature to obtain an initial global receptive field; performs different feature adjustments on the initial global receptive field to obtain an initial first intermediate variable, an initial second intermediate variable and an initial third intermediate variable respectively; performs dimension transformation on the initial first intermediate variable, the initial second intermediate variable, the initial third intermediate variable and the initial global receptive field to obtain a first intermediate variable, a second intermediate variable, a third intermediate variable and a global receptive field, wherein the last dimension of the first intermediate variable, the second intermediate variable, the third intermediate variable and the global receptive field is the number of images in the batch of images; determines the total output feature based on the first intermediate variable, the second intermediate variable, the third intermediate variable, the global receptive field and the model weight parameter of the current processing module; performs dimension inverse transformation on the total output feature to obtain an inverse transformation result; performs four-way scanning and integration on the inverse transformation result to obtain a second image feature; when the current processing module is the last module, determines the image classification result based on the second image feature corresponding to each module. Compared with the related art, which cannot give full play to the parallel computing capability of TPU due to the dimension problem of intermediate variables, the embodiment of the present application transforms the dimension of intermediate variables so that the last dimension of each intermediate variable is the number of images in the batch image, avoiding the problem that the TPU cannot be processed in parallel due to the last dimension being 1, thereby giving full play to the computing potential of high-performance TPU and improving the utilization rate of TPU, thereby improving data processing efficiency. Please continue to refer to the following specific embodiments for details.

[0028] In this embodiment, the description will be made from the perspective of a data processing device, which can be specifically integrated in a client having a storage unit and a microprocessor installed therein and having computing capabilities.

[0029] See also Figure 1 , Figure 1 A flow chart of a data processing method provided in an embodiment of the present application. The data processing method includes: In step 201, a first image feature of each image in a batch of images is obtained.

[0030] Among them, batch images refer to a group of images processed at one time, and these images are usually organized together in the data structure to improve the processing efficiency by using parallel computing. When the first image feature is the feature representation obtained after each image has been processed in the early stage (such as preprocessing in the data preparation stage), and is used for subsequent input to the Block module of the visual state space model for further processing, the first image feature is that it contains some basic feature information of the image, such as color, texture, edge and other initially extracted features; when the first image feature is the image feature output by a Block module of the visual state space model, the first image feature is the result after a series of complex processing. It is the product of deep processing of the input features, and has undergone operations such as shape adjustment, dimension segmentation, element summation and dimension swapping, integrating four-way scanning information, and is a summary and refinement of the multi-dimensional feature information processed before.

[0031] Specifically, obtain the pre-trained model weights of the VMamba model, compile it into a model file that can be executed on the computing energy TPU (for example, a model file in the Bmodel format), and transfer it to the storage space of the computing energy TPU device. Pack and preprocess a batch (for example, 16 images) of image data that need to be classified, and transfer them to the storage space of the computing energy TPU device. Take 16 images with three RGB channels as an example, first scale each image to a data block of size [3, 224, 224]. Among them, 3 is the number of RGB channels, and 224 is the length and width of the scaled image. Then, normalize the data set of each image separately, the specific formula is (x-min) / (max-min), so that its value distribution is [0, 1], and obtain a new data block. Among them, x is the image data block, min is the minimum value of the data block, and max is the maximum value of the data block. Then, the new data blocks are normalized by subtracting the three means (e.g., [0.485, 0.456, 0.406]) obtained from the training data set of the model on the three channels, and dividing the three standard deviations (e.g., [0.229, 0.224, 0.225]) obtained from the training data set of the model on the three channels to obtain the normalized data blocks. Finally, the normalized data blocks of the 16 images are packaged into data blocks of size [16, 3, 224, 224] to obtain the preprocessed image data blocks.

[0032] The reasoning process of the VMamba model is carried out on the computing TPU device. The specific process is to input a batch of image data blocks preprocessed in the above data preparation stage into the VMamba model in the TPU device, and then use the computing TPU device to execute the image block embedding layer, multiple Block modules, and the last few layers of the VMamba model in sequence, and finally obtain the reasoning results output by the model for parsing the model reasoning result stage. The first several layers of the VMamba model are specifically 2D convolution layer, GELU activation layer, 2D convolution layer, and LayerNorm normalization layer in order. The role of the above four-layer neural network is to map the preprocessed image data block to a higher-dimensional data space to obtain the first image feature, which is convenient for subsequent entry into the Block module to continue to extract features. At this time, the image feature size of the first image feature obtained is [batch size, C, H_input, W_input]. Where C is the image data block feature depth, H_input is the height of the image, and W_input is the width of the image.

[0033] In step 202, the first image features are input into the current processing module in the visual state space model, and a four-way scan is performed on each of the first image features to obtain an initial global receptive field.

[0034] Among them, the visual state space model includes multiple Block modules, each Block module includes a cross-scan module (Cross-Scan), and multiple Block modules are processed in sequence, and each Block module has its own independent model weight parameters. For the Block module currently to be processed, each first image feature is scanned in four directions through the cross-scan module (Cross-Scan) to obtain the initial global receptive field. The initial global receptive field is the feature representation obtained after the four-way scan, which contains the information after the image features are integrated in different directions, so that the model can capture more comprehensive contextual information in the image and provide a richer information basis for subsequent feature processing.

[0035] In some implementations, performing four-directional scanning on each of the first image features to obtain an initial global receptive field includes: (1) For each of the first image features, the first image feature is copied to obtain two first image features, wherein the first dimension of the first image feature is the feature depth of the image data block, the second dimension is the longitudinal number of the image data blocks, and the third dimension is the lateral number of the image data blocks; (2) multiplying the third dimension of one of the first image features by the fourth dimension, and using the result as the third dimension to obtain a third image feature; (3) swapping the third dimension and the fourth dimension of another of the first image features, and multiplying the swapped third dimension by the swapped fourth dimension, using the result as the third dimension, to obtain a fourth image feature; (4) stacking the third image feature and the fourth image feature in the dimension of feature depth of the image data block to obtain a stacked feature; (5) duplicating the stacking feature to obtain two stacking features; (6) Each of the stacked features is stacked in the dimension of the feature depth of the image data block to obtain an initial global receptive field.

[0036] Among them, the related technology uses a method of partially flipping the current image feature map in the Cross-Scan module to implement a technical four-way scanning strategy. In the Cross-Merge module, the current image feature map is also partially flipped to merge the image information obtained by the four-way scanning strategy. However, in the computing energy TPU architecture, the execution efficiency of flipping image data is very low, resulting in an operator that does not require any computation to flip a large number of images, but takes up a lot of reasoning time. Based on this, an embodiment of the present application provides a new four-way scanning strategy.

[0037] Specifically, Figure 2 As shown, Figure 2A schematic diagram of data processing during four-way scanning of the Cross-Scan module provided in an embodiment of the present application. In the inference process of each Block module of the VMamba model, when executing the Cross-Scan module, the data flipping operation is not used, and instead a specific calculation and data scheduling strategy is used in the subsequent 2D selective scanning (SS2D) module to play the same role. In the Cross-Scan module, the first image feature of the input of the module (the feature dimension size is [the number of images in the batch image, C, H, W], where C is the feature depth of the image data block, H is the vertical number of image data blocks, and W is the horizontal number of image data blocks) is copied to obtain a total of two identical first image features. Then, one of them is reshaped, that is, the third dimension H of the first image feature is multiplied by the fourth dimension W, and the feature size of the third image feature is [number of images, C, H*W]; for the other one, the third dimension H and the fourth dimension W are swapped, and then the shape is adjusted, that is, the swapped third dimension W is multiplied by the swapped fourth dimension H, and the feature size of the fourth image feature is also [number of images, C, H*W]; then, the above two first image features are stacked in C dimension, and the feature size is [number of images, 2*C, H*W]; then, the stacked features are copied and stacked in C dimension, and the data block size is [number of images, 4*C, H*W] The initial global receptive field. The above steps complete the four-way scanning process of the Cross-Scan module, ensuring that each element in the feature integrates information from all other positions in different directions to form an initial global receptive field.

[0038] In step 203, different feature adjustments are performed on the initial global receptive field to obtain an initial first intermediate variable, an initial second intermediate variable, and an initial third intermediate variable, respectively.

[0039] Among them, in the inference process of each Block module of the VMamba model, after the Cross-Scan module is executed normally, other operators may exist in front of the 2D selective scanning (SS2D) module. Specifically, the initial global receptive field with a feature size of [number of images, 4*C, L] obtained in step 202, L is H*W, needs to pass through a 1D convolution layer, and the convolution result is divided to obtain the first intermediate variable, the second intermediate variable, and the third intermediate variable. Then, the first intermediate variable is input into the next 1D convolution layer to obtain a new first intermediate variable with a size of [batch size, 4*C, L, N] to replace the original first intermediate variable, and obtain the initial first intermediate variable. The second and third intermediate variables are repeated C times in the third dimension respectively, and the new second and third intermediate variables with a size of [batch size, 4*C, N, L] are obtained and replace the original second and third intermediate variables to obtain the initial third intermediate variable. Then, the second intermediate variable is further subjected to matrix multiplication and shape transformation with the weight parameters in the model to obtain an updated second intermediate variable of size [batch size, 4*C, L, N] and replace the original second intermediate variable to obtain the initial second intermediate variable. Wherein, L is H*W, which represents the total number of image blocks after image segmentation; N is a preset hyperparameter for controlling the parameter amount, which is set to 1 in the embodiment of the present application.

[0040] In step 204, the initial first intermediate variable, the initial second intermediate variable, the initial third intermediate variable and the initial global receptive field are dimensionally transformed to obtain the first intermediate variable, the second intermediate variable, the third intermediate variable and the global receptive field, wherein the last dimension of the first intermediate variable, the second intermediate variable, the third intermediate variable and the global receptive field is the number of images in the batch of images.

[0041] Among them, before executing the 2D selective scan (SS2D) module, the intermediate variables involved in the operation are transformed in dimension, and the dimension representing the batch size (that is, the number of images) is placed as the last dimension, which is more friendly to batch data transmission and TPU parallel computing, and better utilizes the TPU architecture's parallel computing capabilities for four-dimensional data, improving IO efficiency and TPU utilization.

[0042] Specifically, since the size of the initial first intermediate variable is [number of images, 4*C, L, N], the size of the initial second intermediate variable is [number of images, 4*C, L, N], the size of the initial third intermediate variable is [number of images, 4*C, N, L], and the size of the initial global receptive field is [number of images, 4*C, L], it can be seen that the last dimension of the initial first intermediate variable, the initial second intermediate variable, the initial third intermediate variable, and the initial global receptive field is not the number of images, so the dimension of the number of images is set to the last dimension for easy use by TPU.

[0043] In some embodiments, the first dimension of the initial first intermediate variable and the initial second intermediate variable is the number of images of the batch images, the second dimension is the feature depth of the image data blocks at a specified multiple, the third dimension is the total number of image data blocks, and the fourth dimension is a preset hyperparameter; the first dimension of the initial third intermediate variable is the number of images, the second dimension is the feature depth of the image data blocks at a specified multiple, the third dimension is the preset hyperparameter, and the fourth dimension is the total number of image data blocks; the first dimension of the initial global receptive field is the number of images, the second dimension is the feature depth of the image data blocks at a specified multiple, and the third dimension is the total number of image data blocks; The step of performing dimension transformation on the initial first intermediate variable, the initial second intermediate variable, the initial third intermediate variable, and the initial global receptive field to obtain the first intermediate variable, the second intermediate variable, the third intermediate variable, and the global receptive field includes: (1) respectively swapping the first dimension and the fourth dimension of the initial first intermediate variable, the initial second intermediate variable, and the initial third intermediate variable to obtain a first intermediate variable, a second intermediate variable, and a third intermediate variable; (2) The first dimension and the third dimension of the initial global receptive field are swapped to obtain a global receptive field.

[0044] Among them, see Figure 3 , Figure 3 A schematic diagram of transforming the dimensions of intermediate variables provided in an embodiment of the present application. For a first intermediate variable with a shape of [number of images, 4*C, L, N], the first and fourth dimensions are swapped, and the resulting variable shape is [N, 4*C, L, number of images]; for a second intermediate variable with a shape of [number of images, 4*C, L, N], the first and fourth dimensions are swapped, and the resulting variable shape is [N, 4*C, L, number of images]; for a third intermediate variable with a shape of [number of images, 4*C, N, L], the first and fourth dimensions are swapped, and the resulting variable shape is [L, 4*C, N, number of images]; for an initial global receptive field with a shape of [number of images, 4*C, L], the first and third dimensions are swapped, and the resulting variable shape is [L, 4*C, N, number of images] of the global receptive field.

[0045] In step 205, the total output feature is determined based on the first intermediate variable, the second intermediate variable, the third intermediate variable, the global receptive field, and the model weight parameter of the current processing module.

[0046] Among them, the model weight parameters are the parameters learned by the visual state space model during the training process, which are used to transform and process the input data and determine the behavior and performance of the model. Different modules have different weight parameters, which are continuously optimized through training so that the model can accurately complete tasks, such as image classification. The total output feature is the final feature representation obtained by calculating and fusing the first intermediate variable, the second intermediate variable, the third intermediate variable, the global receptive field and the model weight parameters. It integrates the information of multiple intermediate variables and the knowledge carried by the model weights, providing key feature basis for subsequent tasks (such as classification).

[0047] Specifically, specific operations are performed on the first intermediate variable, the second intermediate variable, the third intermediate variable, the global receptive field, and the model weight parameters of the module to finally obtain the total output feature.

[0048] In some implementations, determining the total output feature based on the first intermediate variable, the second intermediate variable, the third intermediate variable, the global receptive field, and a model weight parameter of the current processing module includes: (1) dividing the current second dimension of the first intermediate variable, the second intermediate variable and the third intermediate variable to obtain two divided first intermediate variables, two divided second intermediate variables and two divided third intermediate variables, and dividing the second dimension of the product of the global receptive field and the model weight parameter of the module to obtain two divided fourth intermediate variables; (2) extracting the first variable slice to be calculated from one of the first intermediate variables after division, one of the second intermediate variables after division, one of the third intermediate variables after division, and one of the fourth intermediate variables after division in a first preset order, and extracting the second variable slice to be calculated from another of the first intermediate variables after division, another of the second intermediate variables after division, another of the third intermediate variables after division, and another of the fourth intermediate variables after division in a second preset order, wherein the first preset order is opposite to the second preset order; (3) Calculate each of the first variable slices to obtain a first partial result, and calculate each of the second variable slices to obtain a second partial result; (4) storing the first part of the results in the first preset order, storing the second part of the results in the second preset order, and increasing the number of loop executions by one to obtain the current number of loop executions; (5) When the number of executions of the current loop does not reach half of the total number of the image data blocks, return to execute the step of extracting the first variable slice to be calculated from one of the first intermediate variables after division, one of the second intermediate variables after division, one of the third intermediate variables after division, and one of the fourth intermediate variables after division in the first preset order, and extracting the second variable slice to be calculated from another of the first intermediate variables after division, another of the second intermediate variables after division, another of the third intermediate variables after division, and another of the fourth intermediate variables after division in the second preset order, until the number of executions of the current loop reaches half of the total number of the image data blocks, thereby obtaining a plurality of first partial results and a plurality of second partial results stored, and concatenating each of the first partial results and each of the second partial results to obtain a total output feature.

[0049] Among them, the 2D selective scan (SS2D) module is executed using variables with the batch size dimension as the post-position, and the calculation and data scheduling strategy that can avoid flipping operations and local variable cache is used to keep as many intermediate variables as possible in the TPU cache to reduce the number and total amount of data transfer. The size of the model weight parameter is [4*C,1].

[0050] Specifically, Figure 4 As shown, Figure 4A schematic diagram of the calculation of the total output features provided for the embodiment of the present application. First, the 4*C dimensions of the first intermediate variable, the second intermediate variable, and the third intermediate variable are respectively segmented to obtain two first intermediate variables after segmentation, two second intermediate variables after segmentation, and two third intermediate variables after segmentation. Then, the 4*C dimension (i.e., the second dimension) of the product of the global receptive field and the model weight parameter of the module is segmented to obtain two fourth intermediate variables after segmentation. Then, L loops are executed. In each loop, the first variable slices after segmentation are extracted from the L-dimensional order (i.e., the first preset order) and the reverse order (i.e., the second preset order) to participate in the operation, and the partial results of the module output feature one (the shape is [1,2*C, 5 number of images]) and the partial results of the module output feature two (the shape is [1,2*C, number of images]). After L cycles, the completed module output feature 1 (i.e., multiple first partial results) and module output feature 2 (i.e., multiple second partial results, with a shape of [L, 2*C, number of images]) are obtained, and they are spliced ​​in the 2*C dimension to obtain a total output feature with a final shape of [L, 4*C, number of images]. That is, in each cycle, different slices (first variable slices and second variable slices) are selected from each intermediate variable in turn through the first preset order and the second preset order for calculation, and finally the first partial results calculated by the first variable slices are stored according to the first preset order, and the second partial results calculated by the second variable slices are stored according to the second preset order, until the current cycle execution times reaches half of the total number of image data blocks, and multiple first partial results and multiple second partial results are spliced ​​to obtain a total output feature with a size of [L, 4*C, number of images].

[0051] Therefore, when executing the Cross-Scan module, the data flipping operation is not used. Instead, a specific calculation and data scheduling strategy is used in the 2D selective scanning (SS2D) module to achieve the same effect, thereby avoiding the problem that the execution efficiency of flipping the image data is very low, resulting in the flipping of a large number of images, an operator with no computational amount, which takes up a lot of inference time, and improving the inference efficiency.

[0052] In step 206, the total output feature is subjected to an inverse dimensional transformation to obtain an inverse transformation result.

[0053] Among them, the inverse dimension transformation is the opposite operation of the previous dimension transformation, which restores the dimension of the total output feature to a structure suitable for subsequent processing. Since the dimension transformation was performed previously to facilitate certain calculations, the dimension needs to be restored now to connect with other modules or operations. The inverse transformation result is the result obtained after the total output feature is inversely transformed in dimension, and its dimension structure matches the input dimension structure required for subsequent processing of the model at this stage.

[0054] In some implementations, the first dimension of the total output feature is the total number of image data blocks, the second dimension is the feature depth of the image data blocks at a specified multiple, and the third dimension is the number of images. The inverse dimensional transformation of the total output feature to obtain the inverse transformation result includes: (1) swapping the first dimension and the third dimension of the total output feature to obtain the total output feature after dimension transformation; (2) Reshape the second dimension of the total output features after the dimensional transformation to obtain an inverse transformation result, wherein the first dimension of the inverse transformation result is the number of images, the second dimension is the specified multiple, the third dimension is the feature depth of the image data block, the fourth dimension is the vertical number, and the fifth dimension is the horizontal number.

[0055] Among them, see Figure 5 , Figure 5 A specific schematic diagram of the inverse transformation process provided for an embodiment of the present application. The output result of the 2D selective scanning module (i.e., the total output feature) is inversely transformed in dimension, and the dimension representing the number of images is prepended to the first dimension. Specifically, for the module output result of the shape [L, 4*C, number of images], the first and third dimensions are swapped, and the output result shape is [number of images, 4*C, L], and the dimensions are adjusted to obtain an inverse transformation result of the shape [number of images, 4, C, H, W].

[0056] In step 207, four-way scanning integration is performed on the inverse transformation result to obtain a second image feature.

[0057] Among them, the second image feature is the image feature obtained after the four-way scanning integration of the Cross-Merge module. It is the result of information integration again after the total output feature undergoes dimensional inverse transformation. Compared with the previous features, it may be more representative and discriminative, and provide better feature representation for the final image classification and other tasks.

[0058] In some implementations, the horizontal number is the same as the vertical number, and the inverse transformation result is subjected to four-way scanning integration to obtain the second image feature, including: (1) multiplying the fourth dimension and the fifth dimension of the inverse transformation result by the total number of image data blocks as the fourth dimension to obtain a first feature representation; (2) dividing the first feature representation in a third dimension to obtain a first sub-feature representation and a second sub-feature representation, wherein the first dimension of the first sub-feature representation and the second sub-feature representation is the number of images, the second dimension is half of the specified multiple, the third dimension is the feature depth of the image data block, and the fourth dimension is the total number of the image data blocks; (3) performing a bitwise sum of the first sub-feature representation and the second sub-feature representation to obtain a second feature representation, wherein a first dimension of the second image feature representation is the number of images, a second dimension is half of the specified multiple, a third dimension is the feature depth of the image data block, and a fourth dimension is the total number of the image data blocks; (4) dividing the second feature representation in a third dimension to obtain a third sub-feature representation and a fourth sub-feature representation, wherein the first dimension of the third sub-feature representation and the fourth sub-feature representation is the number of images, the second dimension is one-fourth of the specified multiple, the third dimension is the feature depth of the image data block, and the fourth dimension is the total number of the image data blocks; (5) reshaping the fourth dimension represented by the third sub-feature to obtain the longitudinal quantity as the fourth dimension and the lateral quantity as the fifth dimension; (6) reshaping the fourth dimension represented by the fourth sub-feature to obtain the horizontal quantity as the fourth dimension and the vertical quantity as the fifth dimension; (7) Performing bitwise summation on the third sub-feature representation and the fourth sub-feature representation to obtain a second image feature.

[0059] Among them, Figure 6 As shown, Figure 6A schematic diagram of the four-way scanning integration process provided for an embodiment of the present application. When executing the Cross-Merge module, the data flipping operation is not used. The batch size dimension, which is in the first dimension by default, is omitted. Specifically, in the Cross-Merge module, the inverse transformation result of the input of the module (the image feature dimension size is [number of images, 4, C, H, W]) is reshaped, that is, the total number of image data blocks obtained by multiplying the fourth dimension and the fifth dimension of the inverse transformation result is used as the fourth dimension, and the obtained feature size is [number of images, 4, C, H*W] The first feature representation; then, the first feature representation is divided in the third dimension to obtain two first sub-feature representations and a second sub-feature representation with a feature size of [number of images, 2, C, H*W]; then, the elements in the same position of the current two features are summed, and the obtained feature size is [number of images, 2, C, H*W]; then the third dimension is divided to obtain two third sub-feature representations and a fourth sub-feature representation with a feature size of [number of images, C, H*W]; then, the third sub-feature representation is reshaped, that is, the fourth dimension is reshaped to obtain the longitudinal number as the third dimension and the transverse number as the fourth dimension, and the obtained feature size is [number of images, C, H, W]; the fourth sub-feature representation is reshaped and the dimensions H and W are swapped, that is, the fourth dimension of the fourth sub-feature representation is reshaped to obtain the transverse number as the third dimension and the longitudinal number as the fourth dimension, and the obtained feature size is [number of images, C, W, H]. In the embodiment of the present application, H and W are equal. Finally, the elements of the same position of the reshaped third sub-feature representation and the fourth sub-feature representation are summed to obtain the final output feature of size [number of images, C, H, W]. The above steps complete the process of the Cross-Merge module integrating the four-way scanning image features to obtain a single image feature of size [number of images, C, H, W]. The time consumption of the Cross-Scan module and the Cross-Merge module is greatly reduced, further improving the model reasoning speed.

[0060] In step 208, when the current processing module is the last module, an image classification result is determined based on the second image feature.

[0061] Among them, since the VMamba model includes multiple Block modules, if the current processing module is the last module, it means that there are no other Block modules subsequently. Then, after further processing of the last several layers of the VMamba model (such as the layernrom normalization layer, the global average layer, and the linear classification layer), the second image feature block is mapped to the corresponding image category, which is convenient for the subsequent analysis of the inference results.

[0062] Specifically, the inference results of the VMamba model in the computing TPU are obtained, and the inference results are parsed according to the task type and the dataset label list to obtain the classification results, detection results, or semantic segmentation results of the model for the current batch of image data. Taking image classification as an example, the maximum classification prediction probability of each sample in the inference result on different categories is calculated, and the label corresponding to the maximum value is taken as the predicted image classification result.

[0063] In some embodiments, the method further comprises: When the current processing module is not the last module, the second image feature is determined as the first image feature, the next module is determined as the current processing module, and the step of inputting the first image feature into the current processing module in the visual state space model and performing a four-way scan on each of the first image features to obtain an initial global receptive field is returned to execution until the current processing module is the last module.

[0064] Among them, since multiple Block modules are executed sequentially, if the current processing module is not the last module, it means that the second image feature needs to be used as the input of the next Block module, then the second image feature is determined as the first image feature, and the next module is determined as the current processing module, and the process returns to execute the step of inputting the first image feature into the current processing module in the visual state space model, performing a four-way scan on each of the first image features, and obtaining the initial global receptive field, until the current processing module is the last module.

[0065] As can be seen from the above, the embodiment of the present application obtains the first image feature of each image in the batch of images; inputs the first image feature into the current processing module in the visual state space model, performs a four-way scan on each first image feature to obtain an initial global receptive field; performs different feature adjustments on the initial global receptive field to obtain an initial first intermediate variable, an initial second intermediate variable and an initial third intermediate variable respectively; performs dimensional transformation on the initial first intermediate variable, the initial second intermediate variable, the initial third intermediate variable and the initial global receptive field to obtain a first intermediate variable, a second intermediate variable, a third intermediate variable and a global receptive field, wherein the last dimension of the first intermediate variable, the second intermediate variable, the third intermediate variable and the global receptive field is the number of images in the batch of images; determines the total output feature based on the first intermediate variable, the second intermediate variable, the third intermediate variable, the global receptive field and the model weight parameter of the current processing module; performs a dimensional inverse transformation on the total output feature to obtain an inverse transformation result; performs a four-way scan integration on the inverse transformation result to obtain a second image feature; when the current processing module is the last module, determines the image classification result based on the second image feature corresponding to each module. Compared with the related art, in which the parallel computing capability of the TPU cannot be fully utilized due to the dimension problem of the intermediate variables, the embodiment of the present application transforms the dimensions of the intermediate variables so that the last dimension of each intermediate variable is the number of images in the batch of images, thereby avoiding the problem that the TPU cannot be processed in parallel due to the last dimension being 1, thereby exerting the computing potential of the high-performance TPU and improving the utilization rate of the TPU, thereby improving data processing efficiency.

[0066] The specific implementation of the above steps can be found in the previous embodiments, which will not be described in detail here.

[0067] In order to better implement the data processing method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above data processing method. The meanings of the terms are the same as those in the above data processing method, and the specific implementation details can refer to the description in the method embodiment.

[0068] See also Figure 7 , Figure 7 The data processing device provided in the embodiment of the present application is a schematic diagram of the structure of the data processing device, which is applied to a computer device. The data processing device may include an acquisition unit 601, an input unit 602, an adjustment unit 603, a transformation unit 604, a first determination unit 605, an inverse transformation unit 606, an integration unit 607, and a second determination unit 608.

[0069] An acquisition unit 601 is used to acquire a first image feature of each image in a batch of images; An input unit 602, configured to input the first image features into a current processing module in a visual state space model, and perform four-way scanning on each of the first image features to obtain an initial global receptive field; An adjustment unit 603 is used to perform different feature adjustments on the initial global receptive field to obtain an initial first intermediate variable, an initial second intermediate variable, and an initial third intermediate variable respectively; A transformation unit 604 is used to perform dimension transformation on the initial first intermediate variable, the initial second intermediate variable, the initial third intermediate variable, and the initial global receptive field to obtain a first intermediate variable, a second intermediate variable, a third intermediate variable, and a global receptive field, wherein the last dimension of the first intermediate variable, the second intermediate variable, the third intermediate variable, and the global receptive field is the number of images in the batch of images; A first determining unit 605 is used to determine a total output feature based on the first intermediate variable, the second intermediate variable, the third intermediate variable, the global receptive field, and a model weight parameter of a current processing module; An inverse transformation unit 606 is used to perform a dimensional inverse transformation on the total output feature to obtain an inverse transformation result; An integration unit 607, configured to perform four-way scanning integration on the inverse transformation result to obtain a second image feature; The second determining unit 608 is configured to determine an image classification result based on the second image feature when the current processing module is the last module.

[0070] In some embodiments, the first dimension of the initial first intermediate variable and the initial second intermediate variable is the number of images of the batch of images, the second dimension is the feature depth of the image data blocks at a specified multiple, the third dimension is the total number of image data blocks, and the fourth dimension is a preset hyperparameter; the first dimension of the initial third intermediate variable is the number of images, the second dimension is the feature depth of the image data blocks at a specified multiple, the third dimension is the preset hyperparameter, and the fourth dimension is the total number of image data blocks; the first dimension of the initial global receptive field is the number of images, the second dimension is the feature depth of the image data blocks at a specified multiple, and the third dimension is the total number of image data blocks; A first swapping subunit is used to swap the first dimension and the fourth dimension of the initial first intermediate variable, the initial second intermediate variable and the initial third intermediate variable, respectively, to obtain a first intermediate variable, a second intermediate variable and a third intermediate variable; The second exchange subunit is used to exchange the first dimension and the third dimension of the initial global receptive field to obtain a global receptive field.

[0071] In some embodiments, the first determining unit 605 includes: a division subunit, configured to divide the current second dimension of the first intermediate variable, the second intermediate variable, and the third intermediate variable to obtain two divided first intermediate variables, two divided second intermediate variables, and two divided third intermediate variables, and to divide the second dimension of the product of the global receptive field and the model weight parameter of the module to obtain two divided fourth intermediate variables; an extracting subunit, configured to extract the first variable slice to be calculated from one of the divided first intermediate variables, one of the divided second intermediate variables, one of the divided third intermediate variables, and one of the divided fourth intermediate variables respectively in a first preset order, and to extract the second variable slice to be calculated from another of the divided first intermediate variables, another of the divided second intermediate variables, another of the divided third intermediate variables, and another of the divided fourth intermediate variables respectively in a second preset order, wherein the first preset order is opposite to the second preset order; a calculation subunit, configured to calculate each of the first variable slices to obtain a first partial result, and to calculate each of the second variable slices to obtain a second partial result; a storage subunit, configured to store the first part of the results according to the first preset order, store the second part of the results according to the second preset order, and increase the number of loop executions by one to obtain a current number of loop executions; The splicing subunit is used to return to the step of extracting the first variable slice to be calculated from one of the first intermediate variables after division, one of the second intermediate variables after division, one of the third intermediate variables after division, and one of the fourth intermediate variables after division in a first preset order, and extracting the second variable slice to be calculated from another of the first intermediate variables after division, another of the second intermediate variables after division, another of the third intermediate variables after division, and another of the fourth intermediate variables after division in a second preset order, when the current loop is executed the number of times less than the total number of image data blocks, to obtain a plurality of first partial results stored and a plurality of second partial results stored, and to splice each of the first partial results and each of the second partial results to obtain a total output feature.

[0072] In some embodiments, the input unit 602 includes: a first copying subunit, configured to copy the first image feature for each of the first image features to obtain two first image features, wherein the first dimension of the first image feature is the feature depth of the image data block, the second dimension is the longitudinal number of the image data blocks, and the third dimension is the lateral number of the image data blocks; a first multiplication operator unit, configured to multiply the third dimension and the fourth dimension of one of the first image features, and use the result as the third dimension to obtain a third image feature; a third swap subunit, configured to swap the third dimension and the fourth dimension of another of the first image features, and multiply the swapped third dimension by the swapped fourth dimension, and use the result as the third dimension to obtain a fourth image feature; A first stacking subunit, configured to stack the third image feature and the fourth image feature in a dimension of feature depth of the image data block to obtain a stacked feature; A second replication subunit is used to replicate the stacking feature to obtain two stacking features; The second stacking subunit is used to stack each of the stacking features in the dimension of the feature depth of the image data block to obtain an initial global receptive field.

[0073] In some embodiments, the first dimension of the total output feature is the total number of the image data blocks, the second dimension is the feature depth of the image data blocks at a specified multiple, and the third dimension is the number of images. The inverse transform unit 606 includes: A fourth exchange subunit is used to exchange the first dimension and the third dimension of the total output feature to obtain the total output feature after dimension transformation; The first reshaping subunit is used to reshape the second dimension of the total output features after the dimensional transformation to obtain an inverse transformation result, wherein the first dimension of the inverse transformation result is the number of images, the second dimension is the specified multiple, the third dimension is the feature depth of the image data block, the fourth dimension is the longitudinal number, and the fifth dimension is the lateral number.

[0074] In some embodiments, the horizontal quantity is the same as the vertical quantity, and the integration unit 607 includes: a second multiplication operator unit, configured to obtain a first feature representation by multiplying the fourth dimension and the fifth dimension of the inverse transformation result by a total number of image data blocks as the fourth dimension; A first segmentation subunit is used to segment the first feature representation in a third dimension to obtain a first sub-feature representation and a second sub-feature representation, wherein the first dimension of the first sub-feature representation and the second sub-feature representation is the number of images, the second dimension is half of the specified multiple, the third dimension is the feature depth of the image data block, and the fourth dimension is the total number of the image data blocks; A first bitwise summation subunit is used to perform bitwise summation on the first sub-feature representation and the second sub-feature representation to obtain a second feature representation, wherein a first dimension of the second image feature representation is the number of images, a second dimension is half of the specified multiple, a third dimension is the feature depth of the image data block, and a fourth dimension is the total number of the image data blocks; A second segmentation subunit is used to segment the second feature representation in a third dimension to obtain a third sub-feature representation and a fourth sub-feature representation, wherein the first dimension of the third sub-feature representation and the fourth sub-feature representation is the number of images, the second dimension is one-fourth of the specified multiple, the third dimension is the feature depth of the image data block, and the fourth dimension is the total number of the image data blocks; A second reshaping subunit is used to reshape the third dimension represented by the third sub-feature to obtain the longitudinal quantity as the third dimension and the lateral quantity as the fourth dimension; A third reshaping subunit is used to reshape the third dimension represented by the fourth sub-feature to obtain the horizontal quantity as the third dimension and the vertical quantity as the fourth dimension; The second bitwise summing subunit is used to perform bitwise summing of the third sub-feature representation and the fourth sub-feature representation to obtain a second image feature.

[0075] In some embodiments, the apparatus further comprises: An execution unit is used to, when the current processing module is not the last module, determine the second image feature as the first image feature, determine the next module as the current processing module, and return to execute the step of inputting the first image feature into the current processing module in the visual state space model, performing a four-way scan on each of the first image features to obtain an initial global receptive field, until the current processing module is the last module.

[0076] The specific implementation of each of the above units can be found in the previous embodiments, which will not be described in detail here.

[0077] As can be seen from the above, the embodiment of the present application obtains the first image feature of each image in the batch of images through the acquisition unit 601; the input unit 602 inputs the first image feature into the current processing module in the visual state space model, and performs a four-way scan on each of the first image features to obtain an initial global receptive field; the adjustment unit 603 performs different feature adjustments on the initial global receptive field to obtain an initial first intermediate variable, an initial second intermediate variable, and an initial third intermediate variable, respectively; the transformation unit 604 performs a dimensional transformation on the initial first intermediate variable, the initial second intermediate variable, the initial third intermediate variable, and the initial global receptive field to obtain the first intermediate variable, the second intermediate variable, the third intermediate variable, and the initial global receptive field. and global receptive field, the first intermediate variable, the second intermediate variable, the third intermediate variable and the last dimension of the global receptive field are the number of images of the batch image; the first determination unit 605 determines the total output feature based on the first intermediate variable, the second intermediate variable, the third intermediate variable, the global receptive field and the model weight parameter of the current processing module; the inverse transformation unit 606 performs a dimensional inverse transformation on the total output feature to obtain an inverse transformation result; the integration unit 607 performs a four-way scanning integration on the inverse transformation result to obtain a second image feature; the second determination unit 608 determines the image classification result based on the second image feature when the current processing module is the last module. Compared with the related art, the parallel computing capability of the TPU cannot be fully utilized due to the dimension problem of the intermediate variables. The embodiment of the present application transforms the dimension of the intermediate variables so that the last dimension of each intermediate variable is the number of images of the batch image, avoiding the problem that the TPU cannot be processed in parallel due to the last dimension being 1, thereby exerting the computing potential of the high-performance TPU and improving the utilization rate of the TPU, thereby improving the data processing efficiency.

[0078] The specific implementation of each of the above units can be found in the previous embodiments, which will not be described in detail here.

[0079] Reference Figure 8 , Figure 8A block diagram of a portion of a computer device 1000 as a server for implementing an embodiment of the present disclosure. The computer device 1000 may have relatively large differences due to different configurations or performances, and may include one or more tensor processing units (TPUs) 622 (e.g., one or more tensor processing units) and a memory 632, and one or more storage media 630 (e.g., one or more mass storage devices) storing application programs 642 or data 644. Among them, the memory 632 and the storage medium 630 may be short-term storage or persistent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), each of which may include a series of instruction operations on the server 600. Furthermore, the tensor processing unit 622 may be configured to communicate with the storage medium 630 to execute a series of instruction operations in the storage medium 630 on the computer device 1000.

[0080] The computer device 1000 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input and output interfaces 658, and / or one or more operating systems 641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0081] The tensor processing unit 622 in the computer device 1000 may be used to execute the data processing method of the embodiment of the present disclosure, for example: Obtaining the first image feature of each image in the batch of images; Inputting the first image features into a current processing module in a visual state space model, performing four-directional scanning on each of the first image features to obtain an initial global receptive field; Performing different feature adjustments on the initial global receptive field to obtain an initial first intermediate variable, an initial second intermediate variable, and an initial third intermediate variable respectively; Performing dimension transformation on the initial first intermediate variable, the initial second intermediate variable, the initial third intermediate variable, and the initial global receptive field to obtain a first intermediate variable, a second intermediate variable, a third intermediate variable, and a global receptive field, wherein the last dimension of the first intermediate variable, the second intermediate variable, the third intermediate variable, and the global receptive field is the number of images in the batch of images; Determine a total output feature based on the first intermediate variable, the second intermediate variable, the third intermediate variable, the global receptive field, and a model weight parameter of a current processing module; Performing an inverse dimensionality transformation on the total output feature to obtain an inverse transformation result; Performing four-way scanning integration on the inverse transformation result to obtain a second image feature; When the current processing module is the last module, an image classification result is determined based on the second image feature.

[0082] The embodiments of the present disclosure further provide a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the data processing methods of the aforementioned embodiments.

[0083] The present disclosure also provides a computer program product, which includes a computer program. The processor of a computer device reads and executes the computer program, so that the computer device executes the above-mentioned data processing method. For example: Obtaining the first image feature of each image in the batch of images; Inputting the first image features into a current processing module in a visual state space model, performing four-directional scanning on each of the first image features to obtain an initial global receptive field; Performing different feature adjustments on the initial global receptive field to obtain an initial first intermediate variable, an initial second intermediate variable, and an initial third intermediate variable respectively; Performing dimension transformation on the initial first intermediate variable, the initial second intermediate variable, the initial third intermediate variable, and the initial global receptive field to obtain a first intermediate variable, a second intermediate variable, a third intermediate variable, and a global receptive field, wherein the last dimension of the first intermediate variable, the second intermediate variable, the third intermediate variable, and the global receptive field is the number of images in the batch of images; Determine a total output feature based on the first intermediate variable, the second intermediate variable, the third intermediate variable, the global receptive field, and a model weight parameter of a current processing module; Performing an inverse dimensionality transformation on the total output feature to obtain an inverse transformation result; Performing four-way scanning integration on the inverse transformation result to obtain a second image feature; When the current processing module is the last module, an image classification result is determined based on the second image feature.

[0084] In addition, the terms "comprises" and "includes" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, product or apparatus.

[0085] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0086] It should be understood that in the description of the embodiments of the present application, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0087] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0088] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0089] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0090] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0091] It should also be understood that the various implementations provided in the embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0092] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0093] The above is a specific description of the implementation method of the present application, but the present application is not limited to the above-mentioned implementation method. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A data processing method, characterized in that: include: Obtaining the first image feature of each image in the batch of images; Inputting the first image features into a current processing module in a visual state space model, performing four-directional scanning on each of the first image features to obtain an initial global receptive field; Performing different feature adjustments on the initial global receptive field to obtain an initial first intermediate variable, an initial second intermediate variable, and an initial third intermediate variable respectively; Performing dimension transformation on the initial first intermediate variable, the initial second intermediate variable, the initial third intermediate variable, and the initial global receptive field to obtain a first intermediate variable, a second intermediate variable, a third intermediate variable, and a global receptive field, wherein the last dimension of the first intermediate variable, the second intermediate variable, the third intermediate variable, and the global receptive field is the number of images in the batch of images; Determine a total output feature based on the first intermediate variable, the second intermediate variable, the third intermediate variable, the global receptive field, and a model weight parameter of a current processing module; Performing an inverse dimensionality transformation on the total output feature to obtain an inverse transformation result; Performing four-way scanning integration on the inverse transformation result to obtain a second image feature; When the current processing module is the last module, an image classification result is determined based on the second image feature.

2. The data processing method according to claim 1, characterized in that: The first dimension of the initial first intermediate variable and the initial second intermediate variable is the number of images of the batch images, the second dimension is the feature depth of the image data blocks of the specified multiple, the third dimension is the total number of image data blocks, and the fourth dimension is the preset hyperparameter. The first dimension of the initial third intermediate variable is the number of images, the second dimension is the feature depth of the image data blocks of the specified multiple, the third dimension is the preset hyperparameter, and the fourth dimension is the total number of image data blocks. The first dimension of the initial global receptive field is the number of images, the second dimension is the feature depth of the image data blocks of the specified multiple, and the third dimension is the total number of image data blocks. The step of performing dimension transformation on the initial first intermediate variable, the initial second intermediate variable, the initial third intermediate variable, and the initial global receptive field to obtain the first intermediate variable, the second intermediate variable, the third intermediate variable, and the global receptive field includes: Respectively swapping the first dimension and the fourth dimension of the initial first intermediate variable, the initial second intermediate variable, and the initial third intermediate variable to obtain a first intermediate variable, a second intermediate variable, and a third intermediate variable; The first dimension and the third dimension of the initial global receptive field are swapped to obtain a global receptive field.

3. The data processing method according to claim 2, characterized in that: The determining of the total output feature based on the first intermediate variable, the second intermediate variable, the third intermediate variable, the global receptive field, and the model weight parameter of the current processing module includes: The current second dimension of the first intermediate variable, the second intermediate variable and the third intermediate variable is divided to obtain two divided first intermediate variables, two divided second intermediate variables and two divided third intermediate variables, and the second dimension of the product of the global receptive field and the model weight parameter of the module is divided to obtain two divided fourth intermediate variables; Extract the first variable slice to be calculated from one of the first intermediate variables after division, one of the second intermediate variables after division, one of the third intermediate variables after division, and one of the fourth intermediate variables after division in a first preset order, and extract the second variable slice to be calculated from another of the first intermediate variables after division, another of the second intermediate variables after division, another of the third intermediate variables after division, and another of the fourth intermediate variables after division in a second preset order, wherein the first preset order is opposite to the second preset order; Calculate each of the first variable slices to obtain a first partial result, and calculate each of the second variable slices to obtain a second partial result; The first part of the results is stored in the first preset order, the second part of the results is stored in the second preset order, and the number of loop executions is increased by one to obtain the current number of loop executions; When the number of executions of the current loop does not reach half of the total number of the image data blocks, the steps of extracting the first variable slice to be calculated from one of the first intermediate variables after division, one of the second intermediate variables after division, one of the third intermediate variables after division and one of the fourth intermediate variables after division in the first preset order, and extracting the second variable slice to be calculated from another of the first intermediate variables after division, another of the second intermediate variables after division, another of the third intermediate variables after division and another of the fourth intermediate variables after division in the second preset order are returned to be executed until the number of executions of the current loop reaches half of the total number of the image data blocks, and a plurality of first partial results and a plurality of second partial results are stored, and each of the first partial results and each of the second partial results are spliced ​​to obtain the total output feature.

4. The data processing method according to any one of claims 1 to 3, characterized in that: The four-directional scanning of each of the first image features to obtain an initial global receptive field includes: For each of the first image features, the first image feature is copied to obtain two first image features, wherein the first dimension of the first image feature is the feature depth of the image data block, the second dimension is the longitudinal number of the image data blocks, and the third dimension is the lateral number of the image data blocks; multiplying the third dimension of one of the first image features by the fourth dimension, and taking the result as the third dimension, to obtain a third image feature; Swapping the third dimension and the fourth dimension of another of the first image features, and multiplying the swapped third dimension by the swapped fourth dimension, using the result as the third dimension, to obtain a fourth image feature; stacking the third image feature and the fourth image feature in the dimension of feature depth of the image data block to obtain a stacked feature; Copying the stacking feature to obtain two stacking features; Each of the stacked features is stacked in the dimension of the feature depth of the image data block to obtain an initial global receptive field.

5. The data processing method according to claim 4, characterized in that: The first dimension of the total output feature is the total number of the image data blocks, the second dimension is the feature depth of the image data blocks of a specified multiple, and the third dimension is the number of images. The inverse dimension transformation of the total output feature to obtain the inverse transformation result includes: Swapping the first dimension and the third dimension of the total output feature to obtain a total output feature after dimension transformation; The second dimension of the total output feature after the dimensional transformation is reshaped to obtain an inverse transformation result, wherein the first dimension of the inverse transformation result is the number of images, the second dimension is the specified multiple, the third dimension is the feature depth of the image data block, the fourth dimension is the longitudinal number, and the fifth dimension is the lateral number.

6. The data processing method according to claim 5, characterized in that: The horizontal number is the same as the vertical number, and the inverse transformation result is subjected to four-way scanning integration to obtain a second image feature, including: The total number of image data blocks obtained by multiplying the fourth dimension and the fifth dimension of the inverse transformation result is used as the fourth dimension to obtain a first feature representation; The first feature representation is divided in a third dimension to obtain a first sub-feature representation and a second sub-feature representation, wherein the first dimension of the first sub-feature representation and the second sub-feature representation is the number of images, the second dimension is half of the specified multiple, the third dimension is the feature depth of the image data block, and the fourth dimension is the total number of the image data blocks; Performing a bitwise summation on the first sub-feature representation and the second sub-feature representation to obtain a second feature representation, wherein a first dimension of the second image feature representation is the number of images, a second dimension is half of the specified multiple, a third dimension is the feature depth of the image data block, and a fourth dimension is the total number of the image data blocks; The second feature representation is divided in a third dimension to obtain a third sub-feature representation and a fourth sub-feature representation, wherein the first dimension of the third sub-feature representation and the fourth sub-feature representation is the number of images, the second dimension is one-fourth of the specified multiple, the third dimension is the feature depth of the image data block, and the fourth dimension is the total number of the image data blocks; Reshape the third dimension represented by the third sub-feature to obtain the longitudinal quantity as the third dimension and the lateral quantity as the fourth dimension; Reshape the third dimension represented by the fourth sub-feature to obtain the horizontal quantity as the third dimension and the vertical quantity as the fourth dimension; The third sub-feature representation and the fourth sub-feature representation are bitwise summed to obtain a second image feature.

7. The data processing method according to claim 5, characterized in that: The method further comprises: When the current processing module is not the last module, the second image feature is determined as the first image feature, the next module is determined as the current processing module, and the step of inputting the first image feature into the current processing module in the visual state space model and performing a four-way scan on each of the first image features to obtain an initial global receptive field is returned to execution until the current processing module is the last module.

8. A data processing device, characterized in that: include: An acquisition unit, used for acquiring a first image feature of each image in a batch of images; An input unit, used for inputting the first image features into a current processing module in a visual state space model, performing four-directional scanning on each of the first image features to obtain an initial global receptive field; An adjustment unit, used to perform different feature adjustments on the initial global receptive field to obtain an initial first intermediate variable, an initial second intermediate variable, and an initial third intermediate variable respectively; a transformation unit, configured to perform dimension transformation on the initial first intermediate variable, the initial second intermediate variable, the initial third intermediate variable, and the initial global receptive field to obtain a first intermediate variable, a second intermediate variable, a third intermediate variable, and a global receptive field, wherein the last dimension of the first intermediate variable, the second intermediate variable, the third intermediate variable, and the global receptive field is the number of images in the batch of images; A first determining unit, configured to determine a total output feature based on the first intermediate variable, the second intermediate variable, the third intermediate variable, the global receptive field, and a model weight parameter of a current processing module; An inverse transformation unit, used to perform a dimensional inverse transformation on the total output feature to obtain an inverse transformation result; An integration unit, used for performing four-way scanning integration on the inverse transformation result to obtain a second image feature; The second determining unit is used to determine the image classification result based on the second image feature when the current processing module is the last module.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the data processing method according to any one of claims 1 to 7.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data processing method according to any one of claims 1 to 7 is implemented.

Images