Data processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure provide a data processing method and device, electronic equipment and storage medium. The method comprises: acquiring an image, determining a to-be-processed feature corresponding to the image; inputting the to-be-processed feature into an autoregressive sequence generation model to obtain a target feature of the image; wherein the autoregressive sequence generation model comprises a lightweight self-attention subnetwork and a feedforward neural subnetwork; and performing analysis and processing on the image based on the target feature. The technical solution of the embodiments of the present disclosure reduces the requirement of the model on computing power, and even if the model is deployed on a mobile terminal, good data processing effect can be achieved. At the same time, the calculation amount in the model running process is reduced, and real-time processing of related data is facilitated.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development of artificial intelligence, neural networks have been widely used in various fields, and various types of data can be processed based on relevant models.

[0003] While existing neural networks deployed on clients can process the corresponding data, their processing performance is unsatisfactory. Therefore, a deep neural network with a self-attention mechanism is proposed. This neural network can achieve better data processing; however, it requires significant computing power. When deployed on mobile devices, due to the limited computing power of mobile devices, it suffers from poor processing performance and inadequate real-time performance. Summary of the Invention

[0004] This disclosure provides a data processing method, apparatus, electronic device, and storage medium that reduces the computational requirements of the model. Even when the model is deployed on a mobile device, it can achieve good data processing results. At the same time, it reduces the amount of computation during model operation and facilitates real-time processing of relevant data.

[0005] In a first aspect, embodiments of this disclosure provide a data processing method, including:

[0006] Acquire images;

[0007] Determine the features to be processed in the image;

[0008] The features to be processed are input into an autoregressive sequence generation model to obtain the target features of the image; wherein, the autoregressive sequence generation model includes a lightweight self-attention subnetwork and a feedforward neural subnetwork;

[0009] The image is analyzed and processed based on the target features.

[0010] Secondly, embodiments of this disclosure also provide a data processing apparatus, including:

[0011] The acquisition module is used to acquire images;

[0012] A feature determination module is used to determine the features to be processed in the image;

[0013] The target feature determination module is used to input the features to be processed into the autoregressive sequence generation model to obtain the target features of the image; wherein, the autoregressive sequence generation model includes a lightweight self-attention subnetwork and a feedforward neural subnetwork;

[0014] An analysis and processing module is used to analyze and process the image based on the target features.

[0015] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0016] One or more processors;

[0017] Storage device for storing one or more programs.

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in any of the embodiments of this disclosure.

[0019] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a data processing method as described in any of the embodiments of this disclosure.

[0020] The technical solution of this disclosure first acquires an image and determines the features to be processed in the image, that is, performs preliminary feature extraction on the image, and then inputs the features to be processed into an autoregressive sequence generation model. The lightweight self-attention subnetwork and the lightweight feedforward neural subnetwork in the model process the features sequentially. Furthermore, the image is analyzed and processed based on the target features. By modifying the structure of the autoregressive sequence generation model, the computational requirements of the model are reduced. Even if the model is deployed on a mobile device, good data processing results can be achieved. At the same time, the amount of computation during the model operation is reduced, which facilitates real-time processing of relevant data. Attached Figure Description

[0021] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0022] Figure 1 This is an overall architecture diagram of a client-side model provided in the embodiments of this disclosure;

[0023] Figure 2 This is a schematic flowchart of a data processing method provided in an embodiment of the present disclosure;

[0024] Figure 3 This is an architecture diagram of a lightweight self-attention subnetwork provided in an embodiment of this disclosure;

[0025] Figure 4This is an architectural diagram of the lightweight feedforward neural subnetwork provided in the embodiments of this disclosure;

[0026] Figure 5 This is an architecture diagram of the lightweight feature processing module provided in the embodiments of this disclosure;

[0027] Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present disclosure;

[0028] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0029] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0030] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0031] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0032] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules, or units, and are not used to limit the order of functions performed by these devices, modules, or units or their interdependencies. It should also be noted that the modifications of "a" and "a plurality of" mentioned in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0033] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0034] Before introducing this technical solution, it is possible to first combine it with Figure 1 The application scenarios and model architecture adopted in the embodiments of this disclosure are described.

[0035] When processing data using traditional deep neural network models, the models typically require significant computational resources. This indirectly necessitates that the devices deploying these models possess strong computing power. However, with the increasing number of mobile applications, there is a growing demand to deploy related models on mobile devices. To address this demand and solve the problem of insufficient computing power on mobile devices preventing efficient model operation, the architecture of the model can be adjusted and optimized based on the solutions in this disclosure.

[0036] See Figure 1 In the model architecture adopted in the embodiments of this disclosure, the sub-networks in the autoregressive sequence generation model are first lightweight optimized to obtain the lightweight self-attention sub-network (Mobile Self-Attention, MoSA) and the lightweight feed-forward network (Mobile Feed Forward Network, MoFFN) shown in the figure. It can be understood that when the algorithm containing this model is deployed to a mobile device and the initially extracted features are input into the optimized model, the features can be processed sequentially based on the above two lightweight sub-networks, thereby obtaining higher-order and more abstract features with less computational resources.

[0037] Of course, in determining the input to the optimized autoregressive sequence generation model, one can first use methods such as... Figure 1 The Convolutional Neural Networks (CNNs) shown perform initial processing on the input data, namely, extracting local features from the data. Correspondingly, after the autoregressive sequence generation model outputs the processing results, various functional models can be used to process these higher-level, more abstract features according to actual needs, thereby obtaining the output corresponding to the input data. For example, when an application is performing a classification task, it can use models such as... Figure 1 The model shown outputs the corresponding probability array.

[0038] Figure 2 This is a schematic flowchart of a data processing method provided in an embodiment of the present disclosure. The embodiments of the present disclosure are applicable to the situation where the structure of a deep neural network is modified and the modified model is deployed on a client, thereby processing relevant data based on the client's computing resources. The method can be executed by a data processing device, which can be implemented in the form of software and / or hardware, or optionally by an electronic device, such as a mobile terminal, a PC, or a server.

[0039] like Figure 2 As shown, the method includes:

[0040] S110. Acquire the image and determine the features to be processed in the image.

[0041] Before introducing the solutions of the embodiments of this disclosure, it should first be noted that the model optimized based on the embodiments of this disclosure can be deployed on a client with relatively low computing power. The client can be a program corresponding to the server, providing local services to the user. Simultaneously, the client and server can communicate based on the Hypertext Transfer Protocol (HTTP). For example, the model can be integrated into application software that supports various functions such as special effects video processing and natural language processing. This software can be installed on an electronic device, optionally a mobile terminal or a PC. The application software can be a type of software that processes data such as images, videos, and audio. Specific application software will not be described in detail here, as long as it can process data such as images, videos, and audio. It can also be a specially developed application program to add and display special effects, or integrated into a corresponding page, allowing users to process relevant data through the integrated page on a PC.

[0042] In this embodiment, in addition to processing the acquired images, various types of data can also be processed, such as image data, text data, audio data, image-text pairs, and video-text pairs. These data can be collectively referred to as the data to be processed. It is understood that the data to be processed is related to the functions that the client can perform. For example, when the client provides image processing functions to the user, the data to be processed is image data; when the client provides natural language processing functions to the user, the data to be processed is text data. Those skilled in the art should understand that, in the corresponding application scenarios, data that can be used as input to the algorithm model and from which corresponding global or local features can be extracted can all be used as the data to be processed. This embodiment does not impose specific limitations on this.

[0043] Optionally, the data to be processed includes images to be processed. For example, when a client provides a stylized image processing function to a user, it first needs to determine one or more images as input data to be processed, and then process these images based on an algorithm integrated in the client to output a stylized image that meets the user's expectations.

[0044] In this embodiment, before processing the relevant data based on the optimized autoregressive sequence generation model, other neural network architectures can be used to perform preliminary processing on the data to be processed. For example, convolutional neural networks can be used to extract corresponding local features from the data to be processed. The following section combines... Figure 1 This process will be explained.

[0045] Optionally, the data to be processed is processed based on at least one convolutional neural network layer to obtain features corresponding to the data to be processed. CNN is a type of feedforward neural network that includes convolutional computation and has a deep structure; it is one of the representative algorithms of deep learning. CNN also has representation learning capabilities, enabling translation-invariant classification of input information according to its hierarchical structure. Further details are omitted here. When the data to be processed is an image, the features to be processed are the local image features of the image. Compared with global image features, local image features are rich in quantity within the image, have low correlation between features, and do not affect the detection and matching of other features due to the disappearance of some features under occlusion. Therefore, the features to be processed can also be understood as local expressions of image features, reflecting the local characteristics of the image. These features are applicable to applications such as image matching and retrieval.

[0046] Taking the case where the data to be processed is an image as an example, see [link / reference]. Figure 1 It can be seen that when the client receives the image to be processed as input, it can perform a convolution operation on the image with a kernel size of 3×3 and a stride of 2. Furthermore, a local region can be treated as a block and stacked multiple times based on the CNN network, such as... Figure 1 As shown, layers N1 can be stacked 3 times, layers N2 can be stacked 5 times, and layers N3 can be stacked 12 times. After the stacking operation is completed, the features to be processed corresponding to the image are obtained.

[0047] S120. Input the features to be processed into the autoregressive sequence generation model to obtain the target features of the image.

[0048] In this embodiment, once the features to be processed corresponding to the image are determined, these features can be input into the autoregressive sequence generation model for processing. The autoregressive sequence generation model can be a transformer model with lightweight optimizations to the sub-networks. This model is a model based on attention mechanisms to accelerate deep learning algorithms, and includes a lightweight Mobile Self-Attention (MoSA) sub-network and a Mobile Feed Forward Network (MoFFN) sub-network. It can be understood that when the autoregressive sequence generation model receives local features corresponding to the data to be processed, the self-attention sub-network processes the features first, and then the output is input into the feedforward neural network to obtain the target features corresponding to the data to be processed. The data processing procedures of the two sub-networks are described below.

[0049] Optionally, after the autoregressive sequence generation model receives the features to be processed, it first performs a nonlinear transformation on the features based on the branch-sharing units in the self-attention subnetwork to obtain the features to be used. The self-attention subnetwork is the neural network used to implement the model's self-attention mechanism. Those skilled in the art should understand that the self-attention mechanism is a mechanism for filtering out a small amount of important information from a large amount of information. Attention types are divided into spatial attention and temporal attention. In practical applications, it can also be divided into Soft Attention and Hard Attention. For Soft Attention, all data is paid attention to, and corresponding attention weights are calculated without setting filtering conditions. For Hard Attention, after generating all attention, some attention that does not meet the conditions is filtered out, even if its attention weight is 0. This embodiment will not elaborate further. The branch-sharing unit can be understood as a unit used to simplify the computation of the attention map. The following is combined with... Figure 3 The process of generating features to be used is explained.

[0050] Specifically, based on the background matrix processing module in the branch-sharing unit, the feature to be processed is processed to obtain a background matrix; based on the depthwise convolutional separable unit, the background matrix is ​​processed to obtain a matrix to be superimposed; the background matrix is ​​transposed to obtain a transposed matrix, and matrix multiplication is performed between the transposed matrix and the background matrix to obtain a first matrix; by performing matrix multiplication between the first matrix and the background matrix, a second matrix is ​​obtained, and by performing matrix addition between the second matrix and the matrix to be superimposed, the feature to be used is obtained.

[0051] See Figure 3It is known that in the traditional data processing of self-attention subnetworks, the three inputs Q (Query), K (Key), and V (Value), which are the features to be processed, need to be calculated separately. After obtaining the corresponding processing results (such as multiple matrices obtained through processing), a matrix needs to be randomly initialized and multiplied with the processed matrix to obtain the final attention map. However, in the autoregressive sequence generation model of this embodiment, the calculation of the attention map is simplified through the branch-sharing unit mechanism. Specifically, the feature V to be processed is first processed based on the matrix processing module in the branch-sharing unit to obtain the corresponding background matrix. After obtaining the background matrix, under the branch-sharing mechanism, not only is the background matrix processed using the depthwise convolutional separable unit to obtain the matrix to be superimposed, but the background matrix is ​​also transposed to obtain the transposed matrix. Further, the background matrix and the transposed matrix are multiplied to obtain the first matrix, and then the first matrix and the background matrix are multiplied to obtain the second matrix. Finally, the matrix to be superimposed is multiplied with the second matrix to obtain the feature to be used. It is understandable that the above processing avoids the computation of Q and K, which are the features to be processed, by the self-attention sub-network, thereby reducing the computational load by two-thirds. In practical applications, this processing method reduces the computational load of MoSA by about 3.6 times compared to traditional self-attention modules.

[0052] Furthermore, after obtaining the features to be used, the features to be processed and the features to be used can be added together to obtain the features to be applied, and then the features to be used can be processed based on the subsequent lightweight feedforward neural subnetwork.

[0053] In this embodiment, the features to be applied are processed based on at least two lightweight feature processing modules in the feedforward neural subnetwork to obtain feedforward features. The following section combines... Figure 4 The data processing procedure of a lightweight feedforward neural subnetwork is explained.

[0054] Optionally, after obtaining the features to be applied, the features are input into a lightweight feature processing module to obtain intermediate features, and then the intermediate features are input into the next lightweight feature processing module to obtain feedforward features. Feedforward features are higher-order, more abstract features compared to other features.

[0055] See Figure 4 It can be seen that in a lightweight feedforward subnetwork, at least two lightweight feature processing modules are required to process the input features in order to obtain the feedforward features corresponding to the data to be processed. The output of the first lightweight feature processing module can be the first feature to be concatenated, and the output of the second feature processing module can be the second feature to be concatenated. Finally, the two features are concatenated to obtain the feedforward features.

[0056] In this embodiment, the lightweight feature processing module is... Figure 4 See the Ghost Module in [the documentation / reference]. Figure 5 As can be seen, this module includes a depthwise convolutional layer and a depthwise separable convolutional layer. When the features to be applied are used as input to the Ghost Module, they can first be processed by a depthwise convolutional layer with a kernel size of 1×1 to obtain the depthwise convolutional processing result. Further, the depthwise convolutional processing result is processed based on the depthwise separable convolutional layer. This process can be understood as connecting a depthwise convolutional structure while keeping the features channel separated, thereby achieving spatial convolution.

[0057] For example, suppose there is a 3×3 convolutional layer with 16 input channels and 32 output channels. Specifically, the 32 3×3 convolutional kernels will traverse each data point in the 16 channels, generating 512 feature maps. These feature maps are then fused by superimposing the feature maps corresponding to each input channel to obtain one feature map. Finally, the required 32 output channels are obtained. Based on this, 16 3×3 convolutional kernels can be used to traverse the data in each of the 16 channels to obtain 16 feature maps. Before the fusion operation, 32 1×1 convolutional kernels are used to traverse these 16 feature maps and perform addition and fusion, thereby completing the depthwise separable convolution operation and obtaining either the first or second feature to be concatenated. In this embodiment, by introducing the Ghost Module to perform depthwise separable convolution on the features, the computational cost required in the feature extraction process is greatly reduced. The processing procedures of at least two lightweight feature processing modules in the lightweight feedforward neural subnetwork are described below.

[0058] Specifically, the features to be applied are input into a deep convolutional layer to obtain the first feature to be concatenated; the first feature to be concatenated is input into a deep separable convolutional layer to obtain the second feature to be concatenated; by concatenating the first and second features, an intermediate feature is obtained. The intermediate feature is input into a deep convolutional layer in the next lightweight feature processing module to obtain the third feature to be concatenated, and the third feature to be concatenated is input into a deep separable convolutional layer in the next lightweight feature processing module to obtain the fourth feature to be concatenated; by concatenating the third and fourth features, the feedforward feature is obtained. This process can be understood as performing at least two "cheap convolutions" on the features to be applied, thereby greatly reducing the computational cost of the feedforward neural network. Furthermore, compared to traditional feedforward neural networks, by introducing a Ghost Module to perform lightweight optimization on the feedforward neural subnetwork, it gains spatial dimension perception capabilities and a larger receptive field than ordinary feedforward networks. Therefore, in addition to extracting features in the token channel dimension, spatial dimension feature fusion can also be performed based on the lightweight feedforward neural subnetwork.

[0059] In this embodiment, after the feedforward neural subnetwork outputs feedforward features, the target features are obtained by performing matrix addition on the feedforward features and the features to be applied. The target features are those that can be used as input to various functional algorithms. For example, the classification model used in a classification task can directly take the target features corresponding to the image to be processed as input, process the features, and output a probability array corresponding to each element in the image. Furthermore, in this embodiment, the specific residual processing process between the augmented features and the features to be applied is similar to the residual processing process in a lightweight self-attention subnetwork, and will not be described in detail here.

[0060] It should be noted that, compared with traditional self-attention subnetworks and feedforward neural subnetworks, the two lightweight subnetworks mentioned above have eliminated the Layer Norm layer, that is, eliminated the process of layer normalization of features, thereby further reducing the computational cost of the model.

[0061] It should also be noted that, since the resolution of feature maps decreases with increasing network depth, while the number of channels adaptively increases, this embodiment of the present disclosure chooses to first process the features to be processed based on a convolutional neural network, and then input the processing result into an autoregressive sequence generation model, that is, to adopt an algorithm architecture that combines CNN and transformer neural networks. It can be understood that in the shallow layers of the network, the characteristics of CNN in extracting local information are first used to process the data to be processed, thereby obtaining the corresponding features to be processed. Then, the characteristics of transformer in extracting global information are used to process the features to be processed in depth, thereby obtaining the target features. This approach greatly reduces computational overhead and memory consumption.

[0062] S130. Analyze and process the image based on the target features.

[0063] In this embodiment, after the lightweight feedforward neural subnetwork outputs the target features as the processing result, the algorithm deployed on the client can analyze and process the target features based on the feature analysis network to obtain the target processing result. The feature analysis network can be various functional models, such as models used to perform segmentation tasks, detection tasks, or various stylized image processing tasks. Correspondingly, the target processing result is the processing result of multiple tasks. For example, when the feature analysis network includes a segmentation task, the target processing result is a probability array of multiple elements; when the feature analysis network includes a stylized image processing task, the target processing result is an image of a specific style corresponding to the image to be processed. The following section combines... Figure 1 Taking the image classification task in the example, the above processing procedure will be explained in detail.

[0064] See Figure 1 After the lightweight feedforward neural network outputs the target features, it first performs a 1×1 convolution operation on the target features. Then, it performs GAP transformation on the output of the convolutional layer based on the Global Average Pooling (GAP) layer. For example, if the output of the convolutional layer is a 6×6×3 three-dimensional feature map, after GAP transformation, it becomes an output of size 1×1×3. This can be understood as the width and height of each layer being averaged to a single value. Further, the fully connected layer, acting as a classifier, classifies the features based on their combinations, thereby obtaining a probability array corresponding to each element in the image to be processed. This array is then used as the target processing result corresponding to the image to be processed.

[0065] It should be noted that the analysis and processing in the embodiments of this disclosure includes one or more of the following: scene classification; object detection; instance segmentation; and two-dimensional / three-dimensional pose estimation. Furthermore, those skilled in the art should understand that the above image classification task is merely an example to illustrate the feature analysis network. In practical applications, the feature analysis network can be edited and deployed according to actual needs, and this disclosure does not impose specific limitations on it.

[0066] The technical solution of this disclosure first acquires an image and determines the features to be processed in the image, that is, performs preliminary feature extraction on the image, and then inputs the features to be processed into an autoregressive sequence generation model. The lightweight self-attention subnetwork and the lightweight feedforward neural subnetwork in the model process the features sequentially. Furthermore, the image is analyzed and processed based on the target features. By modifying the structure of the autoregressive sequence generation model, the computational requirements of the model are reduced. Even if the model is deployed on a mobile device, good data processing results can be achieved. At the same time, the amount of computation during the model operation is reduced, which facilitates real-time processing of relevant data.

[0067] Figure 6 This is a schematic diagram of a data processing device structure provided in an embodiment of the present disclosure, as shown below. Figure 6 As shown, the device includes: an acquisition module 210, a feature determination module 220, a target feature determination module 230, and an analysis and processing module 240.

[0068] Acquisition module 210 is used to acquire images;

[0069] The feature determination module 220 is used to determine the features to be processed in the image.

[0070] The target feature determination module 230 is used to input the features to be processed into the autoregressive sequence generation model to obtain the target features of the image; wherein, the autoregressive sequence generation model includes a lightweight self-attention subnetwork and a feedforward neural subnetwork.

[0071] The analysis and processing module 240 is used to analyze and process the image based on the target features.

[0072] Based on the above technical solutions, the target feature determination module 230 includes a feature determination unit to be used, a feature determination unit to be applied, an augmented feature determination unit, and a target feature determination unit.

[0073] The feature determination unit is used to process the feature to be processed based on the branch sharing unit in the self-attention subnetwork to obtain the feature to be used.

[0074] The feature determination unit is used to obtain the feature to be applied by performing matrix addition on the feature to be processed and the feature to be used.

[0075] An augmented feature determination unit is used to process the features to be applied based on at least two lightweight feature processing modules in the feedforward neural subnetwork to obtain feedforward features.

[0076] The target feature determination unit is used to obtain the target feature by performing matrix addition on the feedforward feature and the feature to be applied.

[0077] Optionally, the feature determination unit is further configured to process the feature to be processed based on the background matrix processing module in the branch sharing unit to obtain a background matrix; process the background matrix based on the depthwise convolutional separable unit to obtain a matrix to be superimposed; transpose the background matrix to obtain a transposed matrix, and perform matrix multiplication on the transposed matrix and the background matrix to obtain a first matrix; perform matrix multiplication on the first matrix and the background matrix to obtain a second matrix, and perform matrix addition on the second matrix and the matrix to be superimposed to obtain the feature to be used.

[0078] Optionally, the augmented feature determination unit is further configured to input the feature to be applied into the lightweight feature processing module to obtain intermediate features, and input the intermediate features into the next lightweight feature processing module to obtain the feedforward features.

[0079] Based on the above technical solutions, the lightweight feature processing module includes a depthwise convolutional layer and a depthwise separable convolutional layer.

[0080] Optionally, the augmented feature determination unit is further configured to input the feature to be applied into the deep convolutional layer to obtain a first feature to be concatenated; input the first feature to be concatenated into the depthwise separable convolutional layer to obtain a second feature to be concatenated; concatenate the first feature to be concatenated and the second feature to be concatenated to obtain the intermediate feature; input the intermediate feature into the deep convolutional layer in the next lightweight feature processing module to obtain a third feature to be concatenated, and input the third feature to be concatenated into the depthwise separable convolutional layer in the next lightweight feature processing module to obtain a fourth feature to be concatenated; and concatenate the third feature to be concatenated and the fourth feature to be concatenated to obtain the feedforward feature.

[0081] Optionally, the feature determination module 220 is further configured to process the image based on at least one convolutional neural network layer to obtain features to be processed corresponding to the image.

[0082] The technical solution provided in this embodiment first acquires an image and determines the features to be processed in the image, that is, performs preliminary feature extraction on the image. Then, the features to be processed are input into the autoregressive sequence generation model. The lightweight self-attention subnetwork and the lightweight feedforward neural subnetwork in the model process the features sequentially. Furthermore, the image is analyzed and processed based on the target features. By modifying the structure of the autoregressive sequence generation model, the computational requirements of the model are reduced. Even if the model is deployed on a mobile device, good data processing results can be achieved. At the same time, the amount of computation during the model operation is reduced, which facilitates real-time processing of relevant data.

[0083] The data processing apparatus provided in this disclosure can execute the data processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0084] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0085] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Refer to the following... Figure 7 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 7 The diagram below shows the structure of the terminal device or server 300. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0086] like Figure 7 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, pattern processor, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 306 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An edit / output (I / O) interface 305 is also connected to the bus 304.

[0087] Typically, the following devices can be connected to I / O interface 305: editing devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0088] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 306, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of embodiments of this disclosure.

[0089] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0090] The electronic device provided in this embodiment and the data processing method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0091] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the data processing method provided in the above embodiments.

[0092] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0093] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0094] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0095] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0096] Acquire images;

[0097] Determine the features of the image to be processed;

[0098] The features to be processed are input into an autoregressive sequence generation model to obtain the target features of the image; wherein, the autoregressive sequence generation model includes a lightweight self-attention subnetwork and a feedforward neural subnetwork;

[0099] The image is analyzed and processed based on the target features.

[0100] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0102] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0103] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0104] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0105] Example

[0106] According to one or more embodiments of this disclosure, [Example 1] provides a data processing method, the method comprising:

[0107] Acquire images;

[0108] Determine the features to be processed in the image;

[0109] The features to be processed are input into an autoregressive sequence generation model to obtain the target features of the image; wherein, the autoregressive sequence generation model includes a lightweight self-attention subnetwork and a feedforward neural subnetwork;

[0110] The image is analyzed and processed based on the target features.

[0111] According to one or more embodiments of this disclosure, [Example 2] provides a data processing method, which further includes:

[0112] Optionally, the features to be processed are processed based on the branch-sharing units in the self-attention subnetwork to obtain the features to be used;

[0113] The feature to be applied is obtained by performing matrix addition on the feature to be processed and the feature to be used.

[0114] Based on at least two lightweight feature processing modules in the feedforward neural subnetwork, the features to be applied are processed to obtain feedforward features;

[0115] The target feature is obtained by performing matrix addition on the feedforward feature and the feature to be applied.

[0116] According to one or more embodiments of this disclosure, [Example 3] provides a data processing method, which further includes:

[0117] Optionally, the background matrix processing module in the branch sharing unit is used to process the feature to be processed to obtain a background matrix;

[0118] The background matrix is ​​processed using depthwise separable convolutional units to obtain the matrix to be superimposed.

[0119] The background matrix is ​​transposed to obtain a transposed matrix, and the transposed matrix is ​​multiplied by the background matrix to obtain a first matrix.

[0120] The second matrix is ​​obtained by performing matrix multiplication on the first matrix and the background matrix, and the feature to be used is obtained by performing matrix addition on the second matrix and the matrix to be superimposed.

[0121] According to one or more embodiments of this disclosure, [Example 4] provides a data processing method, which further includes:

[0122] Optionally, the features to be applied are input into a lightweight feature processing module to obtain intermediate features, and the intermediate features are input into the next lightweight feature processing module to obtain the feedforward features.

[0123] According to one or more embodiments of this disclosure, [Example 5] provides a data processing method, which further includes:

[0124] Optionally, the lightweight feature processing module includes a depthwise convolutional layer and a depthwise separable convolutional layer.

[0125] According to one or more embodiments of this disclosure, [Example Six] provides a data processing method, which further includes:

[0126] Optionally, the features to be applied are input into the deep convolutional layer to obtain the first features to be concatenated;

[0127] The first feature to be spliced ​​is input into the depth-separable convolutional layer to obtain the second feature to be spliced.

[0128] The intermediate feature is obtained by splicing the first feature to be spliced ​​and the second feature to be spliced.

[0129] The intermediate features are input into the deep convolutional layer in the next lightweight feature processing module to obtain the third feature to be concatenated, and the third feature to be concatenated is input into the deep separable convolutional layer in the next lightweight feature processing module to obtain the fourth feature to be concatenated.

[0130] The feedforward feature is obtained by splicing the third and fourth features to be spliced ​​together.

[0131] According to one or more embodiments of this disclosure, [Example Seven] provides a data processing method, which further includes:

[0132] Optionally, the image can be processed based on at least one convolutional neural network layer to obtain features to be processed corresponding to the image.

[0133] According to one or more embodiments of this disclosure, [Example Eight] provides a data processing method, which further includes:

[0134] Optionally, the analysis process includes one or more of the following: scene classification; object detection; instance segmentation; two-dimensional / three-dimensional pose estimation.

[0135] According to one or more embodiments of this disclosure, [Example Nine] provides a data processing apparatus, the apparatus comprising:

[0136] The acquisition module is used to acquire images;

[0137] A feature determination module is used to determine the features to be processed in the image;

[0138] The target feature determination module is used to input the features to be processed into the autoregressive sequence generation model to obtain the target features of the image; wherein, the autoregressive sequence generation model includes a lightweight self-attention subnetwork and a feedforward neural subnetwork;

[0139] An analysis and processing module is used to analyze and process the image based on the target features.

[0140] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0141] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0142] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A data processing method, characterized in that, include: Acquire images; Determine the features to be processed in the image; The features to be processed are input into an autoregressive sequence generation model to obtain the target features of the image; wherein, the autoregressive sequence generation model includes a lightweight self-attention subnetwork and a feedforward neural subnetwork, the self-attention subnetwork includes a branch-sharing unit, the branch-sharing unit is used to perform nonlinear transformation processing on the features to be processed to obtain the features to be used; the input of the feedforward neural subnetwork is determined based on the features to be used and the features to be processed; The image is analyzed and processed based on the target features; The branch-sharing unit is a unit used to simplify the computation of attention maps. The branch-sharing unit obtains the features to be used in the following ways: Based on the background matrix processing module in the branch sharing unit, the feature to be processed is processed to obtain a background matrix; the background matrix is ​​processed based on the depthwise convolutional separable unit to obtain a matrix to be superimposed; the background matrix is ​​transposed to obtain a transposed matrix, and matrix multiplication is performed between the transposed matrix and the background matrix to obtain a first matrix; a second matrix is ​​obtained by performing matrix multiplication between the first matrix and the background matrix, and the feature to be used is obtained by performing matrix addition between the second matrix and the matrix to be superimposed.

2. The method according to claim 1, characterized in that, The step of inputting the features to be processed into the autoregressive sequence generation model to obtain the target features of the image includes: The features to be processed are obtained by processing the branch-sharing units in the self-attention subnetwork to obtain the features to be used; The feature to be applied is obtained by performing matrix addition on the feature to be processed and the feature to be used. Based on at least two lightweight feature processing modules in the feedforward neural subnetwork, the features to be applied are processed to obtain feedforward features; The target feature is obtained by performing matrix addition on the feedforward feature and the feature to be applied.

3. The method according to claim 2, characterized in that, The process of processing the features to be applied based on at least two lightweight feature processing modules in the feedforward neural subnetwork to obtain feedforward features includes: The features to be applied are input into a lightweight feature processing module to obtain intermediate features, and the intermediate features are input into the next lightweight feature processing module to obtain the feedforward features.

4. The method according to claim 3, characterized in that, The lightweight feature processing module includes a depthwise convolutional layer and a depthwise separable convolutional layer.

5. The method according to claim 4, characterized in that, The step of inputting the features to be applied into a lightweight feature processing module to obtain intermediate features, and then inputting the intermediate features into a next lightweight feature processing module to obtain the feedforward features, includes: The features to be applied are input into the deep convolutional layer to obtain the first feature to be concatenated; The first feature to be spliced ​​is input into the depth-separable convolutional layer to obtain the second feature to be spliced. The intermediate feature is obtained by splicing the first feature to be spliced ​​and the second feature to be spliced. The intermediate features are input into the deep convolutional layer in the next lightweight feature processing module to obtain the third feature to be concatenated, and the third feature to be concatenated is input into the deep separable convolutional layer in the next lightweight feature processing module to obtain the fourth feature to be concatenated. The feedforward feature is obtained by splicing the third and fourth features to be spliced ​​together.

6. The method according to any one of claims 1-5, characterized in that, The process of determining the features to be processed in the image includes: The image is processed based on at least one convolutional neural network layer to obtain the features to be processed corresponding to the image.

7. The method according to any one of claims 1-6, characterized in that, The analysis and processing includes one or more of the following: scene classification; target detection; instance segmentation; 2D / 3D pose estimation.

8. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire images; A feature determination module is used to determine the features to be processed in the image; A target feature determination module is used to input the features to be processed into an autoregressive sequence generation model to obtain the target features of the image; wherein, the autoregressive sequence generation model includes a lightweight self-attention subnetwork and a feedforward neural subnetwork, the self-attention subnetwork includes a branch-sharing unit, the branch-sharing unit is used to perform nonlinear transformation processing on the features to be processed to obtain the features to be used; the input of the feedforward neural subnetwork is determined based on the features to be used and the features to be processed; An analysis and processing module is used to analyze and process the image based on the target features; The branch-sharing unit is a unit used to simplify the computation of attention maps. The branch-sharing unit obtains the features to be used in the following ways: Based on the background matrix processing module in the branch sharing unit, the feature to be processed is processed to obtain a background matrix; the background matrix is ​​processed based on the depthwise convolutional separable unit to obtain a matrix to be superimposed; the background matrix is ​​transposed to obtain a transposed matrix, and matrix multiplication is performed between the transposed matrix and the background matrix to obtain a first matrix; a second matrix is ​​obtained by performing matrix multiplication between the first matrix and the background matrix, and the feature to be used is obtained by performing matrix addition between the second matrix and the matrix to be superimposed.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in any one of claims 1-7.

10. A storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to perform the data processing method as described in any one of claims 1-7.

Images