Data processing methods, apparatus, equipment, readable storage media and program products

Abstract

This application provides data processing methods, apparatus, devices, readable storage media, and program products, relating to fields such as artificial intelligence, with application scenarios including but not limited to online inference scenarios. The method includes: acquiring an image to be processed; inputting the image to be processed into a fused neural network model, performing online inference processing, and determining the image processing result; wherein the fused neural network model is obtained by fusing the neural network model to be processed, and during the fusion processing, at least one memory-intensive operator of the neural network model to be processed is replaced by at least one computationally intensive operator, and the at least one memory-intensive operator and at least one computationally intensive operator are used to calculate the feature map of the image; using the fused neural network model for online inference reduces the time consumed by memory read / write operations and improves the speed of online inference.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data processing method, apparatus, device, readable storage medium, and program product. Background Technology

[0002] In existing technologies, two typical operators are often found in engineering deployment scenarios of neural network models: computationally intensive operators and memory-intensive operators. In real-world engineering projects, due to the limitations of the target hardware's memory bandwidth, and because memory read / write operations corresponding to memory-intensive operators result in significant time consumption, memory-intensive operators often become the bottleneck for neural network model performance, reducing the speed of online inference. Summary of the Invention

[0003] This application addresses the shortcomings of existing methods by proposing a data processing method, apparatus, device, computer-readable storage medium, and computer program product to solve the problem of how to improve the speed of online inference.

[0004] In a first aspect, this application provides a data processing method, including:

[0005] Obtain the image to be processed;

[0006] The image to be processed is input into the fused neural network model for online inference processing to determine the image processing result.

[0007] The fused neural network model is obtained by fusing the neural network model to be processed. During the fusion process, at least one memory-intensive operator of the neural network model to be processed is replaced by at least one computationally intensive operator. The at least one memory-intensive operator and the at least one computationally intensive operator are used to calculate the feature map of the image.

[0008] In one embodiment, at least one memory-intensive operator includes a channel rearrangement operator, and at least one computationally intensive operator includes a first convolution operator.

[0009] In one embodiment, before inputting the image to be processed into the fused neural network model for online inference processing to determine the image processing result, the method further includes:

[0010] Based on the channel rearrangement operator of the neural network model to be processed, a fusion process is performed, replacing the channel rearrangement operator with the first convolution operator to obtain the fused neural network model, which includes the first convolution operator.

[0011] In one embodiment, before inputting the image to be processed into the fused neural network model for online inference processing to determine the image processing result, the method further includes:

[0012] The channel rearrangement operator and the second convolution operator of the neural network model to be processed are fused together. The channel rearrangement operator and the second convolution operator are replaced with the first convolution operator to obtain the fused neural network model. The fused neural network model includes the first convolution operator. At least two of the multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator are different from the channel sequence numbers corresponding to the convolution kernel of the second convolution operator.

[0013] In one embodiment, a fusion process is performed based on the channel rearrangement operator and the second convolution operator of the neural network model to be processed. The channel rearrangement operator and the second convolution operator are replaced with the first convolution operator to obtain the fused neural network model, including:

[0014] Based on the grouping parameters of the channel rearrangement operator, the channel sequence number corresponding to the convolution kernel of the second convolution operator, and the total number of channels corresponding to the convolution kernel of the second convolution operator, the channels of the convolution kernel of the second convolution operator are rearranged to determine the multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator.

[0015] Among them, the grouping parameter of the channel rearrangement operator is used to characterize the number of groups in which the channel rearrangement operator performs channel rearrangement operation, the channel sequence number corresponding to the convolution kernel of the second convolution operator is used to characterize the order of each channel in the convolution kernel of the second convolution operator, and the multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator are used to characterize the order of each channel in the convolution kernel of the first convolution operator.

[0016] Based on multiple channel sequence numbers, the channel rearrangement operator and the second convolution operator are replaced with the first convolution operator to obtain the fused neural network model.

[0017] In one embodiment, based on the grouping parameters of the channel rearrangement operator, the channel sequence number corresponding to the convolution kernel of the second convolution operator, and the total number of channels corresponding to the convolution kernel of the second convolution operator, the convolution kernel of the second convolution operator is rearranged to determine multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator, including:

[0018] Based on the grouping parameters of the channel rearrangement operator and the convolution kernel of the second convolution operator, determine the remainder between any channel sequence number and the grouping parameters;

[0019] Based on the grouping parameters and the total number of channels corresponding to the convolution kernel of the second convolution operator, determine the multiple between the grouping parameters and the total number of channels;

[0020] Based on the remainder between any channel sequence number and the grouping parameter, the multiple between the grouping parameter and the total number of channels, and any channel sequence number and the grouping parameter, determine one of the multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator, and there is a correspondence between any channel sequence number and a channel sequence number.

[0021] In one embodiment, the image to be processed is input into the fused neural network model for online inference processing to determine the image processing result, including:

[0022] The image to be processed is input into the fused neural network model to perform image classification and determine the image classification result.

[0023] Secondly, this application provides a data processing apparatus, comprising:

[0024] The first processing module is used to acquire the image to be processed;

[0025] The second processing module is used to input the image to be processed into the fused neural network model, perform online inference processing, and determine the image processing result.

[0026] The fused neural network model is obtained by fusing the neural network model to be processed. During the fusion process, at least one memory-intensive operator of the neural network model to be processed is replaced by at least one computationally intensive operator. The at least one memory-intensive operator and the at least one computationally intensive operator are used to calculate the feature map of the image.

[0027] Thirdly, this application provides an electronic device, including: a processor, a memory, and a bus;

[0028] A bus is used to connect the processor and memory;

[0029] Memory, used to store operation instructions;

[0030] A processor is used to execute the data processing method of the first aspect of this application by invoking operation instructions.

[0031] Fourthly, this application provides a computer-readable storage medium storing a computer program, which is used to perform the data processing method of the first aspect of this application.

[0032] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data processing method in the first aspect of this application.

[0033] The technical solution provided in this application has at least the following beneficial effects:

[0034] The process involves acquiring an image to be processed; inputting the image into a fused neural network model for online inference to determine the image processing result; wherein the fused neural network model is obtained by fusing the original neural network model, and during the fusion process, at least one memory-intensive operator in the original neural network model is replaced by at least one computationally intensive operator. Both the memory-intensive and computationally intensive operators are used to compute the feature maps of the image. Since memory read / write operations corresponding to memory-intensive operators can be time-consuming, the memory-intensive operators in the original neural network model are replaced with computationally intensive operators to obtain the fused neural network model. Using the fused neural network model for online inference reduces the time consumed by memory read / write operations and improves the speed of online inference. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0036] Figure 1 This is a schematic diagram of the architecture of the data processing system provided in the embodiments of this application;

[0037] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0038] Figure 3 A schematic diagram of data processing provided for embodiments of this application;

[0039] Figure 4 A schematic diagram of data processing provided for embodiments of this application;

[0040] Figure 5 A schematic diagram of data processing provided for embodiments of this application;

[0041] Figure 6 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0042] Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0043] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0045] Those skilled in the art will understand that, unless otherwise stated, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” indicates implementation as “A,” or implementation as “B,” or implementation as “A and B.”

[0046] It is understood that in the specific embodiments of this application, data related to data processing is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0048] This application embodiment is a data processing method provided by a data processing system, which relates to fields such as artificial intelligence.

[0049] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0050] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0051] To better understand and explain the solutions of the embodiments of this application, some technical terms involved in the embodiments of this application will be briefly explained below.

[0052] PyTorch is an open-source Python machine learning library based on Torch, used for applications such as natural language processing.

[0053] Computational graph: A computational graph is a graphical model used to describe mathematical calculations. Computational graphs are often used in the forward and backward propagation algorithms of neural networks.

[0054] The solutions provided in this application relate to artificial intelligence technology. The technical solutions of this application will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0055] To better understand the solution provided in the embodiments of this application, the solution will be described below in conjunction with a specific application scenario.

[0056] In one embodiment, Figure 1 The diagram shows an architecture of a data processing system applicable to embodiments of this application. It is understood that the data processing methods provided in these embodiments can be applied to, but are not limited to, applications such as... Figure 1 In the application scenarios shown.

[0057] In this example, as Figure 1As shown, the architecture of the data processing system in this example may include, but is not limited to, server 10, terminal 20, and database 30. Server 10, terminal 20, and database 30 can interact via network 40. Server 10 acquires the image to be processed; server 10 inputs the image to be processed into a fused neural network model for online inference processing to determine the image processing result; wherein, the fused neural network model is obtained by fusing the neural network model to be processed, and during the fusion processing, at least one memory-intensive operator of the neural network model to be processed is replaced by at least one computationally intensive operator, and the at least one memory-intensive operator and at least one computationally intensive operator are used to calculate the feature map of the image. Server 10 sends the image processing result to terminal 20, and terminal 20 displays the image processing result to the user. Server 10 sends the image processing result to database 30 and stores the image processing result in database 30.

[0058] It is understood that the above is only one example, and this embodiment is not limited here.

[0059] Terminals include, but are not limited to, smartphones (such as Android phones, iOS phones, etc.), mobile phone emulators, tablets, laptops, digital broadcast receivers, MIDs (Mobile Internet Devices), PDAs (Personal Digital Assistants), smart voice interaction devices, smart home appliances, and in-vehicle terminals.

[0060] A server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0061] The aforementioned networks may include, but are not limited to, wired networks and wireless networks. Wired networks include local area networks (LANs), metropolitan area networks (MANs), and wide area networks (WANs). Wireless networks include Bluetooth, Wi-Fi, and other networks that enable wireless communication. Specific details can be determined based on actual application scenario requirements and are not limited here.

[0062] See Figure 2 , Figure 2This illustration shows a flowchart of a data processing method provided in an embodiment of this application. The method can be executed by any electronic device, such as a server. As an optional implementation, the method can be executed by a server. For ease of description, the following descriptions of some optional embodiments will use a server as the execution subject of the method. Figure 2 As shown, the data processing method provided in this application includes the following steps:

[0063] S201, Obtain the image to be processed.

[0064] Specifically, for example, the image to be processed is obtained through the NPU (Neural-network Processing Unit) in the server.

[0065] S202, the image to be processed is input into the fused neural network model for online inference processing to determine the image processing result; wherein, the fused neural network model is obtained by fusing the neural network model to be processed, and during the fusion processing, at least one memory-intensive operator of the neural network model to be processed is replaced by at least one computationally intensive operator, and at least one memory-intensive operator and at least one computationally intensive operator are used to calculate the feature map of the image.

[0066] Specifically, the neural network model to be processed is, for example, a deep learning model, and the neural network model after fusion processing is, for example, a deep learning model. Memory-intensive operators spend most of their time on memory access, such as the Transpose operator, channel shuffle operator, and Concat operator; computation-intensive operators spend most of their time on computation, such as the conv operator and fully connected operator.

[0067] For example, the image to be processed is input into the fused neural network model through the NPU in the server for online inference processing to determine the image processing result; among which, online inference includes image classification, image detection, etc.

[0068] In this embodiment, an image to be processed is acquired; the image to be processed is input into a fused neural network model for online inference processing to determine the image processing result; wherein, the fused neural network model is obtained by fusing the neural network model to be processed, and during the fusion processing, at least one memory-intensive operator of the neural network model to be processed is replaced by at least one computationally intensive operator, and the at least one memory-intensive operator and at least one computationally intensive operator are used to calculate the feature map of the image. Since memory read and write operations corresponding to memory-intensive operators will cause time consumption, the memory-intensive operators of the neural network model to be processed are replaced with computationally intensive operators to obtain the fused neural network model; using the fused neural network model for online inference reduces the time consumption caused by memory read and write operations and improves the speed of online inference.

[0069] In one embodiment, at least one memory-intensive operator includes a channel rearrangement operator, and at least one computationally intensive operator includes a first convolution operator.

[0070] Specifically, for example, the computation graph of the neural network model to be processed contains only one channel shuffle operator, i.e., one channel rearrangement operator; the first convolution operator is, for example, a conv1×1 convolution operator; this channel shuffle operator is replaced with a conv1×1 convolution operator, meaning that the fused neural network model does not contain the channel shuffle operator, but it does contain the conv1×1 convolution operator. The channel shuffle operator can be a channel shuffle operator in PyTorch.

[0071] In one embodiment, before inputting the image to be processed into the fused neural network model for online inference processing to determine the image processing result, the method further includes:

[0072] Based on the channel rearrangement operator of the neural network model to be processed, a fusion process is performed, replacing the channel rearrangement operator with the first convolution operator to obtain the fused neural network model, which includes the first convolution operator.

[0073] Specifically, for example, such as Figure 3As shown, the tensor (multidimensional array, i.e., feature vector) corresponding to the image is input into the neural network model to be processed. The feature map corresponding to the tensor is calculated by the channel shuffle operator in the computation graph of the neural network model to obtain the output data (channel_shuffle_out) of the neural network model to be processed. There is only one channel shuffle operator in the computation graph of the neural network model to be processed, i.e., one channel rearrangement operator. The tensor is (N, C, H, W), where N represents the batch number, H represents the height of the tensor, W represents the width of the tensor, and C represents the channels of the tensor. For example, (N, C, H, W) is (1, 4, 2, 2).

[0074] For example, the feature map corresponding to the tensor is computed using the channel shuffle operator in the computation graph of the neural network model to be processed, including steps A1-A3:

[0075] Step A1: Group the tensor (N, C, H, W) using the channel shuffle operator, and reshape the dimensions of each group to obtain (N, C / / Group, Group, H, W); where Group is the grouping parameter, and C / / Group indicates that C and Group are divisible by each other, such as... Figure 3 As shown, in the tensor (N, C, H, W), C takes the value of 4, which means there are 4 channels. These 4 channels are channel 0, channel 1, channel 2 and channel 3. The channel sequence numbers corresponding to channel 0, channel 1, channel 2 and channel 3 are 0, 1, 2 and 3 respectively. Group takes the value of 2, which means that two channels are grouped together, that is, channel 0 and channel 1 are grouped together, and channel 2 and channel 3 are grouped together.

[0076] Step A2 involves transposing the output of step A1 using the channel shuffle operator to obtain (N, C / / Group, Group, H, W); for example, ... Figure 3 As shown, Dimension 1 (Channel 1) and Dimension 2 (Channel 2) are transposed.

[0077] Step A3 involves reshaping the output of step A2 using the channel shuffle operator to obtain the output data (channel_shuffle_out) of the neural network model to be processed; for example, ... Figure 3 As shown, channel_shuffle_out is (N, C, H, W), and the channel sequence numbers of channel_shuffle_out are 0, 2, 1, and 3 respectively.

[0078] It should be noted that the channel shuffle operator performs a channel rearrangement operation, rearranging the channel sequence numbers 0, 1, 2, and 3 of the tensor (N, C, H, W) to 0, 2, 1, and 3, respectively. That is, the channel sequence numbers of channel_shuffle_out are 0, 2, 1, and 3 in sequence.

[0079] For example, such as Figure 3 As shown, based on the channel shuffle operator of the neural network model to be processed, a fusion process is performed, replacing the channel shuffle operator with a conv1×1 convolution operator to obtain the fused neural network model. The fused neural network model includes a conv1×1 convolution operator, where 1×1 in conv1×1 represents the size of the convolution kernel. The four channels of the conv1×1 convolution operator are channel H0, channel H1, channel H2, and channel H3. The data block marked as 1 in channel H0 corresponds to channel 0, the data block marked as 1 in channel H1 corresponds to channel 2, the data block marked as 1 in channel H2 corresponds to channel 1, and the data block marked as 1 in channel H3 corresponds to channel 3. Therefore, by using the conv1×1 convolution operator to calculate the tensor (N, C, H, W), the output data (channel_shuffle_out) of the fused neural network model is obtained. The channel sequence numbers of channel_shuffle_out are 0, 2, 1, and 3, which correspond to channels 0, 2, 1, and 3, respectively. The output data of the fused neural network model is completely identical to the output data of the neural network model to be processed, that is, the error between the two is zero.

[0080] In one embodiment, before inputting the image to be processed into the fused neural network model for online inference processing to determine the image processing result, the method further includes:

[0081] The channel rearrangement operator and the second convolution operator of the neural network model to be processed are fused together. The channel rearrangement operator and the second convolution operator are replaced with the first convolution operator to obtain the fused neural network model. The fused neural network model includes the first convolution operator. At least two of the multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator are different from the channel sequence numbers corresponding to the convolution kernel of the second convolution operator.

[0082] Specifically, for example, such as Figure 4As shown, in the computational graph of the neural network model to be processed, the channel shuffle operator is performed after the second convolution operator convK×K. Based on the channel shuffle operator and the second convolution operator convK×K, a fusion process is performed to combine the channel shuffle operator and the second convolution operator convK×K. The shuffle operator and the second convolution operator convK×K are replaced with the first convolution operator convK×K to obtain the fused neural network model. The fused neural network model includes the first convolution operator convK×K, where K×K represents the size of the convolution kernel. The multiple channel sequence numbers corresponding to the convolution kernel of the second convolution operator convK×K are K0, K1, K2, and K3, respectively, with each K0, K1, K2, and K3 corresponding to one channel. The multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator convK×K are K0, K2, K1, and K3, respectively, with each K0, K2, K1, and K3 corresponding to one channel. Among the multiple channel sequence numbers corresponding to the convolution kernel of the second convolution operator convK×K, two channel sequence numbers K1 and K2 are different from the channel sequence numbers K2 and K1 corresponding to the convolution kernel of the first convolution operator convK×K.

[0083] For example, such as Figure 4 As shown, in the computation graph of the neural network model to be processed, the channel shuffle operator, following the second convolution operator convK×K, inputs the tensor corresponding to the image into the neural network model to be processed. The feature map corresponding to this tensor is calculated by the channel shuffle operator and the second convolution operator convK×K in the computation graph of the neural network model to be processed, obtaining the output data (channel_shuffle_out) of the neural network model to be processed. The tensor corresponding to the image is input into the fused neural network model, and the feature map corresponding to this tensor is calculated by the first convolution operator convK×K in the computation graph of the fused neural network model, obtaining the output data (channel_shuffle_out) of the fused neural network model. The output data of the neural network model to be processed and the output data of the fused neural network model are completely identical, that is, the error between the two is zero.

[0084] In one embodiment, a fusion process is performed based on the channel rearrangement operator and the second convolution operator of the neural network model to be processed. The channel rearrangement operator and the second convolution operator are replaced with the first convolution operator to obtain the fused neural network model, including:

[0085] Based on the grouping parameters of the channel rearrangement operator, the channel sequence number corresponding to the convolution kernel of the second convolution operator, and the total number of channels corresponding to the convolution kernel of the second convolution operator, the channels of the convolution kernel of the second convolution operator are rearranged to determine the multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator.

[0086] Among them, the grouping parameter of the channel rearrangement operator is used to characterize the number of groups in which the channel rearrangement operator performs channel rearrangement operation, the channel sequence number corresponding to the convolution kernel of the second convolution operator is used to characterize the order of each channel in the convolution kernel of the second convolution operator, and the multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator are used to characterize the order of each channel in the convolution kernel of the first convolution operator.

[0087] Based on multiple channel sequence numbers, the channel rearrangement operator and the second convolution operator are replaced with the first convolution operator to obtain the fused neural network model.

[0088] Specifically, for example, the grouping parameter Group based on the channel rearrangement operator and the channel sequence number Index corresponding to the convolution kernel of the second convolution operator. i The total number of channels C corresponding to the convolution kernel of the second convolution operator out The convolution kernel of the second convolution operator undergoes channel rearrangement, i.e., offline data rearrangement, to determine the multiple channel sequence numbers (Index) corresponding to the convolution kernel of the first convolution operator. o Determine the multiple channel sequence numbers (Index) corresponding to the convolution kernel of the first convolution operator. o The calculation formula (1) is shown below:

[0089]

[0090] Where Group represents the grouping parameter of the channel shuffle operator, C out Index represents the total number of channels corresponding to the convolution kernel of the second convolution operator convK×K. i Index represents the channel sequence number corresponding to the convolution kernel of the second convolution operator convK×K. o Index represents the channel sequence number corresponding to the convolution kernel of the first convolution operator convK×K. i The range of values ​​for is [0, C]. out Index i %Group represents Index i The remainder between the group and the group.

[0091] For example, such as Figure 4 As shown, Group takes the value 2, C out The value is 4, Index iThe values ​​are 0, 1, 2, and 3 respectively, that is, K0 takes the value of 0, K1 takes the value of 1, K2 takes the value of 2, and K3 takes the value of 3; through formula (1), Index o The values ​​are 0, 2, 1 and 3 respectively, that is, the order of the channels corresponding to the convolution kernel of the first convolution operator convK×K is K0, K2, K1 and K3 respectively.

[0092] In one embodiment, based on the grouping parameters of the channel rearrangement operator, the channel sequence number corresponding to the convolution kernel of the second convolution operator, and the total number of channels corresponding to the convolution kernel of the second convolution operator, the convolution kernel of the second convolution operator is rearranged to determine multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator, including:

[0093] Based on the grouping parameters of the channel rearrangement operator and the convolution kernel of the second convolution operator, determine the remainder between any channel sequence number and the grouping parameters;

[0094] Based on the grouping parameters and the total number of channels corresponding to the convolution kernel of the second convolution operator, determine the multiple between the grouping parameters and the total number of channels;

[0095] Based on the remainder between any channel sequence number and the grouping parameter, the multiple between the grouping parameter and the total number of channels, and any channel sequence number and the grouping parameter, determine one of the multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator, and there is a correspondence between any channel sequence number and a channel sequence number.

[0096] Specifically, for example, the index of any channel sequence corresponding to the Group based on the channel shuffle operator and the convolution kernel of the second convolution operator convK×K. i Determine the index of any channel sequence number. i The remainder between the group parameter and the group (Index) i %Group); The total number of channels C corresponding to the convolution kernel based on the grouping parameter Group and the second convolution operator convK×K. out Determine the grouping parameter Group and the total number of channels C. out Multiples between Based on any channel serial number Index i The remainder between the group parameter and the group (Index) i %Group), grouping parameter Group, and total number of channels C out Multiples between Any channel serial number Index i And the grouping parameter Group determines the index of one channel sequence number among the multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator convK×K.o That is, the Index is calculated using formula (1). o .

[0097] In one embodiment, such as Figure 5 As shown, the computation graph of the neural network model to be processed contains the second convolution operator convK×K (e.g., conv1×1), while the computation graph of the fused neural network model contains the first convolution operator convK×K (e.g., conv1×1). Before the second convolution operator conv1×1, the channel shuffle operator in the computation graph of the neural network model to be processed inputs the tensor corresponding to the image into the neural network model to be processed. The feature map corresponding to this tensor is then calculated using the channel shuffle operator and the second convolution operator conv1×1 in the computation graph of the neural network model to obtain the output data (Conv_out) of the neural network model to be processed. The tensor corresponding to the image is then input into the fused neural network model. The feature map corresponding to this tensor is calculated using the first convolution operator conv1×1 in the computation graph of the fused neural network model to obtain the output data (Conv_out) of the fused neural network model. The output data of the neural network model to be processed and the output data of the fused neural network model are completely identical, meaning their error is zero.

[0098] The channel sequence number corresponding to the convolution kernel of the first convolution operator conv1×1 can be determined by formula (1). i Index represents the channel sequence number corresponding to the convolution kernel of the second convolution operator conv1×1. o This represents the channel sequence number corresponding to the convolution kernel of the first convolution operator conv1×1; for example, as... Figure 5 As shown, Index i The values ​​are 0, 2, 1, and 3 respectively, Group has a value of 2, and C out The value is 4. The Index can be determined using formula (1). o The values ​​are 0, 1, 2 and 3 respectively.

[0099] In one embodiment, the image to be processed is input into the fused neural network model for online inference processing to determine the image processing result, including:

[0100] The image to be processed is input into the fused neural network model to perform image classification and determine the image classification result.

[0101] Specifically, online inference, such as image classification, and image processing results, such as image classification results.

[0102] Applying the embodiments of this application has at least the following beneficial effects:

[0103] Since memory read / write operations corresponding to the channel shuffle operator can be time-consuming, the channel shuffle operator of the neural network model to be processed is replaced with a convolution operator to obtain a fused neural network model. Using the fused neural network model for online inference reduces the time consumption caused by memory read / write operations and improves the speed of online inference.

[0104] To better understand the methods provided in the embodiments of this application, the solutions of the embodiments of this application will be further explained below with reference to specific application scenarios.

[0105] In a specific application scenario, such as an image classification scenario, see [link to example]. Figure 6 This illustrates the processing flow of a data processing method, such as... Figure 6 As shown, the data processing method provided in this application includes the following steps:

[0106] S601, the server performs fusion processing based on the channel rearrangement operator of the neural network model to be processed, replacing the channel rearrangement operator with the first convolution operator to obtain the fused neural network model, which includes the first convolution operator.

[0107] Specifically, for example, if the computation graph of the neural network model to be processed contains only one channel shuffle operator, this channel shuffle operator is replaced with a conv1×1 convolution operator. That is, the computation graph of the fused neural network model does not contain the channel shuffle operator, but the computation graph of the fused neural network model does contain the conv1×1 convolution operator.

[0108] For example, in the computation graph of the neural network model to be processed, after the channel shuffle operator in the second convolution operator convK×K, a fusion process is performed based on the channel shuffle operator and the second convolution operator convK×K. The channel shuffle operator and the second convolution operator convK×K are replaced with the first convolution operator convK×K to obtain the fused neural network model, which includes the first convolution operator convK×K.

[0109] For example, in the computation graph of the neural network model to be processed, before the channel shuffle operator and the second convolution operator conv1×1, a fusion process is performed based on the channel shuffle operator and the second convolution operator conv1×1. The channel shuffle operator and the second convolution operator conv1×1 are replaced with the first convolution operator conv1×1 to obtain the fused neural network model, which includes the first convolution operator conv1×1.

[0110] S602, the server obtains the image to be processed.

[0111] Specifically, the images to be processed include pictures, video clips, etc.

[0112] S603, the server inputs the image to be processed into the fused neural network model to perform image classification and determine the image classification result.

[0113] Specifically, the computation graph of the fused neural network model does not contain the channel shuffle operator.

[0114] Applying the embodiments of this application has at least the following beneficial effects:

[0115] Since memory read / write operations corresponding to the channel shuffle operator are time-consuming, the channel shuffle operator of the neural network model to be processed is replaced with a convolution operator to obtain a fused neural network model. Using the fused neural network model for online inference reduces the time consumption caused by memory read / write operations and improves the speed of image classification.

[0116] This application also provides a data processing apparatus, the structural schematic diagram of which is shown below. Figure 7 As shown, the data processing device 70 includes a first processing module 701 and a second processing module 702.

[0117] The first processing module 701 is used to acquire the image to be processed;

[0118] The second processing module 702 is used to input the image to be processed into the fused neural network model, perform online inference processing, and determine the image processing result; wherein, the fused neural network model is obtained by fusing the neural network model to be processed, and during the fusion processing, at least one memory-intensive operator of the neural network model to be processed is replaced by at least one computationally intensive operator, and at least one memory-intensive operator and at least one computationally intensive operator are used to calculate the feature map of the image.

[0119] In one embodiment, at least one memory-intensive operator includes a channel rearrangement operator, and at least one computationally intensive operator includes a first convolution operator.

[0120] In one embodiment, the second processing module 702 is further configured to:

[0121] Based on the channel rearrangement operator of the neural network model to be processed, a fusion process is performed, replacing the channel rearrangement operator with the first convolution operator to obtain the fused neural network model, which includes the first convolution operator.

[0122] In one embodiment, the second processing module 702 is further configured to:

[0123] The channel rearrangement operator and the second convolution operator of the neural network model to be processed are fused together. The channel rearrangement operator and the second convolution operator are replaced with the first convolution operator to obtain the fused neural network model. The fused neural network model includes the first convolution operator. At least two of the multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator are different from the channel sequence numbers corresponding to the convolution kernel of the second convolution operator.

[0124] In one embodiment, the second processing module 702 is specifically used for:

[0125] Based on the grouping parameters of the channel rearrangement operator, the channel sequence number corresponding to the convolution kernel of the second convolution operator, and the total number of channels corresponding to the convolution kernel of the second convolution operator, the channels of the convolution kernel of the second convolution operator are rearranged to determine the multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator.

[0126] Among them, the grouping parameter of the channel rearrangement operator is used to characterize the number of groups in which the channel rearrangement operator performs channel rearrangement operation, the channel sequence number corresponding to the convolution kernel of the second convolution operator is used to characterize the order of each channel in the convolution kernel of the second convolution operator, and the multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator are used to characterize the order of each channel in the convolution kernel of the first convolution operator.

[0127] Based on multiple channel sequence numbers, the channel rearrangement operator and the second convolution operator are replaced with the first convolution operator to obtain the fused neural network model.

[0128] In one embodiment, the second processing module 702 is specifically used for:

[0129] Based on the grouping parameters of the channel rearrangement operator and the convolution kernel of the second convolution operator, determine the remainder between any channel sequence number and the grouping parameters;

[0130] Based on the grouping parameters and the total number of channels corresponding to the convolution kernel of the second convolution operator, determine the multiple between the grouping parameters and the total number of channels;

[0131] Based on the remainder between any channel sequence number and the grouping parameter, the multiple between the grouping parameter and the total number of channels, and any channel sequence number and the grouping parameter, determine one of the multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator, and there is a correspondence between any channel sequence number and a channel sequence number.

[0132] In one embodiment, the second processing module 702 is specifically used for:

[0133] The image to be processed is input into the fused neural network model to perform image classification and determine the image classification result.

[0134] Applying the embodiments of this application has at least the following beneficial effects:

[0135] The process involves acquiring an image to be processed; inputting the image into a fused neural network model for online inference to determine the image processing result; wherein the fused neural network model is obtained by fusing the original neural network model, and during the fusion process, at least one memory-intensive operator in the original neural network model is replaced by at least one computationally intensive operator. Both the memory-intensive and computationally intensive operators are used to compute the feature maps of the image. Since memory read / write operations corresponding to memory-intensive operators can be time-consuming, the memory-intensive operators in the original neural network model are replaced with computationally intensive operators to obtain the fused neural network model. Using the fused neural network model for online inference reduces the time consumed by memory read / write operations and improves the speed of online inference.

[0136] This application also provides an electronic device, the structural schematic diagram of which is shown below. Figure 8 As shown, Figure 8 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0137] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0138] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0139] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0140] The memory 4003 stores computer programs that execute embodiments of this application, and its execution is controlled by the processor 4001. The processor 4001 executes the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0141] Electronic devices include, but are not limited to, servers.

[0142] Applying the embodiments of this application has at least the following beneficial effects:

[0143] The process involves acquiring an image to be processed; inputting the image into a fused neural network model for online inference to determine the image processing result; wherein the fused neural network model is obtained by fusing the original neural network model, and during the fusion process, at least one memory-intensive operator in the original neural network model is replaced by at least one computationally intensive operator. Both the memory-intensive and computationally intensive operators are used to compute the feature maps of the image. Since memory read / write operations corresponding to memory-intensive operators can be time-consuming, the memory-intensive operators in the original neural network model are replaced with computationally intensive operators to obtain the fused neural network model. Using the fused neural network model for online inference reduces the time consumed by memory read / write operations and improves the speed of online inference.

[0144] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the steps and corresponding content of the aforementioned method embodiments.

[0145] This application also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0146] Based on the same principles as the methods provided in the embodiments of this application, the embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in any of the optional embodiments of this application described above.

[0147] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0148] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: Obtain the image to be processed; The image to be processed is input into the fused neural network model for online inference processing to determine the image processing result. The fused neural network model is obtained by fusing the neural network model to be processed. During the fusion process, at least one memory-intensive operator of the neural network model to be processed is replaced by at least one computationally intensive operator. The at least one memory-intensive operator and the at least one computationally intensive operator are used to calculate the feature map of the image. The at least one memory-intensive operator includes a channel rearrangement operator, and the at least one computation-intensive operator includes a first convolution operator; Before inputting the image to be processed into the fused neural network model for online inference processing and determining the image processing result, the process further includes: Based on the channel rearrangement operator and the second convolution operator of the neural network model to be processed, a fusion process is performed, replacing the channel rearrangement operator and the second convolution operator with the first convolution operator to obtain the fused neural network model. The fused neural network model includes the first convolution operator, and at least two of the multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator are different from the channel sequence numbers corresponding to the convolution kernel of the second convolution operator. The process of fusing the channel rearrangement operator and the second convolution operator of the neural network model to be processed, replacing the channel rearrangement operator and the second convolution operator with the first convolution operator, to obtain the fused neural network model includes: Based on the grouping parameters of the channel rearrangement operator, the channel sequence number corresponding to the convolution kernel of the second convolution operator, and the total number of channels corresponding to the convolution kernel of the second convolution operator, the convolution kernel of the second convolution operator is rearranged to determine multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator. Wherein, the grouping parameter of the channel rearrangement operator is used to characterize the number of groups in which the channel rearrangement operator performs channel rearrangement operation, the channel sequence number corresponding to the convolution kernel of the second convolution operator is used to characterize the order of each channel in the convolution kernel of the second convolution operator, and the multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator are used to characterize the order of each channel in the convolution kernel of the first convolution operator. Based on the multiple channel sequence numbers, the channel rearrangement operator and the second convolution operator are replaced with the first convolution operator to obtain the fused neural network model.

2. The method according to claim 1, characterized in that, Before inputting the image to be processed into the fused neural network model for online inference processing to determine the image processing result, the method further includes: Based on the channel rearrangement operator of the neural network model to be processed, a fusion process is performed, replacing the channel rearrangement operator with the first convolution operator to obtain the fused neural network model, which includes the first convolution operator.

3. The method according to claim 1, characterized in that, The method involves rearranging the channels of the convolution kernel of the second convolution operator based on the grouping parameters of the channel rearrangement operator, the channel sequence number corresponding to the convolution kernel of the second convolution operator, and the total number of channels corresponding to the convolution kernel of the second convolution operator, to determine multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator, including: Based on the grouping parameters of the channel rearrangement operator and any channel sequence number corresponding to the convolution kernel of the second convolution operator, determine the remainder between the any channel sequence number and the grouping parameters; Based on the grouping parameters and the total number of channels corresponding to the convolution kernel of the second convolution operator, determine the multiple between the grouping parameters and the total number of channels; Based on the remainder between any channel sequence number and the grouping parameter, the multiple between the grouping parameter and the total number of channels, and any channel sequence number and the grouping parameter, one channel sequence number is determined from among the multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator, and any channel sequence number and the one channel sequence number correspond to each other.

4. The method according to claim 1, characterized in that, The step of inputting the image to be processed into the fused neural network model for online inference processing to determine the image processing result includes: The image to be processed is input into the fused neural network model to perform image classification and determine the image classification result.

5. A data processing apparatus, characterized in that, include: The first processing module is used to acquire the image to be processed; The second processing module is used to input the image to be processed into the fused neural network model, perform online inference processing, and determine the image processing result. The fused neural network model is obtained by fusing the neural network model to be processed. During the fusion process, at least one memory-intensive operator of the neural network model to be processed is replaced by at least one computationally intensive operator. The at least one memory-intensive operator and the at least one computationally intensive operator are used to calculate the feature map of the image. The at least one memory-intensive operator includes a channel rearrangement operator, and the at least one computation-intensive operator includes a first convolution operator; The second processing module is further configured to: Based on the channel rearrangement operator and the second convolution operator of the neural network model to be processed, a fusion process is performed, replacing the channel rearrangement operator and the second convolution operator with the first convolution operator to obtain the fused neural network model. The fused neural network model includes the first convolution operator, and at least two of the multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator are different from the channel sequence numbers corresponding to the convolution kernel of the second convolution operator. The second processing module is specifically used for: Based on the grouping parameters of the channel rearrangement operator, the channel sequence number corresponding to the convolution kernel of the second convolution operator, and the total number of channels corresponding to the convolution kernel of the second convolution operator, the convolution kernel of the second convolution operator is rearranged to determine multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator. Wherein, the grouping parameter of the channel rearrangement operator is used to characterize the number of groups in which the channel rearrangement operator performs channel rearrangement operation, the channel sequence number corresponding to the convolution kernel of the second convolution operator is used to characterize the order of each channel in the convolution kernel of the second convolution operator, and the multiple channel sequence numbers corresponding to the convolution kernel of the first convolution operator are used to characterize the order of each channel in the convolution kernel of the first convolution operator. Based on the multiple channel sequence numbers, the channel rearrangement operator and the second convolution operator are replaced with the first convolution operator to obtain the fused neural network model.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-4.

Images