Image data processing method and device, electronic device, and computer program

By interleaving rearrangement of image data to match convolution operations, the method optimizes data arrangement, reducing cache misses and data copy overhead, thereby enhancing processing efficiency in image data processing.

JP7582593B2Active Publication Date: 2024-11-13TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023524148
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-04-26
Filing Date
2022-04-12
Publication Date
2024-11-13
Estimated Expiration
2042-04-12

AI Technical Summary

Technical Problem

Conventional image data processing methods using SIMD face inefficiencies due to cache misses and additional data copy overhead when dealing with large input spaces and small convolution kernels, leading to reduced processing efficiency.

Method used

The method involves interleaving rearrangement of image data to match convolution operations, reducing dimensions and optimizing data arrangement to minimize cache misses and data copy overhead.

Benefits of technology

This approach enhances machine calculation performance and improves image data processing efficiency by reducing cache misses and additional data copy overhead, thus addressing the inefficiencies of traditional methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007582593000001
    Figure 0007582593000001
  • Figure 0007582593000002
    Figure 0007582593000002
  • Figure 0007582593000003
    Figure 0007582593000003
Patent Text Reader

Abstract

This application discloses an image data processing method and device, a storage medium, and an electronic device, which includes the steps of: obtaining a first image data set to be processed, in which the image data in the first image data set is arranged according to a first data format; performing interleaving rearrangement on the data in the first image data set to obtain a second image data set, in which the image data in the second image data set is arranged according to a second data format, the interleaving rearrangement manner is consistent with a convolution operation, and the dimension of the second data format is smaller than that of the first data format; and performing a convolution operation on the second image data set and a pre-acquired second weight data set to obtain a target output result.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This application claims priority from a Chinese patent application bearing application number 202110451609.0 and entitled "Method and apparatus for processing image data, storage medium and electronic device" filed with the China Patent Office on April 26, 2021, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the field of computer technology, and in particular to a method and device for processing image data, as well as an electronic device and a computer program. [Background technology]

[0003] In the current related technology, the conventional calculation mode is that each output is a set of threads, and image data is generally processed using SIMD (Single Instruction Multiple Data). For example, in the process of performing convolution calculation using SIMD, if the input dimension is [N, C, H, W] = [1, 1, 5, 5] and the convolution kernel dimension is [Cout, Cin, kernel_h, kernel_w] = [1, 1, 3, 3], the convolution calculation is performed to finally generate an output with a dimension of [N, C, H, W] = [1, 1, 5, 5], where N, C, H, and W respectively represent the batch, the number of channels, the height, and the width, Cout represents the number of output channels of the convolution kernel, Cin represents the number of input channels of the convolution kernel, kernel_h represents the height of the convolution kernel, and kernel_w represents the width of the convolution kernel.

[0004] In the conventional technical solutions, the data array generally only covers the dimensions [N, C, H, W]. Compared with the conventional data representation method, in the convolution operation, according to the characteristics of computer memory layout, the size of the convolution kernel is relatively small. However, when the size of the input space is relatively large, in order to obtain the completeness of the information, it is necessary to perform operations such as edge complementing on the input. In addition, since data is obtained across channels, cache misses and additional data copy overheads may occur, which may seriously affect the execution performance of the machine in the calculation process and reduce the processing efficiency of image data.

[0005] No effective solution to the above problems has yet been proposed. Summary of the Invention [Problem to be solved by the invention]

[0006] The embodiments of the present application aim to provide an image data processing method and device, as well as an electronic device and a computer program, for solving at least the technical problem of the relatively low efficiency of image data processing in the related art. The objective is to provide the following. [Means for solving the problem]

[0007] According to one aspect of an embodiment of the present application, there is provided a method for processing image data executed by an electronic device, the method comprising: acquiring a first image data set to be processed, image data in the first image data set being arranged according to a first data format; performing interleave rearrangement on the data in the first image data set to obtain a second image data set, the image data in the second image data set being arranged according to a second data format, and the manner of the interleave rearrangement being consistent with a convolution operation; ("The above interleaving rearrangement method corresponds to a convolution operation." The same applies below.)the dimensions of the second data format are smaller than the dimensions of the first data format; and Performing the convolution operation on the second image data set and a second pre-acquired weighted data set to obtain a desired output result.

[0008] According to another aspect of an embodiment of the present application, there is further provided an apparatus for processing image data, comprising: an acquisition module for acquiring a first image data set to be processed, the image data in the first image data set being arranged according to a first data format; a processing module for performing interleaving rearrangement on the data in the first image data set to obtain a second image data set, the image data in the second image data set being arranged according to a second data format, the manner of the interleaving rearrangement matching a convolution operation, and a dimension of the second data format being smaller than a dimension of the first data format; and An execution module is included for performing the convolution operation on the second image data set and the pre-acquired second weighted data set to obtain a desired output result.

[0009] According to yet another aspect of an embodiment of the present application, there is further provided a computer-readable storage medium having a computer program stored therein, the computer program being configured, when executed, to realize the above-mentioned image data processing method.

[0010] According to yet another aspect of an embodiment of the present application, there is further provided an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to realize the above-mentioned image data processing method.

[0011] According to yet another aspect of an embodiment of the present application, a computer program product is provided, which includes computer instructions which, when read and executed by a processor of a computing device, cause the computing device to perform the image data processing method described above. Effect of the Invention

[0012] In the embodiments of the present application, deep optimization is realized for the calculation process in the image data processing process, and corresponding data rearrangement is performed for the input image data, output image data and calculation weights in the calculation process. Compared with the traditional calculation mode, this reduces the additional data copy overhead and the probability of cache miss occurrence, and achieves the technical effects of optimizing the machine's calculation performance and improving the image data processing efficiency, thereby solving the technical problem that the image data processing efficiency in the related art is relatively low. [Brief description of the drawings]

[0013] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application, and the exemplary embodiments and the description thereof are used to interpret the present application, and are not intended to limit the present application. [Figure 1] FIG. 1 is a diagram showing an application environment of an image data processing method in an embodiment of the present application. [Diagram 2] 1 is a flowchart of a method for processing image data in an embodiment of the present application. [Diagram 3] FIG. 2 is a diagram illustrating a method for processing image data in an embodiment of the present application. [Figure 4] FIG. 11 is a diagram showing another method for processing image data in an embodiment of the present application. [Diagram 5] FIG. 11 is a diagram showing another method for processing image data in an embodiment of the present application. [Figure 6] FIG. 11 is a diagram showing another method for processing image data in an embodiment of the present application. [Figure 7]FIG. 11 is a diagram showing another method for processing image data in an embodiment of the present application. [Figure 8] FIG. 11 is a diagram showing another method for processing image data in an embodiment of the present application. [Figure 9] FIG. 11 is a diagram showing another method for processing image data in an embodiment of the present application. [Figure 10] FIG. 11 is a diagram showing another method for processing image data in an embodiment of the present application. [Figure 11] FIG. 11 is a diagram showing another method for processing image data in an embodiment of the present application. [Figure 12] 1 is a configuration diagram of an image data processing device according to an embodiment of the present application. [Figure 13] FIG. 1 is a configuration diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] In order to make those skilled in the art better understand the technical solution of the present application, the technical solution in the embodiments of the present application will be described below clearly and completely in conjunction with the drawings in the embodiments of the present application. It is obvious that the described embodiments are only some of the embodiments of the present application, but not all of the embodiments. In addition, all other embodiments obtained by those skilled in the art without creative labor based on the embodiments in the present application also belong to the scope of protection of the present application.

[0015] It should be noted that the terms "first", "second", and the like in the present specification, claims, and drawings are used to distinguish between similar objects and are not intended to limit the specific order or order of precedence or follow. It should be understood that the data so used may be exchanged, where appropriate, to cause the embodiments of the present invention described herein to be performed according to an order other than that illustrated or described herein. Also, the terms "comprise", "have" and any variations thereof are intended to cover a non-exclusive inclusive, e.g., a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units explicitly listed, but may also include other steps or units that are not explicitly listed or that are inherent to the explicitly listed process, method, product, or apparatus.

[0016] First, some nouns or terms that appear in the process of describing the embodiments of the present application will be explained as follows: CNN: Convolutional Neural Networks; SIMD: Single Instruction Multiple Data; Metal: an abstract framework for accessing graphics processing units (GPUs); Buffer: Metal Buffer; Texture: Metal Texture; [N,C,H,W]: Data dimension representation method, [batch,channel,height,width] respectively represent [batch, number of channels, height, width]; Cout: number of output channels of the convolution kernel; Cin: number of input channels for the convolution kernel; kernel_h: height of the convolution kernel; kernel_w: The width of the convolution kernel.

[0017] The technical solution of the present application will be described below in conjunction with examples.

[0018] According to one aspect of an embodiment of the present application, a method for processing image data is provided. In this embodiment, the above-mentioned method for processing image data can be applied to a hardware environment consisting of a server 101 and a user terminal 103 as shown in FIG. 1. As shown in FIG. 1, the server 101 is connected to the terminal 103 through a network and is used to provide services to the user terminal or a client installed in the user terminal, and the client may be a video client, an instant communication client, a browser client, an education client, a game client, etc. A database 105 may be arranged in the server or independently of the server, which is used to provide the server 101 with a data storage service, for example, an image data storage service. The above-mentioned network may include, but is not limited to, a wired network and a wireless network, among which the wired network includes a local area network, a metropolitan area network, and a wide area network, and the wireless network includes Bluetooth, WIFI, and other networks that can realize wireless communication. The user terminal 103 may be a terminal in which an application program (also called application) 107 is located, and may include at least one of the following, but is not limited to, computer devices such as mobile phones (e.g., Android smartphones, iOS smartphones, etc.), notebook computers, tablet computers, handheld computers, MIDs (Mobile Internet Devices), PADs, desktop computers, and smart TVs. The above-mentioned server may be a single server, a server group consisting of multiple servers, or a cloud server. The application 107 using the above-mentioned image data processing method is displayed by the user terminal 103.

[0019] As shown in FIG. 1, the above-mentioned image data processing method can be realized in the user terminal 103 by the following steps.

[0020] S1: Obtain a first image data set waiting to be processed by an application program 107 of a user terminal 103, in which image data in the first image data set is arranged according to a first data format; S2: performing interleaving rearrangement on the data in the first image data set in the application program 107 of the user terminal 103 to obtain a second image data set, in which the image data in the second image data set is arranged according to a second data format, the manner of interleaving rearrangement is matched with a convolution operation, and the dimension (number of dimensions) of the second data format is smaller than that of the first data format; and S3: The application 107 of the user terminal 103 performs a convolution operation on the second image data set and the pre-acquired second weight data set to obtain a target output result.

[0021] In this embodiment, the above-mentioned image data processing method may be used by a client located on a server, but is not limited thereto.

[0022] In this embodiment, the above-mentioned image data processing method may be used asynchronously by a client configured in the user terminal 103 and the server 101, for example, the above-mentioned steps S1 and S2 may be executed by the application program 107 of the user terminal 103, and the above-mentioned step S3 may be executed by a client arranged in the server 101. Note that this is only an example, and the present embodiment is not limited to this.

[0023] As an embodiment, as shown in FIG. 2, the above-mentioned image data processing method includes the following steps.

[0024] S202: Obtaining a first image data set to be processed, in which image data in the first image data set is arranged according to a first data format; S204: performing interleaving rearrangement on the data in the first image data set to obtain a second image data set, in which the image data in the second image data set is arranged according to a second data format, the manner of interleaving rearrangement is matched with a convolution operation, and the dimension of the second data format is smaller than that of the first data format; and S206: Perform a convolution operation on the second image data set and the pre-acquired second weighted data set to obtain a target output result.

[0025] In this embodiment, the image data in the above-mentioned first data format may be arranged according to a data format of N1×C1×H1×W1 (hereinafter, N1, C1, H1, and W1 may be written as N1, C1, H1, and W1. Other similar descriptions are also applicable.), but are not limited to this, where N1 indicates the number of image data subsets contained in the first image data set, C1 indicates the number of channels in each image data subset, H1 indicates the data height in each image data subset of the first image data set, and W1 indicates the data width in each image data subset of the first image data set.

[0026] For example, Fig. 3 is a diagram showing another image data processing method in an embodiment of the present application. The image data in the above-mentioned first image data set is arranged according to the data format of N1 x C1 x H1 x W1, which may include, but is not limited to, the example shown in Fig. 3, N1 indicates the number of image data subsets contained in the first image data set, N1 = 2 in Fig. 3, in other words, the image data processing process shown in Fig. 3 includes two batches of data to be processed, of which one batch corresponds to one image data subset, C1 indicates the number of channels in each image data subset, C1 = 5 in Fig. 3, which means that the batch of data to be processed includes five channels of data to be processed, H1 indicates the data height in each image data subset, H1 = h in Fig. 3, W1 indicates the data width in each image data subset, W1 = w in Fig. 3, and "A, B, C" indicate image data in different channels.

[0027] In this embodiment, the application scenarios of the above-mentioned image data processing method may include, but are not limited to, various application scenarios that require image data processing, such as medical care, finance, credit investigation, banking, games, energy, education, buildings, games, traffic, IoT, industry, artificial intelligence, etc., and these application scenarios may be applied to, but are not limited to, the neural network forward calculation library. Since the neural network forward calculation library provides the calculation capabilities of all neural network algorithms, the application scenarios of this application cover all application scenarios that use the neural network forward calculation library, and may include, but are not limited to, AI algorithm-related applications related to cloud technology, such as virtual backgrounds, etc.

[0028] Cloud technology refers to a hosting technology that integrates a set of resources, such as hardware, software, and networks, over a wide area network or a local area network to realize data calculation, storage, processing, and sharing.

[0029] Among them, cloud conferencing is an efficient, convenient, and low-cost conferencing format based on cloud computing technology. Users can quickly and efficiently synchronously share voice, data files, and videos with teams and customers around the world by simply performing simple and easy-to-use operations via the Internet interface, and complex technologies such as data transmission and processing during the conference can be handled by the cloud conferencing service provider.

[0030] At present, cloud conferencing mainly focuses on the service content based on SaaS (Software as a Service) mode, such as telephone, network, video and other service formats. Video conferencing based on cloud computing is called cloud conferencing.

[0031] In the era of cloud conferencing, data transmission, processing, and storage are all handled by the computer resources of the video conferencing manufacturer, and users do not need to purchase expensive hardware or install cumbersome software. They simply need to open a browser and log in to the corresponding interface to hold an efficient remote conference.

[0032] Cloud conference systems support dynamic cluster configuration of multiple servers, and provide multiple high-performance servers, greatly improving the stability, security and availability of conferences. In recent years, video conferencing has been welcomed by many users because it can greatly improve communication efficiency, continuously reduce communication costs, and bring about an upgrade in internal management, and has been widely applied in various fields such as transportation, shipping, finance, operators, education, and enterprises. Undoubtedly, after using cloud computing, video conferencing becomes more attractive in terms of convenience, speed, ease of use, etc., and can bring about a new boom in the use of video conferencing.

[0033] For example, take the above-mentioned image data processing method as applied to a cloud conference scenario, and FIG 4 is a diagram showing an image data processing method in an embodiment of the present application. As shown in FIG 4, the method specifically includes, but is not limited to, the following steps:

[0034] S1: The user terminal 402 acquires a first image data set waiting to be processed; S2: A processor 404 within the user terminal 402 or connected to the user terminal 402 performs interleaving and rearrangement on the data in the first image data set to obtain a second image data set; and S3: Perform a convolution operation on the second image data set and the pre-acquired second weighted data set to obtain a target output result.

[0035] Among them, the above-mentioned first image data set may include, but is not limited to, a first image data set awaiting processing stored in the database shown in FIG. 4, and the above-mentioned target output result may include, but is not limited to, a virtual background for display in the virtual background display 408 of the cloud conferencing application 406 shown in FIG. 4, or other image data obtained after processing using the above-mentioned image data processing method.

[0036] It should be noted that the above are merely examples, and the present embodiment is not limited thereto.

[0037] In this embodiment, performing a convolution operation on the above-mentioned second image data set and the pre-acquired second weighted data set to obtain a target output result may include, but is not limited to, performing a convolution operation on the second image data set and the second weighted data set to obtain a third image data set, in which the convolution operation includes a convolution operation, and the target output result includes, but is not limited to, the third image data set, the second image data set is an image data set obtained by performing interleaving rearrangement on image data of M1 channels in each set of image data among the S1 sets of image data, and the S1 set of image data is image data obtained by making the image data of each of the M1 channels in the first image data set belong to one set, and M1≦C1.

[0038] In this embodiment, a first image data set is obtained to be processed, in which the image data in the first image data set is arranged according to a data format of N1×C1×H1×W1, and the data in the first image data set is interleaved and rearranged to obtain a second image data set, in which the image data in the second image data set is arranged according to a data format of N1×H2×W2, and the manner of interleaving and rearrangement is matched with the convolution operation. A deep optimization is realized for the calculation process in the image data processing process, and corresponding data rearrangement is performed for the input image data, the output image data and the calculation weight in the calculation process, so that relatively high-dimensional data is rearranged into relatively low-dimensional data, and when processing the data of different channels in the subsequent convolution calculation process, the data of multiple channels can be grouped, so that data can be extracted from different channels in a cross-group, and the number of times of cross-channel data extraction can be effectively reduced. In contrast, in the prior art, cross-channel extraction needs to be performed each time data is extracted from different channels. Compared with the conventional calculation methods, the technical solution described in this application can reduce additional data copy overhead and the probability of cache miss occurrence, and achieve the technical effects of optimizing the calculation performance of the machine and improving the processing efficiency of image data, thereby solving the technical problem of relatively low image data processing efficiency in related technologies.

[0039] As one embodiment, the image data in the first image data set being arranged according to a first data format includes the image data in the first image data set being arranged according to a data format of N1×C1×H1×W1, where N1 indicates the number of image data subsets included in the first image data set, C1 indicates the number of channels in each image data subset, H1 indicates the data height in each image data subset of the first image data set, and W1 indicates the data width in each image data subset of the first image data set.

[0040] The image data in the second image data set being arranged according to a second data format includes performing interleaving rearrangement on the data in the first image data set to obtain a second image data set, in which the image data in the second image data set is arranged according to a data format of N1×H2×W2, H2 represents the data height of each image data subset of the second image data set, and W2 represents the data width of each image data subset of the second image data set.

[0041] In this embodiment, performing interleaving rearrangement on the data in the first image data set to obtain the second image data set may include, but is not limited to, performing interleaving rearrangement on the image data in the first image data set, thereby reducing the dimensionality of the image data, which is advantageous for a subsequent convolution operation.

[0042] Fig. 5 is a diagram showing another image data processing method in an embodiment of the present application. The above-mentioned interleaved rearrangement of the data in the first image data set to obtain the second image data set may include, but is not limited to, the image data of each of M1 channels in the first image data set belong to one set, where M1≦C1, and the image data of M1 channels in each set are interleaved rearranged to obtain the second image data set, for example, when M1=4, as shown in Fig. 5, each image data is divided into C1 / 4 sets, each set being 4 channels, and the 4-channel dimension image data in each set is rearranged in an interleaved (mixed) manner to obtain the second image data set with the data structure [N,H,C / 4,W,C4] shown in Fig. 5, where "A,B,C" represent image data in different channels.

[0043] For example, the following description will be given taking the example shown in FIG.

[0044] The data of A(1,1) in the upper left corner of the first image data set is located in the first row, first column in the first channel, and the data of B(1,1) is located in the first row, first column in the second channel. When performing a convolution operation, if it is necessary to extract data of B(1,1) after extracting data of A(1,1), the mode of extracting data of A channel needs to be switched to the mode of extracting data of B channel in the prior art, in other words, data needs to be extracted in cross-channel. On the other hand, when the present embodiment is adopted, the data of A(1,1) is arranged in the first row, first column in the second image data set, and the data of B(1,1) is arranged in the first row, second column in the second image data set, so that the data of B(1,1) can be directly extracted after the data of A(1,1), and problems such as cache miss caused by extracting data in cross-channel can be avoided.

[0045] In one embodiment, performing interleaving reordering on data in the first image data set to obtain a second image data set includes: The image data of each of M1 channels in the first image data set is grouped into one set to obtain S1 sets of image data, where M1≦C1; and performing interleaving rearrangement on the M1 channels of image data in each set of image data among the S1 sets of image data to obtain a second image data set.

[0046] In this embodiment, obtaining S1 sets of image data by grouping the image data of each of the M1 channels in the above-mentioned first image data set into one set includes: when C1 is an integer multiple of M1, obtaining S1 sets of image data by grouping the image data of each of the M1 channels in the first image data set into one set, where S1=(C1 / M1)×H1×W1; when C1 is not an integer multiple of M1, increasing the number of channels in the first image data set from C1 to C2 to obtain a third image data set, where C2 is an integer multiple of M1 and the image data in the channels increased in the third image data set is 0 (i.e., making the number of channels an integer multiple of M1); and obtaining S1 sets of image data by grouping the image data of each of the M1 channels in the third image data set into one set, where S1=(C2 / M1)×H1×W1.

[0047] Taking N=1 as an example, FIG. 6 is a diagram illustrating another image data processing method in an embodiment of the present application. The above-mentioned second image data set may include, but is not limited to, the example shown in FIG. 6, in which "A, B, C" indicate image data in different channels, and A(1,1), B(1,1), C(1,1), D(1,1) are interleaved and rearranged to form image data of the same height, and the continuous rearrangement of data of different channels in the image data can improve the local performance of data access, so that the probability of cache miss can be greatly reduced. In the prior art, when the size of the convolution kernel is not 1x1 size, some additional areas need to be set to 0 for the feature extraction of the boundary (edge) area when the feature extraction is performed on the input picture, so that the feature extraction of the boundary (edge) area can be realized. On the other hand, in this embodiment, M1 channels are packaged, so that when the convolution operation is performed using SIMD, there is no need to perform an edge compensation operation on the boundary (edge) area, so that the additional overhead of data copying can be saved.

[0048] Taking FIG. 5 as an example, when the size of the convolution kernel is 3x3, when extracting feature data with B(2,1) as the center point, in the prior art, it is necessary to add three feature points with a value of 0 to the left of B(2,1) in the first image data set to realize feature extraction of B(2,1) data. In contrast, in this embodiment, after rearranging the data, the data to the left of B(2,1) in the second image data set is A(2,1) data, so there is no need to perform edge compensation operation, and feature extraction can be performed directly based on B(2,1). In other words, in the embodiment of the present application, interleaving rearrangement is performed on the data to be processed, so that the number of edge compensation operations in the process of extracting boundary (edge) data using the convolution kernel can be reduced, and the data in different channels can be made to have the same dimension, thereby achieving the effect of saving the additional overhead of data copying.

[0049] In this embodiment, the image data of each of the M1 channels in the first image data set is grouped into one set to obtain S1 sets of image data, where M1≦C1, and the image data of the M1 channels in each set of image data among the S1 sets of image data is interleaved and rearranged to obtain a second image data set. By performing interleaved rearrangement on the data in the first image data set to obtain the second image data set, compared with the conventional calculation mode which is prone to cache misses and additional data copy overhead, this embodiment can reduce the occurrence probability of cache misses, achieve the technical effects of optimizing the calculation performance of the machine and improving the processing efficiency of image data, and solve the technical problem of relatively low image data processing efficiency in the related art.

[0050] In one embodiment, the step of obtaining the S1 set of image data by grouping the image data of each of the M1 channels in the first image data set into one set includes the steps of: When C1 is an integer multiple of M1, the image data of each of the M1 channels in the first image data set is grouped into one set to obtain S1 sets of image data, where S1=(C1 / M1)×H1×W1; and If C1 is not an integer multiple of M1, the number of channels in the first image data set is increased from C1 to C2 to obtain a third image data set, where C2 is an integer multiple of M1 and the image data in the increased channels in the third image data set is 0; and the image data of each of the M1 channels in the third image data set belong to one set to obtain S1 sets of image data, where S1=(C2 / M1)×H1×W1.

[0051] In this embodiment, if the above-mentioned C1 is not an integer multiple of M1, increasing the number of channels in the first image data set from C1 to C2 includes, but is not limited to, rounding up C2=(C1 / M1)×M1, and of course, rounding down or taking an integer in other ways may also be used.

[0052] Taking N=1, C1=5, and M1=4 as an example, if the number of channels C1 is not divisible by 4, the number of channels is made to be an integer multiple of 4, and all the additional active values ​​(corresponding to the aforementioned image data) are filled with 0, so that the second column to the fourth column of the second row are all “0.0f”, as shown in FIG. 5.

[0053] It should be noted that the above are merely examples, and the present embodiment is not limited thereto.

[0054] In one embodiment, performing interleaving rearrangement on image data of M1 channels in each set of image data among the S1 sets of image data to obtain a second image data set includes: When C1 is an integer multiple of M1, perform interleaving rearrangement on the M1 channels of image data in each set of image data among the S1 sets of image data to obtain a second set of image data, where W2=M1×W1 and H2=(C1 / M1)×H1; and If C1 is not an integer multiple of M1, the image data of M1 channels in each set of image data among the S1 sets of image data are interleaved and rearranged to obtain a second image data set, in which W2=M1×W1 and H2=(C2 / M1)×H1.

[0055] In this embodiment, as shown in FIG. 5, N=1, C1=5, and M1=4 are taken as an example. In this case, the above-mentioned W2=M1×W1 is W2=w*C4 shown in FIG. 5, and H2=(C2 / M1)×H1 is H2=H*C / 4 shown in FIG. 5.

[0056] According to this embodiment, after rearranging the data structure to N1×H2×W2, the convolution operation can be accelerated using SIMD even without the need to complement edges in the data, thereby avoiding the additional overhead of copying data due to complementing edges during the convolution operation.

[0057] In one embodiment, the method further comprises: Obtaining a first weight data set, in which the weight data in the first weight data set is arranged according to a data format of N2×C2×H3×W3, where N2 represents the number of weight data subsets included in the first weight data set, C2 represents the number of channels in each weight data subset, H3 represents the data height in each weight data subset, and W3 represents the data width in each weight data subset; and performing interleaving rearrangement on the data in the first weight data set to obtain a second weight data set, in which the weight data in the second weight data set is arranged according to a data format of H4×W4, H4 indicates a data height of the weight data in the second weight data set, and W4 indicates a data width of the weight data in the second weight data set.

[0058] In this embodiment, the above-mentioned first weight data set may include, but is not limited to, weight data adopted when using a convolution kernel to process image data in a convolution calculation process, for example, the above-mentioned image data processing method is applied to a cloud conference scenario, and Fig. 7 is a diagram showing another image data processing method in an embodiment of the present application. As shown in Fig. 7, the method specifically includes, but is not limited to, the following steps:

[0059] S1: A processor 704 in or connected to a user terminal 702 obtains a first weight data set; and S2: A processor 704 within or connected to the user terminal 702 performs interleaving rearrangement on the data in the first weighted data set to obtain a second weighted data set.

[0060] Among them, the above-mentioned first weighting data set may be stored in a database as shown in FIG. 7, and the above-mentioned second weighting data set may be associated with and used to generate a virtual background in the virtual background display area 708 of the cloud conferencing application 706 as shown in FIG. 7 by being processed, but is not limited thereto.

[0061] 8 is a diagram showing another image data processing method in an embodiment of the present application. In this embodiment, the image data in the above-mentioned first image data set is arranged according to a data format of N2×C2×H3×W3, which may include, but is not limited to, the example shown in FIG. 8, where N2 represents the number of weight data subsets included in the first weight data set, and in FIG. 8, N2=3, that is, N2=C out where C2 denotes the number of channels in each weight data subset. In FIG. 8, C2=5, i.e., C2=C in 8, H3 denotes the data height in the weight data subset, H3=4 in FIG. 8, i.e., H3=kernel_h; W3 denotes the data width in each weight data subset, W3=3 in FIG. 8, i.e., W3=kernel_w; and “A, B, C” denote image data in different channels.

[0062] Note that the above are merely examples, and the present embodiment is not limited thereto.

[0063] In this embodiment, the data in the first weighted data set is interleaved and rearranged to obtain the second weighted data set. Compared to the conventional calculation mode, which is prone to cache misses and additional data copy overhead, this embodiment reduces the probability of cache misses occurring, and achieves the technical effects of optimizing the calculation performance of the machine and improving the processing efficiency of image data, thereby resolving the technical problem of relatively low image data processing efficiency in related technologies.

[0064] In one embodiment, the step of performing interleaving rearrangement on the data in the first weighted data set to obtain the second weighted data set includes: The weight data of each of the M2 weight data subsets in the first weight data set is grouped into one set to obtain S2 sets of weight data, where M2≦N2; and The method includes performing interleaving rearrangement on the M2 weight data in each set of weight data of the S2 sets of weight data to obtain a second set of weight data.

[0065] FIG. 9 is a diagram showing another method of processing image data in an embodiment of the present application. In this embodiment, the data in the first weight data set is interleaved and rearranged to obtain the second weight data set, which may include, but is not limited to, the example shown in FIG. 9. Take N2=1 as an example, and the weight data of each M2 weight data subset in the first weight data set is grouped into one set to obtain S2 sets of weight data. The M2 weight data in each set of the S2 sets of weight data is interleaved and rearranged to obtain the second weight data set. Take M2=4 as an example, and in this case, as shown in FIG. 9, the weight data is divided into C / 4 sets, each set being 4 output channels. If the number of output channels is not divisible by 4, the number of channels is made to be an integer multiple of 4, and all additional active values ​​are filled with 0. The 4-channel dimension weight data in each set is rearranged in an interleaved (mixed) manner. In addition, the input channel dimension is arranged in order for the subsequent dimensions, so that a data structure of [Cout / 4, kernel_h, kernel_w, Cin, Cout4] can be obtained, where OC4 is Cout4 and IC is Cin.

[0066] In this embodiment, obtaining S2 sets of weight data by associating the weight data of each of M2 weight data subsets in the above-mentioned first weight data set into one set includes: when N2 is an integer multiple of M2, the weight data of each of M2 weight data subsets in the first weight data set into one set to obtain S2 sets of weight data, where S2=M2×(N2 / M2)×C2×H3×W3; when N2 is not an integer multiple of M2, the number of weight data subsets in the first weight data set is increased from N2 to N3 to obtain a third weight data set, where N3 is an integer multiple of M2 and the weight data in the weight data subset increased to the third weight data set is 0; and, when N2 is an integer multiple of M2, the weight data of each of M2 weight data subsets in the third weight data set into one set to obtain S2 sets of weight data, where S2=M2×(N3 / M2)×C2×H3×W3.

[0067] In one embodiment, the weight data of each of the M weight data subsets in the first weight data set is grouped into one set to obtain the S weight data set, When N2 is an integer multiple of M2, the weight data of each of M2 weight data subsets in the first weight data set is grouped into one set to obtain S2 sets of weight data, where S2=M2×(N2 / M2)×C2×H3×W3; and If N2 is not an integer multiple of M2, increasing the number of weight data subsets in the first weight data set from N2 to N3 to obtain a third weight data set, where N3 is an integer multiple of M2, and the weight data in the weight data subsets increased to the third weight data set is 0; and the weight data of each of the M2 weight data subsets in the third weight data set belong to one set to obtain S2 sets of weight data, where S2=M2×(N3 / M2)×C2×H3×W3.

[0068] In this embodiment, if the above-mentioned N2 is not an integer multiple of M2, increasing the number of weight data subsets in the first weight data set from N2 to N3 to obtain a third weight data set may include rounding down N3=(N2 / M2)×M2, but of course, it may also be rounded up or taken as an integer in other ways.

[0069] Take N2=3 and M2=4 as an example. If N2 is not divisible by 4, make it a multiple of 4, and fill all additional active values ​​(corresponding to the aforementioned image data) with 0, so that the 4th column, 8th column, 12th column, etc. are all "0.0f", as shown in FIG. 9.

[0070] It should be noted that the above are merely examples, and the present embodiment is not limited thereto.

[0071] In this embodiment, when N2 is an integer multiple of M2, the weight data of each of M2 weight data subsets in the first weight data set is grouped into one set to obtain S2 sets of weight data, where S2=M2×(N2 / M2)×C2×H3×W3; when N2 is not an integer multiple of M2, the number of weight data subsets in the first weight data set is increased from N2 to N3 to obtain a third weight data set, where N3 is an integer multiple of M2, and the weight data of the weight data subset increased to the third weight data set is 0; and the weight data of each of M2 weight data subsets in the third weight data set is grouped into one set to obtain S2 sets of weight data, where S2=M2×(N3 / M2)×C2×H3×W3; in this way, the data in the first weight data set is interleaved and rearranged to obtain the second weight data set, thereby reducing the cache miss and additional data copy overhead that are likely to occur under the traditional calculation mode, and improving the cache The purpose of reducing the probability of miss occurrence is achieved, and the technical effects of optimizing the calculation performance of the machine and improving the efficiency of image data processing can be realized, thereby solving the technical problem that the image data processing efficiency in the related art is relatively low.

[0072] In one embodiment, performing interleaving rearrangement on M2 weight data in each set of weight data among the S2 sets of weight data to obtain a second set of weight data includes: When N2 is an integer multiple of M2, perform interleaving rearrangement on M2 weight data in each set of weight data among S2 sets of weight data to obtain a second set of weight data, where W4=M2×(N2 / M2)×C2, H4=H3×W3; and If N2 is not an integer multiple of M2, then perform interleaving rearrangement on M2 weight data in each set of weight data among S2 sets of weight data to obtain a second weight data set, in which W4=M2×(N3 / M2)×C2 and H4=H3×W3.

[0073] In this embodiment, M2=4 is taken as an example. In this case, as shown in FIG. 9, H3=kernel_h*kernel_w, W3=IC*OC4, and the height of each set of weight data is kernel_w and the width is OC4.

[0074] It should be noted that the above are merely examples, and the present embodiment is not limited thereto.

[0075] As one example, the value of N2 is the number of output channels of the convolution kernel, the value of C2 is the number of input channels of the convolution kernel, the convolution operation is a convolution operation performed using the convolution kernel, each weight data subset includes weight data for the C2 input channels, and each output channel includes the C2 input channels.

[0076] In this embodiment, each of the above-mentioned weight data subsets includes weight data for C2 input channels, and a convolution kernel is used to perform a convolution operation on the C2 second image data sets waiting to be processed based on the second weight data set to obtain a target output result.

[0077] In one embodiment, performing a convolution operation on the second image data set and the pre-acquired second weighted data set to obtain a target output result includes: A convolution operation is performed on the second image data set and the second weighted data set to obtain a fourth image data set, wherein the target output result includes a fourth image data set, the second image data set is an image data set obtained by performing interleaving rearrangement on image data of M1 channels in each set of image data among S1 sets of image data, the S1 set of image data is image data obtained by grouping the image data of each of the M1 channels in the first image data set into one set, and M1≦C1.

[0078] In this embodiment, performing the convolution operation on the above-mentioned second image data set and the second weighting data set includes, but is not limited to, obtaining C2 sets of image data from the second image data set, where each set of image data includes multiple image data in the same channel in the first image data set, and each set of image data is image data obtained by shifting the storage address of the previous set of image data in the C2 set of image data by one address, and performing the convolution operation on the C2 set of image data and N2×C2 sets of weighting data in the second weighting data set to obtain N2 sets of image data in the fourth image data set, where each set of weighting data has the same data structure as each set of image data.

[0079] In this embodiment, by performing interleaving rearrangement on the data in the first weighted data set and the data in the first image data set waiting to be processed, the probability of cache misses can be reduced compared to the case where cache misses and additional data copy overhead are likely to occur under conventional calculation modes, and the technical effects of optimizing the calculation performance of the machine and improving the processing efficiency of image data can be achieved, thereby solving the technical problem of relatively low image data processing efficiency in related technologies.

[0080] In one embodiment, performing a row convolution operation on the second image data set and the second weighted data set to obtain a third image data set includes: Obtaining C2 sets of image data from the second image data set, where each set of image data includes a plurality of image data in the same channel in the first image data set, and each set of image data is image data obtained by shifting the storage address of the previous set of image data in the C2 sets of image data by one address; and performing a convolution operation on the C2 sets of image data and the N2×C2 sets of weight data in the second weight data set to obtain N2 sets of image data in a third image data set, where each set of weight data has the same data structure as each set of image data; Among them, the second weight data set is obtained by performing interleaving rearrangement on the data in the first weight data set, and the weight data in the first weight data set is arranged according to a data format of N2×C2×H3×W3, where N2 indicates the number of weight data subsets included in the first weight data set, C2 indicates the number of channels in each weight data subset, H3 indicates the data height in each weight data subset, and W3 indicates the data width in each weight data subset.

[0081] In this embodiment, the fact that each set of image data is obtained by shifting one address from the storage address of the previous set of image data in the C2 set of image data includes processing the image data according to a sliding window of a predetermined step length, that is, the step length of shifting one address from the storage address of the previous set of image data in the C2 set of image data is 1.

[0082] In this embodiment, the above-mentioned sets of weight data have the same data structure as the above-mentioned sets of image data, including, but not limited to, the above-mentioned M1 and M2 being the same.

[0083] In this embodiment, the image data is obtained by shifting one address from the storage address of the previous image data of the C2 sets of image data, and each set of image data is obtained. By performing interleaving rearrangement on the data in the first weight data set and the data in the first image data set waiting to be processed, the frequency of obtaining data across channels in the convolution calculation process can be reduced. Compared with the case where cache misses and additional data copy overhead are easily caused under traditional calculation modes, the occurrence probability of cache misses can be reduced, and the technical effects of optimizing the calculation performance of the machine and improving the processing efficiency of image data can be achieved, so that the technical problem of relatively low image data processing efficiency in the related art can be solved.

[0084] As one embodiment, performing a convolution operation on the C2 set of image data and the N2×C2 sets of weighting data in the second weighting data set to obtain N2 sets of image data in the third image data set includes performing a weighted sum operation between each of the C2 sets of weighting data in the N2×C2 sets of weighting data and the C2 set of image data to obtain N2 sets of image data.

[0085] In this embodiment, the method includes, but is not limited to, performing a weighted sum operation between each C2 set of weight data in the N2×C2 sets of weight data and the C2 set of image data using a convolution kernel one by one according to a predetermined sliding step length to obtain N2 sets of image data.

[0086] Figure 10 is a diagram showing another image data processing method in an embodiment of the present application. As shown in Figure 10, take the convolution kernel size as an example, use the convolution kernel to perform weighted sum on data in the second image data set at the same position according to the corresponding weight parameters recorded in the second weight data set to obtain one image data set of N2 sets of image data, and then continue processing according to a sliding window with a step length of 1 to obtain the above-mentioned N2 sets of image data.

[0087] In one embodiment, the method further includes storing the first image data set and the second image data set in a first memory space; and storing the second weighting data set in a second memory space, wherein the first memory space and the second memory space are independent memory spaces.

[0088] In this embodiment, the above-mentioned first memory space may include a storage space for storing image data, such as a Texture resource, and the above-mentioned second memory space may include a storage space for storing weighting data, such as a Buffer resource, but is not limited to these.

[0089] FIG. 11 is a diagram showing another image data processing method in an embodiment of the present application. As shown in FIG. 11, in the conventional technical solution, for example, when performing GPU calculation using Metal, only one memory (Buffer / Texture) is usually used as a space for loading / storing data. However, currently, under the calculation mode in which the model design is becoming increasingly lighter, the access restriction of memory bandwidth often becomes the final performance bottleneck. In the embodiment of the present application, the Data Buffer resource and the Texture resource in Metal are independent memory spaces. Therefore, compared with the conventional method in which data is stored in only one memory configuration (Buffer / Texture), input / output uses Texture to store data, and weight / bias parameters are stored in Buffer. Using Texture and Buffer separately in this way can obtain a higher memory bandwidth and reduce the occurrence probability of cache miss, thereby improving memory access performance.

[0090] Although the above method embodiments are described as a series of operations for convenience of explanation, those skilled in the art will understand that the present application is not limited to the order of operations described, and some steps may be performed in other orders or simultaneously in the present application, and those skilled in the art will understand that such operations and modules are not necessarily essential to the present application.

[0091] According to another aspect of the embodiment of the present application, there is further provided an image data processing device for performing the above-mentioned image data processing method. As shown in FIG. 12, the device includes:

[0092] An acquisition module 1202: used to acquire a first image data set to be processed, where the image data in the first image data set is arranged according to a first data format; A processing module 1204 is used to perform interleaving rearrangement on the data in the first image data set to obtain a second image data set, in which the image data in the second image data set is arranged according to a second data format, the manner of interleaving rearrangement is matched with a convolution operation, and the dimension of the second data format is smaller than that of the first data format; and An execution module 1206 is used to perform a convolution operation on the second image data set and the pre-acquired second weighted data set to obtain a target output result.

[0093] In one embodiment, the acquisition module includes an acquisition unit, which is used to acquire a first image data set to be processed, in which the image data in the first image data set is arranged according to a data format of N1×C1×H1×W1, N1 indicates the number of image data subsets included in the first image data set, C1 indicates the number of channels in each image data subset, H1 indicates the data height in each image data subset of the first image data set, and W1 indicates the data width in each image data subset of the first image data set; and the processing module includes a processing unit, which is used to perform interleaving rearrangement on the data in the first image data set to acquire a second image data set, in which the image data in the second image data set is arranged according to a data format of N1×H2×W2, H2 indicates the data height in each image data subset of the second image data set, and W2 indicates the data width in each image data subset of the second image data set.

[0094] As one embodiment, the processing module includes: a grouping unit for grouping image data of each of M1 channels in the first image data set into one set to obtain S1 sets of image data (wherein M1≦C1); and an arrangement unit for performing interleaving rearrangement on the image data of the M1 channels in each set of image data among the S1 sets of image data to obtain a second image data set.

[0095] In one embodiment, the grouping unit is used to group the image data of each of the M1 channels in the first image data set into one set to obtain S1 set of image data in the following manner: if C1 is not an integer multiple of M1, increase the number of channels in the first image data set from C1 to C2 to obtain a third image data set, where C2 is an integer multiple of M1, and the image data in the increased channels in the third image data set is 0; and group the image data of each of the M1 channels in the third image data set into one set to obtain S1 set of image data, where S1=(C2 / M1)×H1×W1.

[0096] In one embodiment, the arrangement unit is used to interleave and rearrange the image data of the M1 channels in each set of image data of the S1 sets of image data to obtain a second image data set in the following manner: if C1 is an integer multiple of M1, then the image data of the M1 channels in each set of image data of the S1 sets of image data is interleaved and rearranged to obtain a second image data set, where W2=M1×W1 and H2=(C1 / M1)×H1; if C1 is not an integer multiple of M1, then the image data of the M1 channels in each set of image data of the S1 sets of image data is interleaved and rearranged to obtain a second image data set, where W2=M1×W1 and H2=(C2 / M1)×H1.

[0097] In one embodiment, the device is further configured to: obtain a predetermined first weight data set, in which the weight data in the first weight data set is arranged according to a data format of N2×C2×H3×W3, where N2 represents the number of weight data subsets included in the first weight data set, C2 represents the number of channels in each weight data subset, H3 represents the data height in each weight data subset, and W3 represents the data width in each weight data subset; and perform interleaving rearrangement on the data in the first weight data set to obtain a second weight data set, in which the weight data in the second weight data set is arranged according to a data format of H4×W4, where H4 represents the data height of the weight data in the second weight data set, and W4 represents the data width of the weight data in the second weight data set.

[0098] In one embodiment, the device is further used to perform interleaving rearrangement on the data in the first weight data set to obtain a second weight data set in the following manner: the weight data of each of M2 weight data subsets in the first weight data set belong to a set to obtain S2 sets of weight data, where M2≦N2; and the M2 weight data in each set of weight data of the S2 sets of weight data are interleaved rearranged to obtain the second weight data set.

[0099] In one embodiment, the device further comprises: acquiring an S2 set of weight data by grouping the weight data of each of M2 weight data subsets in the first weight data set into one set, where S2=M2×(N2 / M2)×C2×H3×W3 when N2 is an integer multiple of M2; acquiring a third weight data set by increasing the number of weight data subsets in the first weight data set from N2 to N3 when N3 is an integer multiple of M2, where N3 is an integer multiple of M2, and the weight data in the weight data subset increased to the third weight data set is 0; and acquiring the weight data of each of M2 weight data subsets in the third weight data set into one set, where S2=M2×(N3 / M2)×C2×H3×W3.

[0100] As an embodiment, the device further performs interleaving rearrangement on M2 pieces of weight data in each set of weight data of the S2 sets of weight data to obtain a second weight data set in the following manner: if N2 is an integer multiple of M2, then perform interleaving rearrangement on M2 pieces of weight data in each set of weight data of the S2 sets of weight data to obtain a second weight data set, where W4=M2×(N2 / M2)×C2 and H4=H3×W3; if N2 is not an integer multiple of M2, then perform interleaving rearrangement on M2 pieces of weight data in each set of weight data of the S2 sets of weight data to obtain a second weight data set, where W4=M2×(N3 / M2)×C2 and H4=H3×W3.

[0101] As one example, the value of N2 is the number of output channels of the convolution kernel, the value of C2 is the number of input channels of the convolution kernel, the convolution operation is a convolution operation performed using the convolution kernel, each weight data subset includes weight data for the C2 input channels, and each output channel includes the C2 input channels.

[0102] In one embodiment, the device further performs the convolution operation on the second image data set and the pre-acquired second weighted data set to obtain a target output result in the following manner: performing the convolution operation on the second image data set and the second weighted data set to obtain a fourth image data set, in which the target output result includes the fourth image data set, the second image data set being an image data set obtained by performing interleaving rearrangement on image data of M1 channels in each set of image data among the S1 sets of image data, the S1 set of image data being image data obtained by belonging each of the M1 channels of image data in the first image data set to one set, where M1≦C1.

[0103] In one embodiment, the device further includes: performing a convolution operation on the second image data set and the second weighting data set to obtain a fourth image data set in the following manner: obtaining C2 sets of image data from the second image data set, where each set of image data includes a plurality of image data in the same channel in the first image data set, and each set of image data is obtained by shifting the storage address of the previous set of image data in the C2 set of image data by one address; and performing a convolution operation on the C2 set of image data and N2×C2 sets of weighting data in the second weighting data set to obtain a fourth image data set. N2 sets of image data are obtained in the four image data sets, of which each set of weight data and each set of image data have the same data structure, of which the second weight data set is obtained by performing interleaving rearrangement on the data in the first weight data set, and the weight data in the first weight data set is arranged according to a data format of N2×C2×H3×W3, where N2 indicates the number of weight data subsets contained in the first weight data set, C2 indicates the number of channels in each weight data subset, H3 indicates the data height in each weight data subset, and W3 indicates the data width in each weight data subset.

[0104] In one embodiment, the device further performs a convolution operation on the C2 set of image data and the N2×C2 sets of weighting data in the second weighting data set in the following manner to obtain N2 sets of image data in the third image data set, i.e., performs a weighted sum operation between each of the C2 sets of weighting data in the N2×C2 sets of weighting data and the C2 set of image data to obtain N2 sets of image data.

[0105] In one embodiment, the device further stores the first image data set and the second image data set in a first memory space, and stores the second weighting data set in a second memory space, wherein the first memory space and the second memory space are independent memory spaces.

[0106] According to yet another aspect of the embodiment of the present application, there is further provided an electronic device for implementing the above-mentioned image data processing method, which may be a terminal device or a server as shown in Fig. 1. In this embodiment, an example will be described in which the electronic device is a server. As shown in Fig. 13, the electronic device includes a storage unit 1302 and a processor 1304, in which a computer program is stored, and the processor 1304 is configured to execute the computer program to realize any one of the above-mentioned method embodiments.

[0107] In this embodiment, the electronic device may be at least one of a plurality of network devices of a computer network.

[0108] In this embodiment, the above-mentioned processor may be configured to execute a computer program to perform the following steps.

[0109] S1: obtain a first image data set to be processed, in which the image data in the first image data set is arranged according to a data format of N1×C1×H1×W1, where N1 indicates the number of image data subsets included in the first image data set, C1 indicates the number of channels in each image data subset, H1 indicates the data height in each image data subset, and W1 indicates the data width in each image data subset; S2: performing interleaving rearrangement on the data in the first image data set to obtain a second image data set, in which the image data in the second image data set is arranged according to a data format of N1×H2×W2, and the manner of interleaving rearrangement is consistent with the convolution operation; and S3: Perform a convolution operation on the second image data set and the pre-acquired second weighted data set to obtain a target output result.

[0110] As can be understood by those skilled in the art, the configuration shown in FIG. 13 is merely an example, and the electronic device may be a terminal device such as a smartphone (e.g., an Android smartphone, an iOS smartphone, etc.), a tablet computer, a handheld computer, a MID (Mobile Internet Devices), a PAD, etc. Also, FIG. 13 does not limit the configuration of the electronic device described above. For example, the electronic device may further include more or less than those shown in FIG. 13 (e.g., an interface, etc.), or may have a different arrangement from that shown in FIG. 13.

[0111] Wherein, the storage unit 1302 may be used to store software programs and modules, for example, program instructions / modules corresponding to the image data processing method and device in the embodiment of the present application, and the processor 1304 may execute various functional applications and data processing by executing the software programs and modules stored in the storage unit 1302, that is, the above-mentioned image data processing method can be realized. The storage unit 1302 may include a high-speed random memory, and may further include a non-volatile memory, for example, one or more magnetic memories, flash memories, or other non-volatile solid-state memories. In some embodiments, the storage unit 1302 further includes a remote storage unit configured separately from the processor 1304, and such a remote storage unit is connected to the terminal through a network. Examples of the above-mentioned network may include, but are not limited to, the Internet, an in-house network, a local area network, a mobile communication network, and combinations thereof. Wherein, the storage unit 1302 is specifically used to store information such as image data to be processed, but is not limited to this. As an example, as shown in FIG. 13, the memory 1302 may include the acquisition module 1202, the processing module 1204, and the execution module 1206 in the image data processing device, and may also include other modules or units in the image data processing device, but is not limited to this.

[0112] In this embodiment, the above-mentioned transmission device 1306 is used to transmit and receive data via a network. The above-mentioned network may include a wired network and a wireless network. In one example, the transmission device 1306 includes a NIC (Network Interface Controller), which can communicate with the Internet or a local area network by connecting to other network devices and routers via cables. In one example, the transmission device 1306 is a RF (Radio Frequency) module, which is used to communicate with the Internet in a wireless manner.

[0113] In addition, the above-mentioned electronic device further includes a display 1308, which is used to display image data, and further includes a connection bus 1310, which is used to connect each module component in the above-mentioned electronic device.

[0114] In another embodiment, the terminal device or server may be a node in a distributed system, where the distributed system may be a block chain system, which may be a distributed system formed by multiple nodes connected through a network communication method, where a P2P (Peer To Peer) network may be formed between the nodes, and any type of computer device, such as a server, terminal, or other electronic device, may join the P2P network to become a node in the block chain.

[0115] According to another aspect of the present application, there is further provided a computer program product or computer program, the computer program product or computer program comprising computer instructions, the computer instructions being stored in a computer readable storage medium, a processor of a computing device reading the computer instructions from the computer readable storage medium, and the processor executing the computer instructions, thereby causing the computing device to realize the methods provided in the various realization manners of processing image data as described above, wherein the computer program, when executed, is configured to perform the steps of any of the method embodiments described above.

[0116] In this embodiment, the above-mentioned computer-readable storage medium may be configured to store a computer program that performs the following steps.

[0117] S1: obtain a first image data set to be processed, in which the image data in the first image data set is arranged according to a data format of N1×C1×H1×W1, where N1 indicates the number of image data subsets included in the first image data set, C1 indicates the number of channels in each image data subset, H1 indicates the data height in each image data subset, and W1 indicates the data width in each image data subset; S2: performing interleaving rearrangement on the data in the first image data set to obtain a second image data set, in which the image data in the second image data set is arranged according to a data format of N1×H2×W2, and the manner of interleaving rearrangement is consistent with the convolution operation; and S3: Perform a convolution operation on the second image data set and the pre-acquired second weighted data set to obtain a target output result.

[0118] In this embodiment, as can be understood by those skilled in the art, all or some of the steps in the various methods of the above-mentioned embodiments may be completed by a program instructing relevant hardware of a terminal device, and the program may be stored in a computer-readable storage medium, which may include a flash memory, a ROM (Read-Only Memory), a RAM (Random Access Memory), a magnetic disk, an optical disk, etc.

[0119] It should be noted that the numbers of the above-mentioned embodiments of the present application are for illustrative purposes only and do not indicate superiority or inferiority of the embodiments.

[0120] The integrated units in the above embodiments can be realized in the form of a software functional unit, and can be stored in the above computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially embodied in the form of a software product, or the part that has contributed to the prior art, or all or a part of the technical solution, and the computer software product can be stored in a storage medium and include a plurality of instructions to cause one or more computer devices (such as personal computers, servers, network devices, etc.) to execute all or a part of the steps of the methods described in each embodiment of the present application.

[0121] In the above embodiments of the present application, emphasis is placed on the description of each embodiment, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0122] In some embodiments provided in this application, it can be understood that the disclosed client may be realized in other ways, among which the above-described device embodiments are merely examples, for example, the division of the units is a division of functions, and there are other division ways in actual implementation, for example, multiple units or components may be combined or integrated into another system, or some may be omitted or not implemented. Another point is that the connections or direct connections or communication connections between each other shown or described may be indirect connections or communication connections through some interfaces, units or modules, or may be in electrical or other forms.

[0123] In addition, the units described above as separate components do not have to be physically separate, and the components shown as units do not have to be physical units, i.e., they may be located in one place or distributed among multiple network units, and some or all of the units may be selected according to actual needs to realize the technical solution of the present embodiment.

[0124] Furthermore, each functional unit in each embodiment of the present application may be integrated into one processing unit, may exist physically alone, or two or more of the units may be integrated into one unit. The above-mentioned integrated units may be realized in the form of hardware or software functional units.

[0125] Although the preferred embodiment of the present application has been described above, the present application is not limited to this embodiment, and any modification to the present application falls within the technical scope of the present application as long as it does not depart from the spirit of the present application.

Claims

1. 1. A method for processing image data, implemented by an electronic device, comprising: acquiring a first image data set to be processed, image data in the first image data set being arranged according to a first data format; performing interleaving rearrangement on the data in the first image data set to obtain a second image data set, the image data in the second image data set being arranged according to a second data format, the interleaving rearrangement corresponding to a convolution operation, and a dimension of the second data format being smaller than a dimension of the first data format; and performing the convolution operation on the second image data set and a pre-acquired second weighted data set to obtain a target output result; performing interleaving and reordering of data in the first image data set to obtain a second image data set; A step of grouping image data of each of M1 channels in the first image data set into one set to obtain S1 sets of image data, where M1≦C1; and performing interleaving and rearrangement on M1 channels of image data in each set of image data of the S1 sets of image data to obtain the second image data set.

2. 2. The method of claim 1, The image data in the first image data set is arranged according to a first data format, the image data in the first image data set is arranged according to a data format of N1×C1×H1×W1, N1 represents a number of image data subsets included in the first image data set, C1 represents a number of channels in each of the image data subsets, H1 represents a data height in each of the image data subsets of the first image data set, and W1 represents a data width in each of the image data subsets of the first image data set; The image data in the second image data set is arranged according to a second data format, The method includes: image data in the second image data set is arranged according to a data format of N1×H2×W2, H2 represents a data height in each of the image data subsets of the second image data set, and W2 represents a data width in each of the image data subsets of the second image data set.

3. 2. The method of claim 1, The step of obtaining S1 sets of image data by grouping the image data of each of the M1 channels in the first image data set into one set includes: when C1 is an integer multiple of M1, the image data of each of the M1 channels in the first image data set is grouped into one set to obtain S1 sets of image data, where S1=(C1 / M1)×H1×W1; if C1 is not an integer multiple of M1, increasing the number of channels in the first image data set from C1 to C2 to obtain a third image data set, where C2 is an integer multiple of M1 and the image data in the increased channels in the third image data set is zero; and The method includes a step of grouping image data of each of M1 channels in the third image data set into one set to obtain S1 sets of image data, where S1=(C2 / M1)×H1×W1.

4. 4. The method of claim 3, The step of performing interleaving rearrangement on image data of M1 channels in each set of image data among the S1 sets of image data to obtain the second image data set includes: performing interleaving rearrangement on the M1 channels of image data in each set of image data among the S1 sets of image data, when C1 is an integer multiple of M1, to obtain the second image data set, where W2=M1×W1 and H2=(C1 / M1)×H1; and The method includes a step of performing interleaving rearrangement on M1 channels of image data in each set of image data among the S1 sets of image data to obtain the second image data set, when C1 is not an integer multiple of M1, where W2=M1×W1 and H2=(C2 / M1)×H1.

5. The method of claim 2, further comprising: obtaining a first predetermined weight data set, the weight data in the first weight data set being arranged according to a data format of N2×C2×H3×W3, where N2 represents the number of weight data subsets included in the first weight data set, C2 represents the number of channels in each of the weight data subsets, H3 represents the data height in each of the weight data subsets, and W3 represents the data width in each of the weight data subsets; and The method includes a step of performing interleaving rearrangement on the data in the first weighted data set to obtain the second weighted data set, wherein the weighted data in the second weighted data set is arranged according to a data format of H4×W4, H4 represents a data height of the weighted data in the second weighted data set, and W4 represents a data width of the weighted data in the second weighted data set.

6. The method according to claim 5, performing interleaving rearrangement on the data in the first weighted data set to obtain the second weighted data set, a step of grouping weight data of each of M2 weight data subsets in the first weight data set into one set to obtain S2 sets of weight data, where M2≦N2; and performing interleaving rearrangement on M2 weight data in each set of weight data among the S2 sets of weight data to obtain the two weight data sets.

7. 7. The method of claim 6, The step of obtaining S2 sets of weight data by grouping the weight data of each of the M2 weight data subsets in the first weight data set into one set includes: when N2 is an integer multiple of M2, the weight data of each of M2 weight data subsets in the first weight data set is grouped into one set to obtain the S2 sets of weight data, where S2=M2×(N2 / M2)×C2×H3×W3; if N2 is not an integer multiple of M2, increasing the number of weight data subsets in the first weight data set from N2 to N3 to obtain a third weight data set, where N3 is an integer multiple of M2, and the weight data in the increased weight data subset in the third weight data set is 0; and The method includes a step of grouping weight data of each of M2 weight data subsets in the third weight data set into one set to obtain the S2 set of weight data, where S2=M2×(N3 / M2)×C2×H3×W3.

8. 8. The method of claim 7, The step of performing interleaving rearrangement on M2 weight data in each set of weight data among the S2 sets of weight data to obtain the double weight data set includes: performing interleaving rearrangement on M2 weight data in each set of weight data among the S2 sets of weight data when N2 is an integer multiple of M2 to obtain the double weight data set, where W4=M2×(N2 / M2)×C2 and H4=H3×W3; and When N2 is not an integer multiple of M2, the method includes a step of performing interleaving rearrangement on M2 weight data in each set of weight data among the S2 sets of weight data to obtain the double weight data set, where W4=M2×(N3 / M2)×C2 and H4=H3×W3.

9. 6. The method of claim 5, the value of N2 is a number of output channels of a convolution kernel, the value of C2 is a number of input channels of the convolution kernel, the convolution operation is a convolution operation performed using the convolution kernel, and each of the weight data subsets includes weight data for C2 of the input channels.

10. 10. The method according to any one of claims 1 to 9, comprising: performing the convolution operation on the second image data set and the pre-acquired second weighted data set to obtain a target output result; performing the convolution operation on the second image data set and the convolution data set to obtain a fourth image data set, wherein the target output result includes the fourth image data set, the second image data set being an image data set obtained by performing interleaving rearrangement on image data of M1 channels in each set of image data among S1 sets of image data, the S1 set of image data being image data obtained by belonging each of the M1 channels of image data in the first image data set to one set, and M1≦C1.

11. 11. The method of claim 10, performing a convolution operation on the second image data set and the double image data set to obtain a fourth image data set, obtaining a C2 set of image data from the second image data set, each set of image data including a plurality of image data in the same channel in the first image data set, and each set of image data being obtained by shifting the storage address of a previous set of image data in the C2 set of image data by one address; and performing a convolution operation on the C2 sets of image data and N2×C2 sets of weight data in the second image data set to obtain N2 sets of image data in the fourth image data set, wherein each set of weight data and each set of image data have the same data structure; the second weight data set is obtained by performing interleaving rearrangement on data in the first weight data set, and the weight data in the first weight data set is arranged according to a data format of N2×C2×H3×W3, where N2 represents the number of weight data subsets included in the first weight data set, C2 represents the number of channels in each of the weight data subsets, H3 represents the data height in each of the weight data subsets, and W3 represents the data width in each of the weight data subsets.

12. The method according to any one of claims 1 to 9, further comprising: storing the first image data set and the second image data set in a first memory space; and storing the double data set in a second memory space; The method, wherein the first memory space and the second memory space are independent memory spaces.

13. An electronic device including a storage device and a processor connected to the storage device, The storage device stores a computer program, 10. An electronic device, the processor being configured to execute the computer program to implement the method according to any one of claims 1 to 9.

14. A program for causing a computer to execute the method according to any one of claims 1 to 9.

Citation Information

Patent Citations

  • Tensor processor

    CN110033085A

  • Efficient Data Layout for Convolutional Neural Networks

    JP2019535079A