Preprocessing and data augmentation of frequency-domain data

Abstract

Methods and systems for performing preprocessing operations and augmentation operations on an image dataset converted to a frequency domain representation are provided. The method includes: decoding an image of the image dataset to generate an image dataset in a frequency domain representation; performing a resizing operation on the image dataset in the frequency domain representation based on a resizing factor; performing a reshaping operation on the image dataset in the frequency domain representation based on a reshaping factor; and performing a cropping operation on the image dataset in the frequency domain representation. The method and system may further include performing an augmentation operation on the image dataset in the frequency domain representation. The methods and systems of the present disclosure free the learning model from the computational overhead caused by converting the image dataset to a frequency domain representation. In addition, the computational overhead caused by the inverse transformation operation is also reduced.

Description

Background Art

[0001] Many forms of information, such as audio and images, can be converted into their various representations in the frequency domain and then back to their original representations through inverse transforms. The frequency domain representations of information have many applications in data processing. For example, image coding algorithms typically compress image data by converting the image data into a frequency domain representation, because algorithms that operate on the frequency domain data can easily achieve a reduction in the size of the image data.

[0002] In a machine learning setting where an image-based learning model is applied, the training and inference data sets can be off-the-shelf data sets of compressed images, but can also be raw, uncompressed images captured by an image capture device such as a camera. In a system implemented for machine learning computations on a large volume of data sets, the computing resources can be distributed over a network in, for example, a cloud computing system to maximize computing power and availability. Since uncompressed images cannot be transmitted over the network and for training and inference computations, a machine learning system configured in this way also relies on the transform of the images into the frequency domain and the inverse transform out of the frequency domain in order to compress the image data sets for optimal transmission, storage, and computation.

[0003] In a machine learning environment, converting an image data set into the frequency domain typically helps to determine context information (such as the relative importance of the image data) by applying principles originating from, for example, frequency filtering. However, currently, there is no channel between the encoding of the image representation in the frequency domain and the derivation of the image representation in the frequency domain for training and inference computations. In further applications of machine learning, higher computational efficiency can be achieved by leveraging image coding in the frequency domain. Brief Description of the Drawings

[0004] The detailed description part is elaborated with reference to the accompanying drawings. In the drawings, the leftmost digit of the reference number indicates the drawing in which the reference number first appears. The same reference number is used in different drawings to represent similar or identical items or features.

[0005] Figure 1A A flowchart showing the preprocessing and enhancement process of an image data set according to an exemplary embodiment of the present disclosure, which will be input to a learning model for training and inference.

[0006] Figure 1B and 1C Shows the steps of the preprocessing process performed on an example image.

[0007] Figure 2A An architecture diagram of a cloud computing system deployed with a learning model according to an exemplary embodiment of the present disclosure.

[0008] Figure 2B shows a data center processor of an example according to Figure 2A .

[0009] Figure 3 shows an example system for implementing the processes and methods for data processing and data augmentation for implementing a learning model described herein. DETAILED DESCRIPTION

[0010] The systems and methods discussed herein are directed to implementing data preprocessing and data augmentation for a learning model, and more specifically, performing data preprocessing and data augmentation on an image data set after converting the image data set for model training into the frequency domain and before its inverse transform.

[0011] According to an example embodiment of the present disclosure, a learning model may be a defined computational algorithm executable by one or more processors of a computing system, and the algorithm may be used to perform tasks including processing inputs and output results having various parameters. For example, the learning model is a hierarchical model such as a deep neural network, having a fully connected structure, having a feedforward structure such as a convolutional neural network (“CNN”), having a backpropagation structure such as a recurrent neural network (“RNN”), or having other architectures suitable for computations of specific tasks. For example, the tasks include classification, clustering, matching, regression, and the like.

[0012] The tasks may provide outputs for the execution of functions, such as identifying entities in images and / or videos; real-time tracking of entities moving in video frames; matching entities identified in images and / or videos with other images and / or videos; real-time providing annotations or transcriptions of images, videos, and / or audio; and the like.

[0013] A cloud computing system may provide a collection of servers that host computing resources to provide improvements in the availability of distributed computing, parallel computing, physical or virtual computing resources, and the like benefits. A data center implemented in the cloud computing system may host a learning model to provide these benefits for computing applications using the learning model.

[0014] A cloud computing system can be connected to various terminal devices, which collect data to be input into a learning model to train the learning model and / or data associated with various computing tasks, as well as output the results required to execute these tasks. The terminal devices can be connected to the cloud computing system through the edge nodes of the cloud computing system. An edge node can be any server that provides an outbound connection to other nodes connected to the cloud computing system. Therefore, the edge node can divide the logical edge of the network of the cloud computing system, rather than necessarily the physical edge. In addition, the edge node can be a logic node based on the edge, on which decentralized computing resources of the cloud computing system are deployed, such as cloudlets, fog nodes, etc.

[0015] The terminal devices can be responsible for multiple types of tasks, which involve multiple types of input data. For example, the terminal device can be a security camera that collects still images, videos, and other types of data, and is connected to an edge node of the cloud computing system, which can perform real-time security monitoring on the locations monitored by these security cameras and send each type of data to the cloud computing system in real time to perform various tasks related to real-time security monitoring. According to another example, the terminal device can be an intelligent vehicle in motion, which collects still images, videos, and other types of data during motion and is connected to a roadside network infrastructure, which serves as an edge node of the cloud computing system. The cloud computing system performs real-time traffic monitoring on the intelligent vehicle and sends each type of data to the cloud computing system in real time to perform various tasks related to real-time traffic monitoring.

[0016] These different types of inputs may all need to be computed through the learning model. Since the files of the image datasets used in deep learning are very large, the loading and computing of the image datasets may incur huge computing overhead.

[0017] In the Internet of Things network connected to the cloud computing system, a large number of tasks and associated data sets are transmitted from the edge devices to the cloud computing system for processing. The edge devices of the Internet of Things network may be geographically far from the computing resources of the cloud computing system, resulting in network latency in the transmission of tasks and related data. Compared with the cloud computing system, the computing resource specifications (including processing power, storage, and memory) of the edge devices are relatively low, which also leads to a relative difference in computing power between them. Various architecture designs can be implemented on the learning model to address the differences in network latency and computing power, thereby improving the response time and efficiency of task completion.

[0018] For example, delivering a large amount of local data collected on edge devices to a cloud computing system for computation by a learning model executed on the cloud computing system may result in intolerable latency. In addition, images in their original, uncompressed format are extremely inefficient for machine learning computations because they contain many times more data than compressed images, and this data is often redundant for the purposes of machine learning training and inference. Therefore, it is desirable to compress the images captured by an image capture device before using them in training and inference data sets.

[0019] As is well known, data can be transformed into a frequency representation through a transform algorithm such as the Fourier transform, and image data can be transformed from a spatial domain representation (i.e., image pixel data) to a frequency domain representation through a Fourier-related algorithm such as the discrete cosine transform (“DCT”). The inverse discrete cosine transform (“IDCT”) can also be used to transform the frequency domain representation of image data back to the spatial domain representation. Common image compression formats (such as JPEG) utilize the DCT to transform the original image into a frequency domain representation and selectively discard image data by frequency. DCT-based image encoders save images in a compressed format, and the compressed-format images cannot be used for computation until they are decoded into reconstructed images. However, the reconstructed images are still represented in the frequency domain and must undergo the IDCT to return to the spatial domain representation.

[0020] Currently, the implementation of model training and inference is based on the assumption that an image data set is input to the learning model in a spatial domain representation. In addition, before inputting the image data set to the learning model, the image data set can be modified through processes widely known as preprocessing and augmentation. Preprocessing of an image data set refers to any type of operation performed on the data of the data set images to modify the images in a manner required by the specific needs of model training and / or inference (e.g., to make the images conform to the dimensions required for the input images of the learning model). For example, the images are transformed to different color spaces, resized, reshaped, cropped (at fixed positions on the image, such as the center position or a specific corner position), centered and normalized, or other such operations known to those skilled in the art are performed. Augmentation of an image data set can refer to any type of operation performed on the data of the data set images to generate additional images in the image data set. For example, the images are resized, rotated, reshaped, cropped (at any position on the original image, any number of times), rescaled in any or all of their color channels, moved a certain number of pixels in any direction, changed in brightness or contrast, or flipped in any direction, or other such operations known to those skilled in the art can be performed.

[0021] Therefore, when the image dataset is in its original uncompressed format, during the compression of each image in the image dataset, performing DCT on the entire image dataset first and then IDCT will incur a large amount of computational overhead. Even if the image dataset is in a compressed format, applying preprocessing and enhancement operations to these compressed images requires decoding the compressed images and converting them into a decompressed spatial domain representation. Given that the goal of enhancement is to generate additional images from each input image, the computational overhead of these operations may increase with the number of input images, the number of desired output images, and the pixel size of each image.

[0022] Exemplary embodiments of the present disclosure provide for the training of a learning model using an image dataset converted to a frequency domain representation. Additionally, to facilitate such training, exemplary embodiments of the present disclosure provide preprocessing of the image dataset in the frequency domain representation and enhancement of the image dataset in the frequency domain representation. Thus, exemplary embodiments of the present disclosure enable preprocessing and enhancement of the image dataset to be performed on the frequency domain representation of the images, without performing IDCT through a decoder or converting it into a spatial domain representation, thereby simplifying the computational complexity of the learning and inference calculations of the learning model and allowing the learning model to more directly and effectively evaluate the relative importance of image data using information that is available in the frequency domain but not easily determined in the spatial domain.

[0023] Figure 1A A flowchart showing the preprocessing and enhancement process of an image dataset according to an exemplary embodiment of the present disclosure, which will be input to a learning model for training and inference.

[0024] Before the steps of process 100, the images of the image dataset have been captured by an image capture device in the original image format. For example, the image capture device can be a security camera or a camera of an intelligent vehicle. The image capture device can capture the images constituting the image dataset in the form of still images or videos. The image capture device can also encode the captured images into a compressed format by an encoding mode of a codec that converts the captured images into their frequency domain representation.

[0025] According to an exemplary embodiment of the present disclosure, an image capture device may be connected to a computing device for performing image decoding, preprocessing, and enhancement operations through a data bus connection, an optical data connection, or other connections suitable for transmitting images, or the image capture device may be connected to such a computing device through a wired network connection or a wireless network connection. For example, the computing device may be a personal computing system, a cluster of computing systems, a server of a cloud computing system (such as an edge server), an embedded computing system of a smart device (such as a vehicle or an appliance), etc. According to an exemplary embodiment of the present disclosure, an exemplary computing device may also be used to perform subsequent training and inference calculations using an image dataset according to its respective computing capabilities. In an exemplary embodiment of the present disclosure, the computing device may not be used to perform training and inference calculations, but is connected through a suitable connection (such as a wired network connection or a wireless network connection) to an additional computing device that can be used to perform training and inference calculations. In any case, at least, each step of process 100 may be performed by the computing device described herein.

[0026] Optionally, according to other exemplary embodiments of the present disclosure, the respective images of the image dataset may come from one or more pre-compiled image datasets, rather than the original images captured by the image capture device. According to such an exemplary embodiment, the image dataset obtained in this way may already be encoded into a compressed format by a unified codec for subsequent preprocessing and enhancement steps. These image datasets may be stored on various storage devices, such as the non-volatile storage device of a computing device according to an exemplary embodiment of the present disclosure, and the general computer-readable storage medium defined by the present disclosure.

[0027] In step 102, a compressed image of the image dataset is obtained and input to a decoder.

[0028] According to an exemplary embodiment of the present disclosure, the compressed image has been transformed into a frequency domain representation by DCT and subsequently encoded into a compressed format.

[0029] In step 104, the decoder decodes the compressed image into an image dataset in a frequency domain representation.

[0030] For example, the decoder may apply entropy decoding and dequantization to the compressed image. However, the decoder does not then apply the IDCT to the compressed image to return the compressed image to its spatial domain representation. At this stage, the decoded image in the frequency domain representation may be represented by DCT blocks (i.e., coefficient arrays), and have a channel size n, a dimensions height H, and a dimensions width W. Additionally, the components of the image may be represented by separate DCT blocks. For example, according to the implementation of the JPEG codec, before the image is converted to the frequency domain representation, the image may be converted to its representation in the YCbCr color space such that the luminance component (Y) is separated from the chrominance components (Cb and Cr). Each of the luminance and chrominance components may be represented in the frequency domain by separate DCT blocks.

[0031] In step 106, an enhancement operation is performed on the image data set in the frequency domain representation.

[0032] For the sake of simplicity, the data enhancement operation is described in more detail below with reference to step 112. The enhanced image data set can be used for the training calculation of the learning model. However, according to an exemplary embodiment of the present disclosure, it may be assumed that the learning model requires an image data set with a uniform dimension of height h and width w, and at this stage of process 100, the enhanced image data set may not yet have such a uniform dimension on all images. Therefore, according to an exemplary embodiment of the present disclosure, the enhanced image data set after step 106 may not yet be used for the training calculation of the learning model. Optionally, the enhancement operation performed in step 106 does not affect the image dimensions of the image data set. Optionally, according to an exemplary embodiment of the present disclosure, step 106 may not be performed. Instead, the enhancement operation is performed after modifying the dimensions of the image data set by at least some subsequent preprocessing operations (e.g., as described below with reference to step 212).

[0033] The enhancement of the image data set refers to any type of operation performed on the data of each image in the data set to generate additional images in the image data set. For example, the size, rotation, reshaping, cropping, rescaling of the image may be adjusted in any or all color channels of the image, and a certain number of pixels may also be moved in any direction of the image, changes may be made in brightness or contrast, or the image may be flipped in any direction, or other such operations known to those skilled in the art may be performed. Additionally, the operations that the image undergoes may be operations only on the frequency domain representation, for example, applying frequency jitter to it. Therefore, the image data set after the enhancement operation may contain some images derived from other images in the image data set through any of the above operations or a combination thereof, as well as any other suitable operations known to those skilled in the art.

[0034] Training computations can generally be performed to train a learning model based on a loss function to learn feature embeddings for computing one or more tasks. For example, the tasks can include classification, clustering, matching, regression, etc. Performing such training computations is well known to those skilled in the art and will not be reiterated here.

[0035] In step 108, a resizing operation is performed on the image dataset in the frequency domain representation based on a resizing factor.

[0036] According to an exemplary embodiment of the present disclosure, several preprocessing operations can be performed on the image dataset so that each image of the image dataset has a unified dimension: height h and width w, and the unified height h and width w are less than the original height H and width W required by the learning model. One of these preprocessing operations can be the resizing operation described below. Another of these preprocessing operations can be the reshaping operation described with reference to step 110 below. Another of these preprocessing operations can be the cropping operation described with reference to step 114 below.

[0037] The images of the image dataset can be resized according to any known resizing algorithm applicable to the frequency domain representation of the image (e.g., DCT blocks). For example, a hybrid resizing algorithm ("HRA") can be applied. Generally, the first DCT block of an image with dimensions of height H and width W can be resized to a second DCT block of an image with dimensions of height h = H×P / L and width w = W×Q / M. P / L and Q / M are the respective resizing factors for the height and width of the image to be resized. P, Q, L, and M can be integers, and L and M can be preset values.

[0038] Since the image is composed of DCT blocks, this can be achieved by converting each DCT block (i.e., coefficient array) from dimension N h ×N w to dimension K h ×K w where N h and N w are the integer height and integer width of the dimensions of the DCT block before conversion, and K h and K w are the integer height and integer width of the dimensions of the DCT block after conversion. To be consistent with the resizing factor of the overall image, the conversion should resample the DCT block from dimension LN h ×MN w to dimension PN h ×QN w , first padding the image on adjacent sides or all four sides as needed to make it consistent with LNh ×MN w (Known dimensional values) are proportional.

[0039] Therefore, through the height resampling factor r h = H / (LN h ) and the width resampling factor r w = W / (MN w ) resample the DCT blocks such that r h (LN h )(P / L) = K h and r w (MN w )(Q / M) = K w . Therefore, for HRA calculation, P is derived as P = K h (N h r h ) and Q = K w (N w r w ).

[0040] Therefore, by applying HRA to the first DCT block (n, H, W) having a channel size n, a height dimension H, and a width dimension W, the first DCT block is resized to a second DCT block (n, H×P / L, W×Q / M) having a channel size n, a height dimension H×P / L, and a width dimension W×Q / M. Figure 1B An example in this regard is shown.

[0041] In step 110, a reshaping operation is performed on the image data set in the frequency domain representation based on the reshaping factor.

[0042] Since the image is represented in the frequency domain rather than the spatial domain, the preprocessing operation does not remove pixel information. Therefore, if the preprocessing operation is applied to the DCT blocks in a conventional manner, the preprocessing operation is not easily applicable to the frequency information. For example, since an image can be composed of multiple DCT blocks, each DCT block representing a component of the image (such as Y, Cb, and Cr components), preprocessing each DCT block separately may not effectively retain the associated image data. Therefore, it is desirable to generate a single representation of the image while maintaining the frequency domain representation.

[0043] Therefore, according to an exemplary embodiment of the present disclosure, in order to preprocess an image in the frequency domain representation, the image can be reshaped such that all image data of the same frequency in each component of the image are aggregated in the same channel. To achieve this, the resized image (n, H×P / L, W×Q / M) can be further reshaped to have a channel size n×N h ×N w , a height H×P / L / N hand width W×Q / M / N w of the reshaped image (n×N h ×N w , H×P / L / N h , W×Q / M / N w ).

[0044] Thus, the height and width of each image are reduced by their respective reshaping factors, while each channel dimension is increased proportionally in two dimensions by the same reshaping factor, resulting in a rank-2 N h ×N w matrix tensor. An example of this operation is shown in Figure 1B by step 110A. The rank-2 tensor is then reduced to a rank-1 tensor, which has the same number of channels but reduced dimensions. The number of channels can be the same as the number of DCT coefficients in each block. An example of this operation is shown in Figure 1B by step 110B.

[0045] Next, the reduced-dimension tensors can be aggregated by frequency to produce a rank-2 aggregated tensor, where each channel of the rank-2 aggregated tensor contains all the image data of the same frequency. The number of channels of the aggregated tensor can be equal to the number of coefficients in the original DCT block. An example of this operation is shown in Figure 1C by step 110C. This reshaping operation enables subsequent cropping step 114 to be performed consistently across the frequency channels of the image for each of the Y, Cb, and Cr components.

[0046] At step 212, an enhancement operation is performed on the image data set in the frequency domain representation.

[0047] According to an example embodiment of the present disclosure, the enhancement operation can be performed by reshaping the frequency domain represented image from an H×W image into a three-dimensional (H / N)×(W / N)×(N / N) dimensional tensor.

[0048] According to an exemplary embodiment of the present disclosure, the enhancement operation can be a flipping operation, and the flipping operation can be performed on the tensor in a manner similar to that performed on the spatial domain representation of the image, where the values in the tensor can be mirror-imaged across an axis to generate a new tensor of the same dimension.

[0049] According to an exemplary embodiment of the present disclosure, the enhancement operation can be a rotation operation, and the rotation operation can be performed on the tensor in a manner similar to that performed on the spatial domain representation of the image, where the values in the tensor can be rotated in any suitable direction to generate a new tensor of the same dimension.

[0050] According to an exemplary embodiment of the present disclosure, the augmentation operation may be a cropping operation (at a random position, any number of times), and the cropping operation may be performed on the tensor in a manner similar to that performed on the spatial domain representation of the image, where the values in the tensor may be cropped to any suitable subset to generate a new tensor of a smaller dimension.

[0051] According to an exemplary embodiment of the present disclosure, the augmentation operation may be a noise injection operation, and the noise injection operation may be performed on the tensor in a manner similar to the color jitter performed on the spatial domain representation of the image, where a random value may be added to each value of the tensor to generate a new tensor of the same dimension.

[0052] According to an exemplary embodiment of the present disclosure, the augmentation operation may be a channel reweighting operation, and the channel reweighting operation may be performed on the image representation in the spatial domain by multiplying each channel coefficient by a weighting factor. The weighting factor may be from 0 to 1 (including 1), or may be from 0 to a value greater than 1. The weighting factor may be determined for various purposes, for example, certain resolutions of the image should be emphasized and / or deemphasized: for example, in the case where the images in an image dataset are compressed in JPEG format and the quality factor in the JPEG encoder settings is known, the frequencies of enhancement and non-enhancement in the compressed image may be determined based on the quality factor. Therefore, the weighting factor may be set to weight the frequencies favored by the quality factor to be higher than the frequencies disfavored by the quality factor.

[0053] According to an exemplary embodiment of the present disclosure, other augmentation operations may be any operations on the image representation in the frequency domain or its tensor, which respectively generate any new image or tensor derived from the original image or tensor data.

[0054] Currently, the augmented image dataset can be used for the training calculation of the learning model.

[0055] In step 114, a cropping operation is performed on the image dataset in the frequency domain representation.

[0056] The images of the reshaped image dataset can now be directly cropped to the dimension h×w required as the input of the learning model. An example of this operation is as Figure 1C shown. Before step 114, the size of the image (represented as a tensor as described above) may be further adjusted as needed, for example, during the augmentation step 112 or for the purpose of the cropping step 114. After step 114, the image (represented as a cropped tensor) may be normalized as needed to meet the dimension requirements of each other image in the image dataset processed in this way. According to an exemplary embodiment of the present disclosure, the cropping position may be determined as follows.

[0057] During the execution of step 108, when all four sides of the image are filled, the cropping operation can be a central cropping performed on all four sides of the image.

[0058] During the execution of step 108, when the image is filled in the lower right part, the cropping operation can be a corner cropping performed on the upper left edge of the image.

[0059] In step 116, the image data set in the frequency domain representation is input into the learning model.

[0060] The image data set can be input to the learning model for training or inference calculation, while the image data set that has not undergone IDCT returns to the spatial domain representation. Machine learning inference calculations known to those skilled in the art can be performed more efficiently through the image data set in the frequency domain representation.

[0061] Figure 2A The architecture diagram of the cloud computing system 200 hosting the learning model according to an exemplary embodiment of the present disclosure is shown. As described above, although the exemplary computing device according to the exemplary embodiment of the present disclosure can be used to perform subsequent training and inference calculations using the image data set, in other cases, additional computing devices communicating through a network connection can be used to perform training and inference calculations. Therefore, Figure 2A The architecture of some possible embodiments of the computing device as described above is shown.

[0062] The cloud computing system 200 is implemented by physical or virtual server nodes 204(1), 204(2), ……, 204(N) (any unspecified server node can be referred to as server node 204) connected by physical or virtual network connections of the network 202. Moreover, the network 202 terminates at physical or virtual edge nodes 206(1), 206(2), ……, 206(N) (any unspecified edge node is referred to as edge node 206) located at the physical and / or logical edge of the network 202. The edge nodes 206(1) to 206(N) can be connected to any number of terminal devices 208(1), 208(2), ……, 208(N) (any unspecified terminal device can be referred to as terminal device 208). As an example, the terminal device 208 (e.g., terminal device 208(1)) can collect data captured by any number of sensors 210(1), 210(2), ……, 210(N), where the sensors can be, for example, image capture devices for collecting image data (e.g., still images, video frames, etc.).

[0063] The learning model 214 implemented on the cloud computing system 200 can be stored on the physical or virtual memory 216 (“data center memory”) of the data center 218, and the learning model 214 can be loaded into the physical or virtual memory 220 (“data center memory”) of the data center 218 so that one or more physical or virtual processors 222 (“data center processors”) of the data center 218 can use the learning model 214 to perform training and / or inference calculations to perform calculations on an input data set related to a neural network computing task. The data center processor 222 can be a dedicated computing device that facilitates the calculations of the neural network computing task. For example, the data center processor 222 can be an accelerator such as a Neural Network Processing Unit (“NPU”), a Graphics Processing Unit (“GPU”), a Tensor Processing Unit (“TPU”), etc. Figure 2B An exemplary processor 222 is shown, which includes any number of cores 222A. The processing power of the processor 222 can be distributed across the respective cores 222A. Each core 222A includes a local memory 222B, which is, for example, a constant buffer for batch normalization or quantization for the execution of dedicated calculations. The local memory 114 can contain pre-initialized data such as model weights or data structures. Each core 222A can also be configured to execute one or more sets of computer-executable acceleration engine modules 222C pre-initialized on the local memory 222D of the core 222A. These modules can be executed by the core 222A, including being executed in parallel by multiple cores 222A, to perform or accelerate arithmetic operations such as matrix multiplication or matrix transpose, functional operations such as convolution, specially defined operations such as pooling, interpolation, or region of interest operations of a convolutional layer, and data operations. Each core 222A also includes an instruction sequencer 222E, which receives instructions from an instruction buffer 222F and sorts the received instructions. Some cores 222A (e.g., four) can communicate via a data bus 222G (e.g., a unidirectional ring bus). The software drivers of the respective cores 222A can send executable commands through a command processor interface 222H to control the operations of each core 222A and synchronize their operations.

[0064] According to an exemplary embodiment of the present disclosure, the computing device as described above can be any one of the server node 204, the edge node 206, or the terminal device 208, and an image data set captured at the terminal device 208 can be loaded into the physical or virtual memory 224 (“terminal memory”) of the terminal device 208 so that one or more physical or virtual processors 226 (“terminal processors”) of the terminal device 208 perform the encoding, preprocessing, and / or enhancement operations as described above. Optionally, the image data set captured at the terminal device 208 can be transmitted to any edge node 206 and loaded into the physical or virtual memory 228 (“edge memory”) of the edge node 206 so that one or more physical or virtual processors 230 (“edge processors”) of the edge node 206 perform the encoding, preprocessing, and / or enhancement operations as described above. Optionally, the image data set captured at the terminal device 208 can be transmitted to any server node 204 and loaded into the physical or virtual memory 220 of the cloud computing system 200 so that one or more physical or virtual processors 222 of the cloud computing system 200 perform the encoding, preprocessing, and / or enhancement operations as described above. Optionally, the image data set can be from one or more pre-compiled image data sets rather than being captured, and as long as they are compressed into a unified format, no compression is required. Generally, if it is an image of an image data set captured and compressed according to an exemplary embodiment of the present disclosure, it can be compressed at any terminal device 208 rather than other devices to minimize the data transmitted.

[0065] According to an embodiment of the present disclosure, the preprocessed and / or enhanced image data set can be transmitted to one or more server nodes 204 and the cloud computing network 200 performs training and / or inference calculations based on the learning model 214. Generally, training and inference calculations may consume a large amount of computing resources. Therefore, it is desirable to reduce the calculations by, for example, bypassing the first layer of the learning model when referring to Figures 1A-1C the input image data set as described above, and avoiding the redundant operation of converting the decoded image data into a spatial domain representation before inputting it into the learning model after inputting it into the learning model. The example embodiment of the present disclosure can achieve the goal by performing preprocessing and enhancement operations in the frequency domain and then inputting the preprocessed and enhanced image data set into the learning model without the decoder performing IDCT on the image data set.

[0066] Figure 3 An example system 300 for implementing the above processes and methods for data processing and data enhancement of a learning model in the frequency domain is shown.

[0067] The techniques and mechanisms described herein can be implemented by multiple instances of system 300 and any other computing device, system, and / or environment. System 300 can be a distributed system that includes: multiple physically networked computers or web servers, physical or virtual clusters, computing clouds, or other networked computing architectures that provide physical or virtual computing resources known to those skilled in the art. Examples of system 300 include learning systems, such as the cloud computing system referenced above Figure 2A described cloud computing system. Figure 3 The system 300 shown is only one example of a system and is not intended to impose any limitation on the scope of use or functionality of any computing device for performing the above processes and / or programs. Other known computing devices, systems, environments, and / or configurations that may be suitable for use with the embodiments include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, gaming consoles, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, implementations using field programmable gate arrays (“FPGA”) and application specific integrated circuits (“ASIC”), and so on.

[0068] System 300 includes one or more processors 302 and a system memory 304 communicatively coupled to the processors 302. The processors 302 and the system memory 304 can be physical, virtual, and / or distributed. The processors 302 can execute one or more modules and / or processes to cause the processors 302 to perform various functions. In some embodiments, the processors 302 may include a central processing unit (“CPU”), GPU, NPU, TPU, any combination thereof, or other processing units or components known in the art. Additionally, each processor 302 can have its own local memory, which can also store program modules, program data, and / or one or more operating systems.

[0069] Depending on the exact configuration and type of system 300, the system memory 304 can be volatile (e.g., RAM), non-volatile (e.g., ROM), flash memory, micro hard disk drive, memory card, etc., or some combination thereof. The system memory 304 can include one or more computer-executable modules 306 executable by the processors 302. The modules 306 can be hosted as services on a network by a data processing platform, which can be implemented on a system separate from system 300.

[0070] Module 306 may include, but is not limited to, a dataset acquisition module 308, a decoding module 310, a selective enhancement module 312, a resizing module 314, a reshaping module 316, an enhancement module 318, a cropping module 320, and a model input module 322.

[0071] The dataset acquisition module 308 may be configured to obtain, as described above with reference to Figures 1A-1C the compressed images of the image dataset and input the images into the decoder.

[0072] The decoding module 310 may be configured to decode, as described above with reference to Figures 1A-1C the compressed images into an image dataset in the frequency domain representation.

[0073] The selective enhancement module 312 may be configured to perform enhancement operations on the image dataset in the frequency domain representation before the subsequent described modules perform preprocessing operations, as described above with reference to Figures 1A-1C the above.

[0074] The resizing module 314 may be configured to perform resizing operations on the image dataset in the frequency domain representation based on a resizing factor, as described above with reference to Figures 1A-1C the above.

[0075] The reshaping module 316 may be configured to perform reshaping operations on the image dataset in the frequency domain representation based on a reshaping factor, as described above with reference to Figures 1A-1C the above.

[0076] The enhancement module 318 may be configured to perform enhancement operations on the image dataset in the frequency domain representation before the subsequent described modules perform preprocessing operations, as described above with reference to Figures 1A-1C the above.

[0077] The cropping module 320 may be configured to perform cropping operations on the image dataset in the frequency domain representation, as described above with reference to Figures 1A-1C the above.

[0078] The model input module 322 may be configured to input the image dataset in the frequency domain representation into a learning model, as described above with reference to Figures 1A-1C the above.

[0079] The system 300 may further include an input / output (“I / O”) interface 340 and a communication module 350 to allow the system 300 to communicate with other systems and devices via a network, such as communicating with the edge nodes and terminal devices as described above. The network may include the Internet, wired media (such as wired networks or direct wired connections), and wireless media (such as acoustic, radio frequency (“RF”), infrared, and other wireless media).

[0080] Some or all of the operations of the method as described above can be performed by executing computer-readable instructions stored on a computer-readable storage medium, as defined below. The term "computer-readable instructions" as used in the specification and claims includes routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions can be implemented on a variety of system configurations, including single-processor systems or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based systems, programmable consumer electronics, and combinations thereof, etc.

[0081] The computer-readable storage medium can include volatile memory (e.g., random-access memory ("RAM")) and / or non-volatile memory (e.g., read-only memory ("ROM"), flash memory, etc.). The computer-readable storage medium can also include additional removable and / or non-removable memory, including but not limited to, flash memory, magnetic memory, optical memory, and / or tape memory, which can provide non-volatile storage of computer-readable instructions, data structures, program modules, etc.

[0082] A non-transitory computer-readable storage medium is an example of a computer-readable medium. Computer-readable media include at least two types of computer-readable media, namely computer-readable storage media and communication media. Computer-readable storage media include volatile and non-volatile, removable and non-removable media that are implemented in any process or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, phase change memory ("PRAM"), static random-access memory ("SRAM"), dynamic random-access memory ("DRAM"), other types of random-access memory ("RAM"), read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), flash memory or other storage technologies, compact disk read-only memory ("CD-ROM"), digital versatile disk ("DVD") or other optical memory, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device. In contrast, communication media may embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, or other transmission mechanism. As defined herein, computer-readable storage media do not include communication media.

[0083] Computer-readable instructions stored on one or more non-transitory computer-readable storage media, when executed by one or more processors, can perform the operations described above with reference to FIGS. 1-2. Generally, computer-readable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order of description of the operations is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0084] With the above technical solutions, the present disclosure provides preprocessing operations and enhancement operations on an image data set converted into a frequency-domain representation, such that the image data set is preprocessed and enhanced and then input into a learning model for training and inference in all frequency domains. For example, an image data set that is first captured as a raw image and then compressed into a frequency-domain representation by a unified codec can benefit from these operations, because encoding the image while keeping it in the frequency-domain representation after transformation enables the image to be directly input into a layer of a learning model that performs frequency-domain calculations. The preprocessing and enhancement operations described herein are customized for the frequency-domain representation of images and include some behaviors different from those of spatial-domain preprocessing and enhancement operations. The normal decoder behavior of returning the image data set to the spatial-domain representation through inverse transformation is skipped. Since the calculations based on the image data set are usually performed in the frequency domain, the computationally intensive training and inference calculations of a learning model (such as a neural network) can be freed from the additional computational overhead caused by converting the image data set into the frequency-domain representation for the second time. In addition, the computational overhead caused by the inverse transformation operation is also reduced.

[0085] Example Claims

[0086] A. A method, comprising: decoding an image of an image data set to generate an image data set in a frequency-domain representation; performing a resizing operation on the image data set in the frequency-domain representation based on a resizing factor; performing a reshaping operation on the image data set in the frequency-domain representation based on a reshaping factor; and performing a cropping operation on the image data set in the frequency-domain representation.

[0087] B. The method according to paragraph A, wherein the image of the image data set is converted from a raw image format into a frequency-domain representation and encoded into a compressed format before image decoding.

[0088] C. The method according to paragraph B, wherein an inverse transformation from the frequency-domain representation to the spatial-domain representation is not performed on the image of the image data set.

[0089] D. The method according to paragraph A, wherein no enhancement operation is performed on the image data set in the frequency-domain representation before performing the resizing operation on the image data set.

[0090] E. The method according to paragraph A, further comprising: performing an enhancement operation on the image data set in the frequency-domain representation before performing the resizing operation on the image data set.

[0091] F. The method according to paragraph A, wherein the resizing factor includes a vertical resizing factor and a horizontal resizing factor.

[0092] G. The method according to paragraph A, wherein the resizing operation further comprises: padding adjacent sides of the image according to the ratio of the resizing factor, or padding four sides of the image according to the ratio of the resizing factor.

[0093] H. The method as described in paragraph A, wherein the reshaping operation includes: reshaping an image by dividing the length and width of the image by a reshaping factor and multiplying the channel size of the image by the reshaping factor.

[0094] I. The method as described in paragraph A, further comprising: performing an enhancement operation on the image dataset in the frequency domain representation.

[0095] J. The method as described in paragraph I, wherein performing the enhancement operation includes: reshaping the images of the image dataset into a tensor representation and performing the enhancement operation on the tensor representation.

[0096] K. The method as described in paragraph I, wherein the enhancement operation is performed on the coefficients of the images of the image dataset in the frequency domain representation.

[0097] L. The method as described in paragraph A, wherein a cropping operation is performed on the four sides of the images of the image dataset in the frequency domain representation.

[0098] M. The method as described in paragraph A, wherein a cropping operation is performed on the upper edge and the left edge of the images of the image dataset in the frequency domain representation.

[0099] N. The method as described in paragraph A, wherein the image dataset in the frequency domain representation is input into a learning model for training calculations after performing the enhancement operation on the image dataset.

[0100] O. The method as described in paragraph A, wherein after performing the cropping operation on the image dataset, the image dataset in the frequency domain representation is input into a learning model for inference calculations.

[0101] P. The method as described in paragraph O, wherein the image dataset bypasses the first layer of the learning model and is input into the learning model.

[0102] Q. A system, comprising: one or more processors; and a memory communicatively coupled to the one or more processors, the memory storing computer-executable modules executable by the one or more processors, the executable modules performing associated operations when executed by the one or more processors, the computer-executable modules including:

[0103] A decoding module configured to decode the images of the image dataset to generate an image dataset in the frequency domain representation;

[0104] A resizing module configured to perform a resizing operation on the image dataset in the frequency domain representation based on a resizing factor;

[0105] A reshaping module configured to perform a reshaping operation on the image dataset in the frequency domain representation based on a reshaping factor;

[0106] A cropping module, configured to perform a cropping operation on an image dataset in a frequency domain representation.

[0107] R. The system as described in paragraph Q, wherein the images of the image dataset are converted from the original image format to a frequency domain representation and encoded into a compressed format before image decoding.

[0108] S. The system as described in paragraph R, wherein the decoding module is configured not to perform an inverse transformation on the images of the image dataset from the frequency domain representation to the spatial domain representation.

[0109] T. The system as described in paragraph Q, wherein the enhancement module is configured not to perform an enhancement operation on the image dataset in the frequency domain representation before the resizing module performs a resizing operation on the image dataset.

[0110] U. The system as described in paragraph Q, further comprising: a selective enhancement module, configured to perform an enhancement operation on the image dataset in the frequency domain representation before the resizing module performs a resizing operation on the image dataset.

[0111] V. The system as described in paragraph Q, wherein the resizing factor includes a vertical resizing factor and a horizontal resizing factor.

[0112] W. The system as described in paragraph Q, wherein the resizing module is further configured to pad the adjacent sides of the image according to the ratio of the resizing factor, or pad the four sides of the image according to the ratio of the resizing factor.

[0113] X. The system as described in paragraph Q, wherein the reshaping module is configured to reshape an image by dividing the length and width of the image by a reshaping factor and multiplying the channel dimension of the image by the reshaping factor.

[0114] Y. The system as described in paragraph Q, further comprising an enhancement module, configured to perform an enhancement operation on the image dataset in the frequency domain representation.

[0115] Z. The system as described in paragraph Y, wherein the enhancement module is configured to reshape the images of the image dataset into a tensor representation and perform an enhancement operation on the tensor representation.

[0116] AA. The system as described in paragraph Y, wherein the enhancement module is configured to perform an enhancement operation on the coefficients of the images of the image dataset in the frequency domain representation.

[0117] BB. The system as described in paragraph Q, wherein the cropping module is configured to perform a cropping operation on the four sides of the images of the image dataset in the frequency domain representation.

[0118] CC. The system as described in paragraph Q, wherein the cropping module is configured to perform a cropping operation on the upper edge and the left edge of the image in the image dataset represented in the frequency domain.

[0119] DD. The system as described in paragraph Q, further comprising: a model input module configured to input the image dataset represented in the frequency domain into a learning model for training calculation after the enhancement module performs an enhancement operation on the image dataset.

[0120] EE. The system as described in paragraph Q, further comprising a model input module configured to input the image dataset represented in the frequency domain into a learning model for inference calculation after the cropping module performs a cropping operation on the image dataset.

[0121] FF. The system as described in paragraph EE, wherein the model input module is further configured to input the image dataset into the learning model by bypassing the first layer of the learning model.

[0122] GG. A computer-readable storage medium storing computer-readable instructions executable by one or more processors, which, when executed by one or more processors, cause the one or more processors to perform the following operations, the operations including: decoding an image of an image dataset to generate an image dataset represented in the frequency domain; performing a resizing operation on the image dataset represented in the frequency domain based on a resizing factor; performing a reshaping operation on the image dataset represented in the frequency domain based on a reshaping factor; and performing a cropping operation on the image dataset represented in the frequency domain.

[0123] HH. The computer-readable storage medium as described in paragraph GG, wherein the image of the image dataset is converted from an original image format to a frequency-domain representation and encoded in a compressed format before image decoding.

[0124] II. The computer-readable storage medium as described in paragraph GG, wherein an inverse transform from the frequency-domain representation to the spatial-domain representation is not performed on the image of the image dataset.

[0125] JJ. The computer-readable storage medium as described in paragraph GG, wherein an enhancement operation is not performed on the image dataset represented in the frequency domain before performing the resizing operation on the image dataset.

[0126] KK. The computer-readable storage medium as described in paragraph GG, wherein the operations further include performing an enhancement operation on the image dataset represented in the frequency domain before performing the resizing operation on the image dataset.

[0127] LL. The computer-readable storage medium as described in paragraph GG, wherein the resizing factor includes a vertical resizing factor and a horizontal resizing factor.

[0128] MM. A computer-readable storage medium as described in paragraph GG, wherein the resizing operation further includes: filling adjacent sides of the image according to the ratio of the resizing factor, or filling four sides of the image according to the ratio of the resizing factor.

[0129] NN. A computer-readable storage medium as described in paragraph GG, wherein the reshaping operation includes: reshaping the image by dividing the length and width of an image by a reshaping factor and multiplying the channel size of the image by the reshaping factor.

[0130] OO. A computer-readable storage medium as described in paragraph GG, further including: performing an enhancement operation on the image dataset in the frequency domain representation.

[0131] PP. A computer-readable storage medium as described in paragraph OO, wherein performing the enhancement operation includes: reshaping the images of the image dataset into a tensor representation and performing the enhancement operation on the tensor representation.

[0132] QQ. A computer-readable storage medium as described in paragraph OO, wherein the enhancement operation is performed on the coefficients of the images of the image dataset in the frequency domain representation.

[0133] RR. A computer-readable storage medium as described in paragraph GG, wherein a cropping operation is performed on four sides of the images of the image dataset in the frequency domain representation.

[0134] SS. A computer-readable storage medium as described in paragraph GG, wherein a cropping operation is performed on the upper edge and the left edge of the images of the image dataset in the frequency domain representation.

[0135] TT. A computer-readable storage medium as described in paragraph GG, wherein the operations further include: after performing the enhancement operation on the image dataset, inputting the image dataset in the frequency domain representation into a learning model for training calculation.

[0136] UU. A computer-readable storage medium as described in paragraph GG, wherein the operations further include: after performing the cropping operation on the image dataset, inputting the image dataset in the frequency domain representation into a learning model for inference calculation.

[0137] VV. A computer-readable storage medium as described in paragraph UU, wherein the model input module is further configured to input the image dataset into the learning model by bypassing the first layer of the learning model.

[0138] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. On the contrary, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. A method, comprising: Decoding images of an image dataset to generate an image dataset in a frequency domain representation; Performing a resizing operation on the image dataset in the frequency domain representation based on a resizing factor, where the resizing factor refers to the adjustment ratio of the height and width of each component in the image dataset in the frequency domain representation, so that each image in the image dataset has a unified dimension; Performing a reshaping operation on the image dataset in the frequency domain representation based on a reshaping factor, where the reshaping factor refers to the adjustment ratio of the height, width, and channel size of each component in the image dataset in the frequency domain representation, so that all image data of the same frequency in each component of the image are aggregated in the same channel; and Performing a cropping operation on the image dataset in the frequency domain representation.

2. The method according to claim 1, wherein The resizing operation further includes: padding adjacent sides of an image according to the ratio of the resizing factor, or padding four sides of the image according to the ratio of the resizing factor.

3. The method according to claim 1, wherein, The reshaping module includes: reshaping an image by dividing the length and width of the image by the reshaping factor and multiplying the channel size of the image by the reshaping factor.

4. The method according to claim 1 further comprises: Performing an enhancement operation on the image dataset in the frequency domain representation.

5. The method according to claim 4, wherein, Performing an enhancement operation on the coefficients of the images in the image dataset in the frequency domain representation.

6. The method according to claim 1, wherein Performing a cropping operation on the upper edge and left edge of the images in the image dataset in the frequency domain representation.

7. The method according to claim 1, wherein Performing a cropping operation on the four sides of the images in the image dataset in the frequency domain representation.

8. A system, comprising: One or more processors; And a memory communicatively coupled to the one or more processors, the memory storing computer-executable modules executable by the one or more processors, the computer-executable modules performing associated operations when executed by the one or more processors, the computer-executable modules including: A decoding module configured to decode images of an image dataset to generate an image dataset in a frequency domain representation; A resizing module configured to perform a resizing operation on the image dataset in the frequency domain representation based on a resizing factor, where the resizing factor refers to the adjustment ratio of the height and width of each component in the image dataset in the frequency domain representation, so that each image in the image dataset has a unified dimension; A reshaping module configured to perform a reshaping operation on the image dataset in the frequency domain representation based on a reshaping factor, where the reshaping factor refers to the adjustment ratio of the height, width, and channel size of each component in the image dataset in the frequency domain representation, so that all image data of the same frequency in each component of the image are aggregated in the same channel; A cropping module configured to perform a cropping operation on the image dataset in the frequency domain representation.

9. The system according to claim 8, wherein, The resizing module is further configured to pad adjacent sides of an image according to the ratio of the resizing factor, or pad four sides of the image according to the ratio of the resizing factor.

10. The system according to claim 8, wherein, The reshaping module is configured to reshape an image by dividing the length and width of the image by the reshaping factor and multiplying the channel size of the image by the reshaping factor.

11. The system according to claim 8, further comprising: An enhancement module for performing an enhancement operation on the image dataset in the frequency domain representation.

12. The system according to claim 11, wherein the enhancement module is further configured to: perform an enhancement operation on the coefficients of the images in the image dataset represented in the frequency domain.

13. The system according to claim 8, wherein, The cropping module is configured to perform a cropping operation on the upper edge and the left edge of the images in the image dataset represented in the frequency domain.

14. The system according to claim 8, wherein, The cropping module is configured to perform a cropping operation on the four sides of the images in the image dataset represented in the frequency domain.

15. A computer-readable storage medium storing computer-readable instructions executable by one or more processors, which, when executed by the one or more processors, cause the one or more processors to perform the following operations, the operations including: Decoding an image in an image dataset to generate an image dataset represented in the frequency domain; Performing a resizing operation on the image dataset represented in the frequency domain based on a resizing factor, where the resizing factor refers to the adjustment ratio of the height and width of each component in the image dataset represented in the frequency domain, so that each image in the image dataset has a unified dimension; Performing a reshaping operation on the image dataset represented in the frequency domain based on a reshaping factor, where the reshaping factor refers to the adjustment ratio of the height, width, and channel size of each component in the image dataset represented in the frequency domain, so that all the image data of the same frequency in each component of the image are aggregated in the same channel; and Performing a cropping operation on the image dataset represented in the frequency domain.

16. The computer-readable storage medium according to claim 15, wherein, The resizing operation further includes: padding the adjacent sides of the image according to the ratio of the resizing factor, or padding the four sides of the image according to the ratio of the resizing factor.

17. The computer-readable storage medium according to claim 15, wherein, The reshaping operation includes: reshaping an image by dividing the length and width of the image by the reshaping factor and multiplying the channel size of the image by the reshaping factor.

18. The computer-readable storage medium according to claim 17, further comprising: Performing an enhancement operation on the image dataset represented in the frequency domain.

19. The computer-readable storage medium according to claim 15, wherein, Performing a cropping operation on the upper edge and the left edge of the images in the image dataset represented in the frequency domain.

20. The computer-readable storage medium according to claim 15, wherein, Performing a cropping operation on the four sides of the images in the image dataset represented in the frequency domain.

Images