Image data processing apparatus and methods, electronic equipment

Abstract

This application discloses an image data processing apparatus and method, and an electronic device; wherein the apparatus includes a first data enhancement circuit configured to perform a first image enhancement process on an input image to generate a first enhanced image; a second data enhancement circuit configured to perform a second image enhancement process on the input image to generate a second enhanced image; wherein the second image enhancement process is different from the first image enhancement process and is executed in parallel with the first image enhancement process; a preprocessing circuit configured to convert the first enhanced image and the second enhanced image into a first data format and a second data format, respectively; and a data caching and distribution circuit configured to distribute image data in the first data format and image data in the second data format to a processor; wherein the first data format and the second data format are data formats that the processor can process.

Description

Technical Field

[0001] This application relates to electronic technology, including but not limited to image data processing apparatus and methods, and electronic devices. Background Technology

[0002] Training, a term in the field of artificial intelligence, refers to providing a model (such as a neural network or other models) with sufficient training data, usually called a dataset. The model then "learns" specific features from this training data; that is, it uses this training data to train the model's parameters, enabling the model to perform a specified function. If the amount of input data is insufficient, the model may fail to learn the specified features or may learn incorrect features, resulting in a trained model that cannot meet functional requirements. Therefore, when the amount of training data is insufficient, data augmentation is usually performed on the existing training data to improve the model's learning accuracy. However, current data augmentation methods are relatively inefficient. Summary of the Invention

[0003] In view of this, the image data processing apparatus, method, and electronic device provided in this application can improve the processing efficiency of data augmentation. The image data processing apparatus, method, and electronic device provided in this application are implemented as follows:

[0004] The image data processing apparatus provided in this application includes: a first data enhancement circuit configured to perform a first image enhancement process on an input image to generate a first enhanced image; a second data enhancement circuit configured to perform a second image enhancement process on the input image to generate a second enhanced image; wherein the second image enhancement process is different from the first image enhancement process and is executed in parallel with the first image enhancement process; a preprocessing circuit configured to convert the first enhanced image and the second enhanced image into a first data format and a second data format, respectively; and a data buffer and distribution circuit configured to distribute image data in the first data format and image data in the second data format to a processor; wherein the first data format and the second data format are data formats that the processor can process.

[0005] The electronic device provided in this application includes the image data processing apparatus described in any embodiment of this application.

[0006] In this application, the image data processing apparatus includes at least two different data enhancement circuits that perform image enhancement processing on the same input image in parallel. Thus, on the one hand, since the image enhancement processing of the same input image is completed by multiple parallel-operating hardware circuits, it is unnecessary to repeatedly read the same input image from the storage device to complete different data enhancement operations on the same image, thereby improving the overall efficiency of data enhancement. On the other hand, implementing data enhancement of the input image by hardware circuits increases the data enhancement speed by tens or even hundreds of times compared to software implementation schemes, thereby meeting the real-time requirements of various scenarios. Attached Figure Description

[0007] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without creative effort. New image data processing devices can be recombined based on the various drawings; that is, the various circuit modules can be arbitrarily combined to obtain new image data processing devices.

[0008] Figure 1 A schematic diagram of the image data processing apparatus provided in this application;

[0009] Figure 2 Another structural schematic diagram of the image data processing apparatus provided in this application;

[0010] Figure 3 Another schematic diagram of the image data processing apparatus provided in this application;

[0011] Figure 4 Another schematic diagram of the image data processing apparatus provided in this application;

[0012] Figure 5 Another structural schematic diagram of the image data processing apparatus provided in this application;

[0013] Figure 6 Another schematic diagram of the image data processing apparatus provided in this application;

[0014] Figure 7 Another schematic diagram of the image data processing apparatus provided in this application;

[0015] Figure 8 Another structural schematic diagram of the image data processing apparatus provided in this application;

[0016] Figure 9Another schematic diagram of the image data processing apparatus provided in this application;

[0017] Figure 10 This is a schematic diagram of the result after data augmentation of an image.

[0018] Figure 11 A schematic diagram illustrating the implementation process of an image data enhancement method;

[0019] Figure 12 Another schematic diagram of the image data processing apparatus provided in this application;

[0020] Figure 13 A schematic diagram of an electronic device provided in this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the specific technical solutions of this application will be described in further detail below with reference to the accompanying drawings. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0023] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0024] It should be noted that the terms "first, second, third" used in this application do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the application described herein can be implemented in an order other than that illustrated or described herein.

[0025] This application provides an image data processing apparatus. Figure 1 A schematic diagram of the image data processing apparatus provided in this application is shown below. Figure 1 As shown, the image data processing device 1 includes a first data enhancement circuit 11, a second data enhancement circuit 12, a preprocessing circuit 13, and a data buffer and distribution circuit 14; wherein,

[0026] The first data enhancement circuit 11 is configured to perform a first image enhancement process on the input image to generate a first enhanced image;

[0027] The second data enhancement circuit 12 is configured to perform a second image enhancement process on the input image to generate a second enhanced image; wherein the second image enhancement process is different from the first image enhancement process and is performed in parallel with the first image enhancement process;

[0028] Preprocessing circuit 13 is configured to convert the first enhanced image and the second enhanced image into a first data format and a second data format, respectively; and

[0029] The data caching and distribution circuit 14 is configured to distribute image data in the first data format and image data in the second data format to the processor 15; wherein the first data format and the second data format are data formats that the processor 15 can process.

[0030] In this embodiment, the image data processing apparatus 1 includes at least two different data enhancement circuits that perform image enhancement processing on the same input image in parallel. Thus, on the one hand, since the image enhancement processing of the same input image is completed by multiple parallel-operating hardware circuits, it is unnecessary to repeatedly read the same input image from the storage device to complete different data enhancement operations on the same image, thereby improving the overall efficiency of data enhancement. On the other hand, implementing data enhancement of the input image by hardware circuits increases the data enhancement speed by tens or even hundreds of times compared to software implementation schemes, thereby meeting the real-time requirements of various scenarios.

[0031] For example, the image data processing device 1 can meet the real-time requirements of the training phase of various types of Artificial Intelligence (AI) models. That is, by using this image data processing device, it is not necessary to perform model training after completing image enhancement processing on all images. In other words, by using the image data processing device of this application, image enhancement processing can be performed on the input image while simultaneously using the output enhancement results to train the model parameters of the AI ​​model to be trained.

[0032] Regarding data enhancement circuits:

[0033] It should be noted that this application does not limit the number of data enhancement circuits included in the image data processing apparatus 1. For example, the image data processing apparatus 1 may include 32 or 64 data enhancement circuits with different functions connected in parallel, each used to perform different enhancement processing on the same input image. In summary, the image data processing apparatus 1 includes at least two data enhancement circuits with different functions that can operate in parallel, and the different data enhancement circuits are connected in parallel with each other, thereby enabling the image data processing apparatus to complete different data transformations of the first sample in a very short time. Understandably, different circuit functions mean different internal structures.

[0034] For example, an image data processing device 1, including at least two different data enhancement circuits, can respectively perform image enhancement operations on the input image, such as rotation, flipping, translation, affine transformation, edge enhancement, color replacement, Gaussian blur, adding noise, adding mosaic, scaling, erasing, and filling. That is, one data enhancement circuit performs rotation of the input image, another data enhancement circuit performs flipping of the input image, yet another data enhancement circuit performs translation of the input image, and so on, which will not be elaborated here.

[0035] Regarding preprocessing circuit 13:

[0036] Understandably, in this application, the preprocessing operations on the first enhanced image and the second enhanced image are implemented using hardware circuitry, rather than by the CPU or graphics processing unit (GPU) executing the preprocessing program. This improves the speed of data preprocessing and thus further enhances the overall efficiency of data augmentation.

[0037] In some embodiments, every two or more data enhancement circuits may share a preprocessing circuit 13. Exemplarily, the output of every two or more data enhancement circuits is connected to the input of a preprocessing circuit, so that the preprocessing circuit preprocesses the enhanced images output by each of the data enhancement circuits. When preprocessing multiple received enhanced images, the preprocessing circuit may perform preprocessing operations on these images in chronological order of reception.

[0038] Furthermore, in some embodiments, such as Figure 2 As shown, the preprocessing circuit 13 includes: a first preprocessing sub-circuit 131 and a second preprocessing sub-circuit 132; wherein the first preprocessing sub-circuit 131 and the second preprocessing sub-circuit 132 are connected in parallel; the first preprocessing sub-circuit is configured to convert the first enhanced image into the first data format, and the second preprocessing sub-circuit is configured to convert the second enhanced image into the second data format.

[0039] Therefore, each data enhancement circuit's output corresponds to a preprocessing sub-circuit; the number of preprocessing sub-circuits is equal to the number of data enhancement circuits. Thus, after the data enhancement circuit performs image enhancement processing on the input image, the corresponding preprocessing sub-circuits can immediately perform preprocessing operations on the image data generated at the front end without queuing, thereby further improving the overall processing efficiency of data enhancement and enhancing the real-time performance of the image data processing device.

[0040] In this application, each preprocessing subcircuit includes at least one preprocessing function. For example, a preprocessing subcircuit is configured to perform at least one of the following operations: scaling, cropping, integer-to-floating-point conversion, floating-point normalization, etc. In some embodiments, multiple different preprocessing operations need to be performed on the enhanced image to meet the input requirements of the AI ​​model to be trained.

[0041] However, if these preprocessing functions are implemented by software code executed by the CPU or GPU, the result needs to be written to the storage device after each preprocessing operation, and then read from the storage device before the next preprocessing operation. It can be seen that there are many read and write operations, and the overall efficiency of preprocessing is low.

[0042] In view of this, in some embodiments, the preprocessing sub-circuit includes at least two preprocessing units connected in series. Thus, it is unnecessary to write the result to a storage device after each preprocessing operation, and therefore, it is unnecessary to read the result of the previous preprocessing operation from the storage device before performing the next preprocessing operation. Instead, the processing result of the previous preprocessing unit is directly output to the next preprocessing unit, which then continues its corresponding preprocessing operation based on the received result, thereby effectively improving the overall efficiency of preprocessing. Understandably, these preprocessing units have different functions, which means that the circuit structures of each preprocessing unit with a different function are different.

[0043] Furthermore, in some embodiments, the image data processing device 1 also includes a controller, which can turn on and off corresponding preprocessing units as needed according to pre-configured preprocessing requirements. For example, a certain preprocessing sub-circuit includes a preprocessing unit 1 for scaling, a preprocessing unit 2 for cropping, a preprocessing unit 3 for converting integers to floating-point numbers, and a preprocessing unit 4 for normalizing floating-point numbers. If the current AI model to be trained requires the input data to be integers, then when the data format of the enhanced image output by the data augmentation circuit is integers, the controller can turn off preprocessing units 3 and 4, and turn on preprocessing units 1 and 2. In this way, the accuracy of the data input to the AI ​​model to be trained can be ensured, and the training requirements of AI models with different input requirements can be supported, thereby improving the applicability of the image data processing device 1.

[0044] In some embodiments, such as Figure 3 As shown, the image data processing apparatus 1 further includes a third data enhancement circuit 31, which is configured to generate a third enhanced image based on the first enhanced image and / or the second enhanced image; correspondingly, the preprocessing circuit 13 is configured to convert the third enhanced image into a first data format or a second data format.

[0045] Furthermore, in some embodiments, such as Figure 4 As shown, the preprocessing circuit 13 further includes a third preprocessing sub-circuit 41, configured to convert the third enhanced image into a first data format or a second data format.

[0046] It should be noted that, for ease of description, the data augmentation circuit group configured to perform enhancement processing on the input image is referred to as the first-level enhancement circuit layer, and the data augmentation circuit group configured to perform image enhancement processing on the enhanced image output by the first-level enhancement circuit is referred to as the second-level enhancement circuit layer.

[0047] In this application, the number of enhancement circuit layers included in the image data processing apparatus is not limited, but it must include at least one data enhancement circuit. The next-level enhancement circuit layer is configured to further enhance the enhanced image output from the previous-level enhancement circuit layer, thereby obtaining different enhanced images. Furthermore, the number of previous-level data enhancement circuits corresponding to a particular data enhancement circuit in a next-level enhancement circuit layer is not limited; it can be one or more.

[0048] For example, the third data enhancement circuit 31 is configured to generate a third enhanced image based on at least one enhanced image output by a data enhancement circuit.

[0049] In an embodiment where the third data enhancement circuit 31 is configured to generate a third enhanced image based on the enhanced image of one output, its function is different from that of the data enhancement circuit corresponding to the front end, thus obtaining an enhanced image that is different from any other enhanced image.

[0050] For example, such as Figure 5 As shown, the first-level enhancement circuit layer includes data enhancement circuit 501, data enhancement circuit 502, ..., data enhancement circuit 50N, and the second-level enhancement circuit layer includes data enhancement circuit 511, data enhancement circuit 512, ..., data enhancement circuit 51N; wherein, each data enhancement circuit in the first-level enhancement circuit layer has two output terminals, one output terminal is connected to a certain data enhancement circuit in the next level, and the other output terminal is connected to the input terminal of the preprocessing circuit 13.

[0051] For example, the output of data enhancement circuit 501 is connected to the input of data enhancement circuit 511, and the output of data enhancement circuit 501 is also connected to the input of preprocessing circuit 13. Data enhancement circuit 501 has an image flipping function, and data enhancement circuit 511 has a color changing function. Thus, when an image is input into data enhancement circuit 501, a flipped image is obtained; this flipped image is then input into data enhancement circuit 511, resulting in a color-changed image of the flipped image. In other words, the image input to the front end undergoes flipping and color transformation sequentially. Similarly, data enhancement circuit 502 has a Gaussian blur function, and the connected data enhancement circuit 512 has an edge enhancement function; furthermore, data enhancement circuit 50N has a rotation function, and the connected data enhancement circuit 51N has a translation function. In short, the functions of the series-connected data enhancement circuits are different, but each stage may include the same data enhancement circuit; only the series-connected data enhancement circuits differ.

[0052] In an embodiment where the third data enhancement circuit 31 is configured to generate a third enhanced image based on at least two output enhanced images, the method for generating the third enhanced image based on multiple enhanced images is not limited and can be varied. For example, the third data enhancement circuit 31 may directly fuse the multiple input enhanced images to obtain the third enhanced image; or, the third data enhancement circuit 31 may perform image enhancement processing on each input enhanced image again before fusing them to obtain the third enhanced image.

[0053] Regarding data caching and distribution circuitry 14:

[0054] In some embodiments, the data buffer and distribution circuit 14 is further configured to control the timing of distributing image data in the first data format and / or image data in the second data format based on a distribution timing signal.

[0055] Furthermore, in some embodiments, such as Figure 6 As shown, the image data processing device 1 further includes a timing circuit 61; the timing circuit 61 is configured to provide a first distribution timing signal and a second distribution timing signal; correspondingly, the data buffer and distribution circuit 14 is configured to: distribute image data of the first data format to the processor 15 in response to the first distribution timing signal; and distribute image data of the second data format to the processor 15 in response to the second distribution timing signal; wherein there is a timing difference between the second distribution timing signal and the first distribution timing signal.

[0056] For example, the timing difference corresponds to (e.g., greater than or equal to) the time required for processor 15 to process image data of a first data format and the time required for it to process image data of a second data format; or, the timing difference is the delay in processing an image of a certain format; or, the timing difference is the time difference between two image frames, and so on.

[0057] Regarding processor 15:

[0058] In this application, it is not limited whether the image data processing apparatus 1 includes a processor 15. In some embodiments, such as Figure 7 As shown, the image data processing device 1 further includes a processor 15; the processor 15 is configured to perform training of a neural network model based on image data in the first data format and image data in the second data format.

[0059] Of course, when the image data processing device 1 includes two or more data enhancement circuits, the processor 15 is also configured to perform training of a neural network model based on the image data corresponding to each data enhancement circuit.

[0060] In some embodiments, such as Figure 8 As shown, the image data processing device 1 further includes a data decoding circuit 81 and a data format conversion circuit 82; wherein, the data decoding circuit 81 is configured to decode the sample image; the data format conversion circuit 82 is configured to convert the data format of the image data output by the data decoding circuit 81 into a data format that can be processed by the processor 15 to obtain the input image.

[0061] Understandably, the data decoding function is implemented by hardware circuitry, which can improve decoding efficiency by tens of times compared to implementation by CPU or GPU executing decoding programs (i.e., software implementation). In some embodiments, the data decoding circuit 81 can support the decoding of image data in multiple formats. For example, the data decoding circuit 81 supports the decoding of JPEG and PNG format images.

[0062] Similarly, the data format conversion circuit 82 is implemented by hardware circuitry, which significantly improves the format conversion efficiency compared to the CPU or GPU executing the format conversion program.

[0063] For example, the data format conversion circuit 82 is configured to convert YUV format image data to RGB format image data. In this embodiment, the data format conversion circuit 82 supports multiple format to corresponding format image data conversion. The image data processing device 1 can control the data format conversion circuit 82 to operate in the corresponding mode according to the format conversion requirements and the data format of the image data output by the data decoding circuit 81. For example, the data format conversion circuit 82 includes a first format to second format conversion sub-circuit 1, a third format to second format conversion sub-circuit 2, a fifth format to sixth format conversion sub-circuit 3, a seventh format to sixth format conversion sub-circuit 4, etc.; assuming the format conversion requirement is to convert to the second format and the data format of the image data output by the data decoding circuit 81 is the third format, then the conversion sub-circuit 2 can be turned on; if the data format of the image data output by the data decoding circuit 81 is the first format, then the conversion sub-circuit 1 can be turned on. In short, the image data processing device can flexibly select the conversion sub-circuit that can complete the task according to the format conversion requirements and the data format of the actual input image data.

[0064] Understandably, considering that different sample images may come from different sources, the data format of the corresponding decoded images may also be different. Therefore, to adapt to this situation, in some embodiments, such as... Figure 9 As shown, the image data processing device 1 further includes a first set of switching elements 91 and a second set of switching elements 92; wherein,

[0065] The first set of switching elements 91 is configured to control the on / off state between the output of the data decoding circuit 81 and the input of the data format conversion circuit 82 based on whether the data format of the image data output by the data decoding circuit 81 is processable by the processor 15; and,

[0066] The second set of switching elements 92 is configured to control the on / off state between the output of the data decoding circuit 81 and the input of the corresponding data enhancement circuit based on whether the data format of the image data output by the data decoding circuit 81 is processable by the processor 15.

[0067] For example, the first set of switching elements 91 is configured to disconnect the circuit between the output of the data decoding circuit 81 and the input of the conversion circuit 82 if the data format of the image data output by the data decoding circuit 81 is a format that can be processed by the processor 15; correspondingly, the second set of switching elements 92 is configured to connect the circuit between the output of the data decoding circuit 81 and the input of the corresponding data enhancement circuit.

[0068] The first set of switching elements 91 is configured to connect the circuit between the output terminal of the data decoding circuit 81 and the input terminal of the conversion circuit 82 if the data format of the image data output by the data decoding circuit 81 is not a format that the processor 15 can process; correspondingly, the second set of switching elements 92 is configured to disconnect the circuit between the output terminal of the data decoding circuit 81 and the input terminal of the corresponding data enhancement circuit.

[0069] In this way, the image data processing device 1 can process both data that meets the format requirements after decoding and data that does not meet the format requirements after decoding. This allows for a variety of sample images in the sample image set. Sample images of different formats can be ultimately processed into a data format that the processor 15 can process, thereby reducing the format requirements of the acquired sample images and allowing for a variety of formats of sample images used to train neural network models, thus saving acquisition costs.

[0070] Of course, in some other embodiments, the data decoding circuit 81 and the data format conversion circuit 82 do not include the first set of switching elements 91 and the second set of switching elements 92, and the output terminal of the data decoding circuit 81 is connected to the input terminal of the data format conversion circuit 82.

[0071] In this application, it is not limited whether the image data processing apparatus 1 includes a storage device. In some embodiments, the image data processing apparatus 1 does not include a storage device configured to store image data in the first data format and / or the second data format. In other words, because the image data processing apparatus 1 is sufficient to meet the real-time requirements of the processor 15 in training the neural network model, it is not necessary to collect the data augmentation results of all sample images into a storage device and then read the model parameters for training the neural network model one by one from the storage device. Therefore, the image data processing apparatus 1 provided in this application can indirectly improve the training efficiency of the neural network model.

[0072] It should be noted that, in this application, any of the above circuits can be arbitrarily combined to obtain an image data processing device including at least two data augmentation circuits with different functions connected in parallel. Understandably, in this application, each stage of the entire data augmentation process, such as data decoding, data format conversion, data transformation, data preprocessing, and data caching and distribution, is implemented using hardware circuits. This ensures the real-time performance of the data augmentation processing, allowing the data caching and distribution circuit 14 to directly distribute the image data output by the preprocessing circuit 13 to the processor 15 without writing this data to a storage device, thereby achieving the goal of simultaneously performing data augmentation processing and using the data augmentation results for model training.

[0073] Image-based neural network models typically take images as input. Image data augmentation is relatively straightforward, such as... Figure 10 As shown, originally there was only one image of a hamster. By performing various operations on this image, such as rotation, flipping, translation, affine transformation, edge enhancement, color replacement, Gaussian blurring, adding noise, and adding mosaic, 64 images can be obtained. This process is called data augmentation. No matter how the image is transformed, the human eye can basically determine that it is a hamster. Based on these datasets, a neural network is trained, hoping that the neural network can learn, like the human eye, the intrinsic data features of "why a hamster is a hamster" from these images with interference.

[0074] In some embodiments, the image data augmentation process is implemented either by the CPU or the GPU, primarily using software methods and often offline. This means that after data augmentation, the newly generated data is written to the hard drive. The system only begins reading the augmented data from the hard drive and feeding it to the neural network when training begins. This is because the data augmentation process is purely software-based, relatively time-consuming, and cannot achieve real-time data augmentation and immediate delivery to the neural network. Figure 7 As shown in the figure, the implementation process of this software method is illustrated, and the decomposed steps are described below.

[0075] First, the image is decoded. The image format is, for example, PNG or JPEG, both of which are encoded and compressed formats that require a specific decoder to obtain the image. This image decoding process can be performed directly by the CPU or by the GPU. The decoded image is then in YUV format.

[0076] Next, the YUV format image is converted. Taking the input of the artificial intelligence model as RGB format data as an example, the YUV format image is converted to RGB format.

[0077] Next, data augmentation is performed on the RGB format image. This step can be implemented using a GPU or CPU, and essentially involves performing various image transformations. For example... Figure 11 As shown, when implementing data augmentation via GPU or CPU, i.e., in software, only one type of data augmentation is performed at a time; not all data augmentations are completed in one step.

[0078] Finally, the data augmented is preprocessed to make it usable directly by the neural network. This step typically includes scaling, cropping, integer-to-floating-point conversion, and floating-point normalization. The preprocessed data is then written to storage devices, initially stored on the hard drive for later use. It's important to note that because this step involves numerous preprocessing operations, if implemented using a CPU or GPU, each operation requires writing the result to storage, and then reading it back from storage for the next operation. This frequent writing and reading of storage results in extremely low efficiency.

[0079] Each of the above steps yields only one image transformation, so it needs to be repeated to obtain the various data augmentation data required for the training process.

[0080] As can be seen, the main drawback of the above-described software-implemented embodiments is that they are slow, as software processing is much slower than hardware processing; moreover, they cannot perform real-time data augmentation and can usually only be done offline.

[0081] In view of this, the following embodiments have two focuses: first, to implement the data augmentation process using hardware circuits as much as possible; second, to ensure that the output of the data augmentation circuit can be directly sent to the neural network processor.

[0082] like Figure 12 As shown, the image data processing device 1 includes: a hardware-based image decoder (i.e., data decoding circuit 81), a data format conversion circuit 82, 32 parallel data enhancement circuits with different functions, a preprocessing sub-circuit for neural network model data, and a data buffering and distribution circuit 14; wherein,

[0083] Step 1: Configure the hardware image decoder, which supports JPEG and PNG decoding. The hardware image decoder offers processing speeds tens of times faster than software decoding.

[0084] Step 2: The output of the hardware image decoder is a YUV image, which is fed to the data format conversion circuit 82. This circuit is used to convert the YUV format image into an RGB format image.

[0085] Step 3: Set up 32 data augmentation circuits with different functions to implement different types of data augmentation;

[0086] Step 4: Each data augmentation circuit is paired with a preprocessing sub-circuit at its output to convert the image data into a data type and data arrangement that can be directly used by the neural network model.

[0087] Step 5: The outputs of all preprocessing sub-circuits are fed to a data buffer and distribution circuit 14, which is responsible for distributing the 32 data streams to the processor 15.

[0088] Step 6: The processor 15 performs deep learning on the neural network model to be trained based on the real-time input data.

[0089] In this application, (1) each step of the data augmentation process is made into a hardware circuit, which can greatly improve the processing speed; (2) for the processor 15, real-time data augmentation can be performed starting from the input JPEG or PNG image, shortening the training time and accelerating the deep learning process.

[0090] In some embodiments, the image decoding section may support more encoding formats, such as WebP.

[0091] In some embodiments, the image format conversion section may support more formats.

[0092] In some embodiments, the number of data enhancement circuits can be set to more, not limited to 32.

[0093] This application also provides an electronic device, such as... Figure 13 As shown, the electronic device 130 can be implemented in various types of devices, including the image data processing device 1, or devices comprising an image data processing device consisting of at least two data enhancement circuits with different functions connected in parallel, composed of any combination of circuits. Various types of electronic devices may include, for example, computer devices, laptops, tablets, mobile phones, in-vehicle devices, or wearable devices. In short, this application does not limit the type of electronic device.

[0094] Based on the foregoing embodiments, this application provides an image data processing method, the method comprising: performing a first image enhancement process on an input image through a first data enhancement circuit 11 to generate a first enhanced image; performing a second image enhancement process on the input image through a second data enhancement circuit 12 to generate a second enhanced image; wherein the second image enhancement process is different from the first image enhancement process and is performed in parallel with the first image enhancement process; converting the first enhanced image and the second enhanced image into a first data format and a second data format respectively through a preprocessing circuit 13; and distributing the image data in the first data format and the image data in the second data format to a processor through a data caching and distribution circuit 14; wherein the first data format and the second data format are data formats that the processor can process.

[0095] The descriptions of the above method embodiments are similar to those of the above device embodiments, and have similar beneficial effects. For technical details not disclosed in the method embodiments of this application, please refer to the descriptions of the device embodiments of this application for understanding.

[0096] It should be understood that the terms "one embodiment," "an embodiment," "some embodiments," or "other embodiments" used throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," "in some embodiments," or "in other embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application. The above-described sequence numbers are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0097] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0098] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the circuit division is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0100] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0101] The apparatuses disclosed in the several apparatus embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0102] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0103] The features disclosed in the several method or apparatus embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or apparatus embodiments.

[0104] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image data processing apparatus, characterized in that, include: A first data enhancement circuit is configured to perform a first image enhancement process on an input image to generate a first enhanced image; A second data enhancement circuit is configured to perform a second image enhancement process on the input image to generate a second enhanced image; wherein the second image enhancement process is different from the first image enhancement process and is executed in parallel with the first image enhancement process; The preprocessing circuit is configured to convert the first enhanced image and the second enhanced image into a first data format and a second data format, respectively; and A data caching and distribution circuit is configured to distribute image data in the first data format and image data in the second data format to a processor; wherein the first data format and the second data format are data formats that the processor can process; In this case, every two or more data enhancement circuits share one of the preprocessing circuits; Alternatively, the preprocessing circuit may include: a first preprocessing sub-circuit and a second preprocessing sub-circuit; wherein the first preprocessing sub-circuit and the second preprocessing sub-circuit are connected in parallel; the first preprocessing sub-circuit is configured to convert the first enhanced image into the first data format, and the second preprocessing sub-circuit is configured to convert the second enhanced image into the second data format; the output of each data enhancement circuit corresponds to one preprocessing sub-circuit, and the number of preprocessing sub-circuits is equal to the number of data enhancement circuits.

2. The apparatus according to claim 1, characterized in that, The data buffer and distribution circuit is further configured to control the timing of distributing image data in the first data format and / or image data in the second data format based on a distribution timing signal.

3. The apparatus according to claim 1, characterized in that, The processor is configured to train a neural network model based on image data in the first data format and image data in the second data format.

4. The apparatus according to claim 1, characterized in that, Also includes: A third data enhancement circuit is configured to generate a third enhanced image based on the first enhanced image and / or the second enhanced image; The preprocessing circuit is further configured to convert the third enhanced image into the first data format or the second data format.

5. The apparatus according to claim 1, characterized in that, Also includes: The data decoding circuit is configured to decode the sample image; A data format conversion circuit is configured to convert the data format of the image data output by the data decoding circuit into a data format that the processor can process, thereby obtaining the input image.

6. The apparatus according to claim 5, characterized in that, Also includes: The first set of switching elements is configured to control the on / off state between the output terminal of the data decoding circuit and the input terminal of the data format conversion circuit based on whether the data format of the image data output by the data decoding circuit is processable by the processor; as well as, The second set of switching elements is configured to control the on / off state between the output terminal of the data decoding circuit and the corresponding input terminal of the data enhancement circuit based on whether the data format of the image data output by the data decoding circuit is processable by the processor.

7. The apparatus according to claim 2, characterized in that, Also includes: A timing circuit configured to provide a first timing signal and a second timing signal; The data caching and distribution circuit is configured as follows: In response to the first distribution timing signal, image data in the first data format is distributed to the processor; In response to the second distribution timing signal, image data in the second data format is distributed to the processor; There is a timing difference between the second distribution timing signal and the first distribution timing signal.

8. The apparatus according to any one of claims 1 to 7, characterized in that, The device does not include: A storage device configured to store image data in the first data format and / or the second data format.

9. An electronic device, characterized in that, The electronic device includes the image data processing apparatus according to any one of claims 1 to 8.

Images