Image data processing methods, apparatuses, electronic devices and storage media

Abstract

This disclosure discloses an image data processing method, apparatus, electronic device, and storage medium. The image data processing method includes: acquiring initial image data; converting the data type of the initial image data to a preset data type to obtain intermediate image data, wherein the preset data type is a data type supported by a neural network accelerator; and then using the neural network accelerator to perform linear operations on the intermediate image data to obtain target image data. This disclosure enables the neural network accelerator to process image data of multiple data types compatiblely, reducing the problem of abnormal image processing results caused by data type incompatibility. Furthermore, the fact that the target image data and the initial image data have the same data type ensures the consistency of data input and output.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision applications, and in particular to an image data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Neural network accelerators have multiply-accumulate (MAC) units to perform high-speed multiply-accumulate operations.

[0003] To improve the operational efficiency of multiply-accumulate units, neural network accelerators are typically designed to support operations on signed integer data (e.g., int8 format). However, image data is not signed integer data, which may lead to poor processing results when neural network accelerators process image data.

[0004] Therefore, when using neural network accelerators to process image data, there is an urgent need for an image data processing method that can be compatible with multiple data types. Summary of the Invention

[0005] To address the technical problem of neural network accelerators being unable to handle image data of various types, embodiments of this disclosure provide an image data processing method, apparatus, electronic device, and storage medium. Specifically, the following technical solutions are disclosed:

[0006] According to a first aspect of this disclosure, an embodiment of this disclosure provides an image data processing method, the method comprising the following steps: acquiring initial image data; converting the data type of the initial image data into a preset data type to obtain intermediate image data, wherein the preset data type is a data type supported by a neural network accelerator; performing linear operation processing on the intermediate image data using a neural network accelerator; and performing an inverse data type conversion on the linearly processed image data to obtain target image data, wherein the target image data and the initial image data have the same data type.

[0007] According to a second aspect of this disclosure, embodiments of this disclosure provide an image data processing apparatus applied to a neural network accelerator, the apparatus comprising:

[0008] The data acquisition module is used to acquire initial image data.

[0009] The conversion module is used to convert the data type of the initial image data into a preset data type to obtain intermediate image data; the preset data type is a data type supported by the neural network accelerator.

[0010] The computation and processing module is used to perform linear operations on the intermediate image data to obtain the target image data.

[0011] According to a third aspect of this disclosure, embodiments of this disclosure provide an electronic device comprising:

[0012] Memory, used to store processor-executable instructions;

[0013] A processor is configured to read executable instructions from memory and execute the instructions to implement the image data processing method of the first aspect described above.

[0014] According to a fourth aspect of this disclosure, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the image data processing method of the first aspect described above.

[0015] Embodiments of this disclosure provide an image data processing method, apparatus, electronic device, and storage medium. Based on acquiring initial image data, the data type of the initial image data is converted to a preset data type to obtain intermediate image data. The preset data type is a data type supported by a neural network accelerator. Then, the intermediate image data is processed using a neural network accelerator through linear operations to obtain target image data. This disclosure enables the neural network accelerator to be compatible with and process image data of multiple data types, reducing the problem of abnormal image processing results caused by data type incompatibility. Furthermore, the fact that the target image data and the initial image data have the same data type ensures the consistency of data input and output. Attached Figure Description

[0016] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1 This is a schematic diagram of the image output after directly processing unsigned data on a neural network accelerator;

[0018] Figure 2 This is a schematic diagram of the image output after processing signed data on a neural network accelerator.

[0019] Figure 3 This is a schematic diagram of an image processing scenario to which this disclosure applies;

[0020] Figure 4 This is a schematic flowchart of an image processing method provided in an exemplary embodiment of this disclosure;

[0021] Figure 5 This is a schematic flowchart of an image processing method provided in another exemplary embodiment of this disclosure;

[0022] Figure 6 This is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of the present disclosure;

[0023] Figure 7 This is a schematic diagram of the structure of an image processing apparatus provided in another exemplary embodiment of the present disclosure;

[0024] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0025] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0026] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0027] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0028] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0029] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0030] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0031] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0032] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0033] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0034] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0035] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0036] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0037] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0038] Application Overview

[0039] With the continuous development of science and technology, artificial intelligence is increasingly being applied in people's lives. Deep learning algorithms and related neural network accelerators are frequently used in artificial intelligence technology. These accelerators contain a large number of multiply-accumulate (MAC) units, enabling high-speed multiply-accumulate operations. To improve the efficiency of multiply-accumulate operations, some neural network accelerators typically only support fixed-point operations, for example, only supporting signed integer data input. However, to implement image processing functions (e.g., filtering) on ​​neural network accelerators, it is necessary for them to support the processing of unsigned integer image data. Therefore, there is an urgent need for an image data processing method that is compatible with multiple data types.

[0040] For example, traditional neural network accelerators only support processing signed data types and not unsigned data types. However, the feature maps input to neural network models are often unsigned data types, such as uint8, uint16, and uint32, without restriction. Therefore, traditional neural network accelerators will directly treat uint data types as int data types for computation, resulting in anomalies at the edges of the processed image, such as noise (white spots). Figure 1 As shown.

[0041] It's important to note that `int8` represents an 8-bit signed integer, occupying 1 byte with a value of 8 bits. The highest bit of this 8-bit binary number is 1, indicating a negative value. The value range of `int8` is [-128, 127]. Similarly, `int16` occupies 2 bytes with a value of 16 bits, and its value range is [-32768, 32767]. `int32` occupies 4 bytes with a value of 32 bits, and its value range is [-2147483648, 2147483647]. `int64` occupies 8 bytes with a value of 64 bits, and its value range is [-9223372036854775808, 9223372036854775807]. `uint8` is an 8-bit unsigned integer with a value range of [0, 255]. Similarly, uint16 is a 16-bit unsigned integer with a value range of [0, 65535], uint32 is a 32-bit unsigned integer with a value range of [0, 4294967295], and uint64 is a 64-bit unsigned integer with a value range of [0, 18446744073709551615].

[0042] For example, a neural network accelerator processes image data, which can be represented by matrix A and is of type uint8. The weights used in the filtering process are w, where...

[0043] , ,

[0044]

[0045] For a hardware chip capable of processing uint8 type data, the resulting image data after filtering is 161.25, retaining the integer part 161, which is represented in binary as 0b10100001. However, for a neural network accelerator without uint8 processing capabilities, this matrix A will be directly treated as an int8 matrix B and then Gaussian filtered to obtain the output image data. For example, treating matrix A as an int8 matrix B would be:

[0046]

[0047] Therefore, when using matrix B for filtering, the resulting image data is:

[0048]

[0049] The image data, when rounded to the nearest integer, yields -46, while the correct output is 161. Therefore, treating uint8 data directly as int8 results in a significant deviation in the output, which manifests as a problem in the image. Figure 1 and Figure 2 The differences between them. Figure 2 This is the image after the hardware chip performs filtering operations on the input unsigned number uint8. The image is generally clear and normal, similar to the one described above. The image output result. Figure 1 This is an image obtained by treating image data as a signed int8 data type and performing operations directly. This will result in anomalies on the image edges, similar to... The output result.

[0050] Embodiments of this disclosure provide a method for enabling a neural network accelerator to process image data compatible with multiple data types, so that the output image of the neural network accelerator has clear edges and a normal overall image.

[0051] Exemplary System

[0052] The technical solution disclosed herein is applied to a neural network accelerator. See also Figure 3This is a schematic diagram of an application scenario provided in this embodiment. The scenario includes: a display screen 10, a camera 20, and a neural network accelerator 30. The neural network accelerator 30 is connected to both the camera 20 and the display screen 10. Furthermore, the neural network accelerator 30 can be connected to both the camera 20 and the display screen 10 via an input / output (I / O) interface.

[0053] Camera 20 is used to capture external images, such as the surrounding environment, to obtain at least one frame of image or photograph.

[0054] It should be understood that the aforementioned camera 20 can also be other image acquisition devices, such as monocular cameras, binocular cameras, or camera sensors.

[0055] The neural network accelerator 30 is used to filter, analyze, classify and recognize external images acquired by the camera 20. It involves neural network model construction, matching, image processing and other processes, and displays the images output by the neural network accelerator 30 through the display screen 10.

[0056] It should be understood that the technical solutions disclosed herein can also be applied to other deep learning networks. That is, they can be applied to all scenarios of various applications as long as they enable the neural network accelerator 30 to support the processing of image data compatible with multiple data types (e.g., signed and unsigned data) and output clear and complete images.

[0057] This disclosure adds a conversion function from unit type data to int type data to the original neural network accelerator, so that the neural network accelerator can process both int type data and input unit type image data to obtain clear images without adding extra overhead.

[0058] Exemplary methods

[0059] Figure 4 This is a schematic flowchart of an image data processing method provided in an embodiment of this disclosure. This embodiment can be applied to electronic devices, such as... Figure 3 The neural network accelerator 30 shown is described in the following document. Figure 4 The method includes the following steps:

[0060] Step 302: Obtain initial image data.

[0061] The initial image data can be of either signed or unsigned integer type. Signed integer types can include int8, int16, int32, int64, etc., without limitation. Unsigned integer types can include uint8, uint16, uint32, uint64, etc., without limitation. Each pixel in the initial image data is a binary number.

[0062] Specifically, if the electronic device used in this embodiment is a neural network accelerator, an image acquisition device can be used to capture images of the target object. The image acquisition device then compiles and transmits the captured initial image data to the neural network accelerator, enabling the neural network accelerator to acquire the initial image data. The image acquisition device can be a camera, a monocular camera, a binocular camera, or a camera sensor, etc.

[0063] Step 304: Convert the data type of the initial image data to a preset data type to obtain intermediate image data; the preset data type is a data type supported by the neural network accelerator.

[0064] The data types supported by the neural network accelerator can include signed integers. Intermediate image data consists of image data converted from the initial image data to data types supported by the neural network accelerator.

[0065] For example, if the initial image data is Its data type is uint8, and the data type supported by neural network accelerators is int8. Therefore, the value range of the corresponding int8 data type can be determined as follows: Therefore, it can be determined that the number of selectable values ​​within this range is 256, which can then be converted into intermediate image data. .

[0066] Step 306: After performing linear operations on the intermediate image data using a neural network accelerator, the data type of the linearly processed image data is reverse-converted to obtain the target image data; the target image data and the initial image data have the same data type.

[0067] Specifically, neural network accelerators can perform linear operations on the intermediate image data they can support to achieve noise reduction. These linear operations can include linear filtering algorithms.

[0068] Since the processed image data is a data type supported by the neural network accelerator, in order to ensure that the output image data results can be used for subsequent operations, the data type of the intermediate image data can be reverse-converted to ensure that the data type of the target image data is consistent with that of the initial input image data.

[0069] For example, based on the image data after linear operation processing and the value range of the data types supported by the neural network accelerator, the compensation amount required for data type inverse conversion can be determined. Then, based on this compensation amount, data compensation of the data type is performed on the image data obtained from the linear operation processing to obtain the target image data.

[0070] In this embodiment, initial image data is acquired and its data type is converted to a preset data type to obtain intermediate image data. The preset data type is a data type supported by the neural network accelerator. After linear processing of the intermediate image data using the neural network accelerator, the data type of the linearly processed image data is inversely converted to obtain the target image data. This disclosure enables the neural network accelerator to process image data of multiple data types compatiblely, reducing the problem of abnormal image processing results caused by data type incompatibility. Furthermore, the target image data and the initial image data have the same data type, ensuring the consistency of input and output data types.

[0071] The above embodiments illustrate the image data processing method. When processing the initial image data, in one optional embodiment, the data type supported by the neural network accelerator is signed integer, and the data type of the initial image data is unsigned integer. The binary numbers of unsigned integer and signed integer occupy the same number of bytes.

[0072] Signed integer types include int8, int16, int32, int64, etc., without limitation. Unsigned integer types include uint8, uint16, uint32, uint64, etc., without limitation. Unsigned and signed integers occupy the same number of bytes in binary form; for example, int8 corresponds to uint8, int16 corresponds to uint16, etc., and these will not be listed here.

[0073] like Figure 5 As shown above, in the above Figure 4 Based on the illustrated embodiment, step 304 may include the following steps:

[0074] Step 3042: Obtain the first signed bit matrix corresponding to the initial image data. The first signed bit matrix is ​​used to convert the data type of each pixel from unsigned integer to signed integer.

[0075] Specifically, when the initial image data is obtained, the first sign bit matrix corresponding to the initial image data can be determined based on whether the pixel value of each pixel in the initial image exceeds the value range of the int type data. If the pixel value exceeds the value range of the int type data, the value of the element in the first sign bit matrix corresponding to the pixel value can be determined to be -1; if the pixel value does not exceed the value range of the int type data, the value of the element in the first sign bit matrix corresponding to the pixel value can be determined to be 0.

[0076] For example, if the initial image data is obtained Its data type is uint8. It should be noted that the pixel values ​​of each pixel in the initial image data are binary numbers. For ease of explanation, each binary number is represented by its corresponding decimal number, that is, 128, 127, 126, 131, 132, 133, 255, 254, and 253 are all decimal numbers corresponding to the binary values ​​of the pixel values.

[0077] For the matrix A above, it's clear that 128, 131, 132, 133, 255, 254, and 253 all exceed the value range of int8 type data (-128, 127). Therefore, the corresponding elements in the first sign bit matrix can be set to -1. However, 127 and 126 do not exceed the value range of int type data (-128, 127), so the corresponding elements in the first sign bit matrix can be set to 0. Thus, the first sign bit matrix is ​​obtained: .

[0078] In another alternative embodiment, the corresponding first sign bit matrix can be determined according to a preset first transformation formula. The first transformation formula is as follows: ,in, This is the initial image data of type int, determined based on the binary numbers of each pixel in the initial image data A.

[0079] Furthermore, the method for determining the first sign bit matrix includes:

[0080] Step 402: When the binary value of a pixel in the initial image data exceeds the range of binary values ​​of a signed integer, overflow saturation is performed on the pixel to obtain the first image data.

[0081] For example, if the initial image data obtained is: The initial image data is of type uint8, where each pixel value is a binary number. Since the neural network accelerator defaults to supporting the int8 data type, and the data type of the initial image data cannot be identified solely by binary numbers, the uint8 initial image data A is treated as a signed integer, a data type supported by the neural network accelerator. However, since the range of signed integers is (-128, 127), elements in matrix A exceeding the maximum value of 127 are subjected to overflow saturation processing, thus obtaining the first image data. The overflow saturation processing involves subtracting 256 from each pixel value that exceeds the range of int8 data to obtain the first image data.

[0082] Step 404: Invert the binary value of each pixel in the first image data and subtract 1 to obtain the second image data.

[0083] For example, for the first image data Invert the binary representation of each pixel and subtract 1, that is... This allows us to obtain the second image data: .

[0084] Step 406: Shift the second image data to the right by a first shift value to determine the first sign bit matrix; the first shift value is the value obtained by subtracting 1 from the bit value corresponding to the byte occupied by the initial image data.

[0085] For example, if the initial image data is of data type uint8, then the number of bits corresponding to the bytes occupied by the initial image data is 8, and the first shift value is 7. For the second image data... Shift right by 7 bits to determine the first sign bit matrix: .

[0086] In this embodiment, when the binary value of a pixel in the initial image data exceeds the range of signed integer binary values, overflow saturation is performed on the pixel to obtain the first image data. Then, the binary value of each pixel in the first image data is inverted and subtracted by 1 to obtain the second image data. Next, the second image data is right-shifted by a first shift value to determine the first sign bit matrix. This first sign bit matrix is ​​used to convert the data type of each pixel from unsigned integer to signed integer during subsequent data type conversion.

[0087] Step 3044: Based on the first sign bit matrix, the data type of the initial image data is converted to obtain intermediate image data. Each pixel in the initial image data and the intermediate image data occupies the same number of bytes.

[0088] In this design, each pixel in the initial image data and the intermediate image data occupies the same number of bytes. For example, if the data type of the initial image data is uint8, then it occupies 1 byte, meaning that the data type of the intermediate image data is int8, which is supported by the neural network accelerator.

[0089] Specifically, the first sign bit matrix can be multiplied by the number of possible values ​​of the data type corresponding to the initial image data, and then added to the initial image data and half of the number of possible values ​​to obtain the intermediate image data, which can be obtained through the second conversion formula: The data type of the initial image data is converted. Here, X represents the number of possible values, Y is half of the number of possible values, and C represents the intermediate image data. If the initial image data is of data type uint8, then the number of possible values ​​is 256.

[0090] Furthermore, in an alternative embodiment, step 3044 specifically includes:

[0091] Step 3044a: Obtain the binary value range corresponding to the first compensation parameter, the first image data, and the initial image data. The first compensation parameter represents half the number of values ​​within the binary value range.

[0092] Specifically, when initial image data is acquired, the data type processed by the neural network accelerator can be determined based on the number of bits in the binary number of the initial image data, and thus the range of binary values ​​for the corresponding data type can be determined. For example, if the binary value of a pixel in the initial image data is 0b11111111, then based on the fact that the number of bits in the binary number is 8, the corresponding data type processed by the neural network accelerator is int8, and its corresponding binary value range is (-128, 127). Furthermore, it can be determined that the number of values ​​in the corresponding binary value range is 256, and the first compensation parameter can also be determined to be 128.

[0093] Step 3044b: Multiply the first sign bit matrix by the number of binary numbers in the binary value range to obtain the first sign bit transformation matrix. The number of binary numbers in the binary value range is the value obtained by left-shifting the binary digit 1 according to its bit value.

[0094] For example, continuing with the initial image data... For example, if its data type is uint8, the corresponding bit value is 8. Shifting 1 left by 8 bits gives 256 binary numbers within the range of possible binary values. Therefore, the first sign transformation matrix... .

[0095] Step 3044c: Add the first compensation parameter, the first sign bit transformation matrix, and the first image data to obtain the intermediate image data.

[0096] For example, then for: That is, the intermediate image data is C.

[0097] In this embodiment, the binary value ranges corresponding to the first compensation parameter, the first image data, and the initial image data are obtained. The first sign bit matrix is ​​multiplied by the number of binary numbers in the binary value range to obtain the first sign bit transformation matrix. The first compensation parameter, the first sign bit transformation matrix, and the first image data are added together to obtain the intermediate image data. This disclosure enables the conversion of the initial image data data type to a data type supported by the neural network accelerator, allowing the neural network accelerator to correctly process the initial image data and avoiding the appearance of noise points after linear processing.

[0098] The above embodiments illustrate how to obtain intermediate image data of a type that can be processed by a neural network accelerator. After obtaining the intermediate image data, linear operations can be performed on it. To ensure the consistency of input and output, it is necessary to further perform an inverse transformation on the image data after linear operation. An embodiment is now used to illustrate how to perform linear operations and inverse transformation. Step 306 specifically includes:

[0099] Step 3062: Perform linear operations on the intermediate image data to obtain reference image data; the reference image data is signed integer data.

[0100] The linear processing can be a linear filtering algorithm. The reference image data is the image obtained by filtering the initial image data.

[0101] For example, if the filtering matrix used in the linear filtering algorithm is Then, performing linear operations on the intermediate image data can be... Where C represents intermediate image data and D represents reference image data.

[0102] For example, .

[0103] It should be noted that after linear operation, inverse conversion steps S3064-S3066 are required to ensure that the data type of the output is consistent with that of the input.

[0104] Step 3064: Obtain the second sign bit matrix of the reference image data; the second sign bit matrix is ​​used to convert the data type of each pixel in the reference image data from signed integer to unsigned integer.

[0105] Specifically, when the reference image data is obtained, the second sign bit matrix corresponding to the reference image data can be determined based on whether the pixel value of each pixel in the reference image exceeds the value range of the int type data. If the pixel value exceeds the value range of the int type data, the element in the first sign bit matrix corresponding to the pixel value can be determined to be -1; if the pixel value does not exceed the value range of the int type data, the element in the first sign bit matrix corresponding to the pixel value can be determined to be 0.

[0106] For example, if the obtained reference image data is D=33, its data type is int8. It should be noted that the pixel values ​​of each pixel in the initial image data are binary numbers. For ease of explanation, each binary number is represented by its corresponding decimal number.

[0107] For the reference image data D mentioned above, it is clear that 33 does not exceed the value range of int8 type data (-128, 127), so the second sign bit matrix P can be determined to be -1.

[0108] Furthermore, the method for determining the second sign bit matrix includes:

[0109] Step 502: Invert the binary value of each pixel in the reference image data and subtract 1 to obtain the third image data.

[0110] For example, the reference image data D=33 is inverted and then subtracted by 1, that is... That is, the third image data is L=-34.

[0111] Step 504: Shift the third image data to the right by a second shift value to determine the second sign bit matrix; the second shift value is the value obtained by subtracting 1 from the bit value corresponding to the byte occupied by the reference image data.

[0112] For example, if the data type of the reference image data is int8, then the bit value corresponding to the byte occupied by the reference image data is 8, and the second shift value is 7. The third image data L=-34 is right-shifted by 7 bits to determine the second symbol matrix P=-1.

[0113] In this embodiment of the disclosure, the third image data is obtained by inverting the binary value of each pixel in the reference image data and subtracting 1. The third image data is then right-shifted by a second shift value to determine the second sign bit matrix. This is so that when performing data type conversion to ensure the consistency of input and output data types in the future, the data type of each pixel in the reference image data can be converted to a data result consistent with the data type of the initial image data using the second sign bit matrix, thereby further reducing the generation of image data noise.

[0114] Step 3066: Based on the second sign bit matrix, the data type of the reference image data is reverse-converted to obtain the target image data. The number of bytes occupied by each pixel in the reference image data and the target image data is the same.

[0115] In this context, each pixel in the reference image data and the target image data occupies the same number of bytes. For example, if the data type of the reference image data is int8, then it occupies 1 byte, meaning that the data type of the target image data is the same as the int8 data type of the initial image data.

[0116] Specifically, the target image data can be obtained by multiplying the second sign bit matrix by the number of possible values ​​of the data type corresponding to the reference image data, and then adding this result to the reference image data and half of the number of possible values. This can be achieved using the second conversion formula: The data type of the initial image data is converted, where X is the number of possible values, Y is half of the number of possible values, M is the target image data, P is the second sign bit matrix, and D is the reference image data. If the reference image data is of type int8, then the number of possible values ​​is 256.

[0117] Further, in an alternative embodiment, step 3066 specifically includes:

[0118] Step 3066a: Obtain the second compensation parameter, reference image data, and the corresponding binary value range of the reference image data. The second compensation parameter is used to represent half of the number of values ​​in the binary value range.

[0119] Specifically, when the reference image data is obtained, the data type of the reference image data processed by the neural network accelerator can be determined based on the number of bits in the binary number of the reference image data, and thus the range of binary values ​​of the corresponding data type can be determined.

[0120] For example, if the data type of the reference image data is int8, then the number of bits in the binary number can be determined to be 8, and thus the data type of the reference image data can be determined to be int8, and the binary value range of the int8 data type can be determined to be [-128, 127].

[0121] Furthermore, since the number of values ​​in the binary range of the int8 data type can be determined to be 256, the second compensation parameter can be determined to be 128.

[0122] Similarly, if the number of possible values ​​in the binary range of the int16 data type is 65536, then the second compensation parameter can be determined to be 32768. Furthermore, image data with data types int32 and int64 correspond to different second compensation parameters; the specific determination method can be found by referring to the method for determining image data with data type int8.

[0123] It should be noted that, in one example, the method for determining the second compensation parameter includes:

[0124] Step 602: Obtain the binary value range corresponding to the reference image data.

[0125] Step 604: Divide the number of binary numbers in the binary value range corresponding to the reference image data by 2 to determine the second compensation parameter.

[0126] For example, if the binary value of a pixel in the reference image data is 0b00000001, and the number of bits in the binary number is 8, then the data type processed by the neural network accelerator is int8, with a corresponding binary value range of (-128, 127). Therefore, the number of values ​​within this binary value range can be determined to be 256. Consequently, the second compensation parameter can also be determined to be 128.

[0127] Step 3066b: Multiply the second sign bit matrix by the number of binary numbers in the binary value range to obtain the second sign bit transformation matrix. The number of binary numbers in the binary value range is the value obtained by left-shifting the binary digit 1 according to its bit value.

[0128] Specifically, the second sign bit transformation matrix is ​​obtained by multiplying the second sign bit matrix by the number of binary numbers in the binary value range. The number of binary numbers in the binary value range is the value obtained by left-shifting the binary digit 1 according to its bit value.

[0129] For example, if the second sign bit matrix is ​​P=-1, continue using the reference image data as... For example, if its data type is int8, the corresponding bit value is 8. Shifting 1 left by 8 bits gives a range of 256 binary numbers.

[0130] Furthermore, the second sign bit matrix is ​​multiplied by the number of binary numbers in the range of binary values, which is the second sign transformation matrix. .

[0131] Step 3066c: Add the second compensation parameter, the second sign bit transformation matrix, and the reference image data to obtain the target image data.

[0132] For example, continuing with the reference image data... Taking a second compensation parameter of 128 and a second sign bit matrix of -256 as an example, the target image data can be obtained. That is, the output at this time is the binary number corresponding to the decimal number of int8 data type - 95: 10100001. Since the data type of the output at this time is uint data type, the binary number 10100001 in uint8 data type corresponds to the decimal number 161, which means that the target image data M is 161.

[0133] Compared to traditional techniques, where neural network accelerators lacking uint8 processing capabilities directly treat matrix A as int8 matrix B and then perform Gaussian filtering on matrix B, resulting in an output image data of -46, this disclosure clearly provides the correct output image data of 161. Furthermore, it yields... Figure 2 The image shown is clear and normal.

[0134] In this embodiment, based on the obtained second compensation parameter, reference image data, and the corresponding binary value range of the reference image data, the second sign bit matrix is ​​multiplied by the number of binary numbers in the binary value range to obtain the second sign bit transformation matrix. Then, the second compensation parameter, the second sign bit transformation matrix, and the reference image data are added together to obtain the target image data. This disclosure can achieve the inverse conversion of the data type of the reference image data to ensure consistency with the data type of the initial image data.

[0135] It should be understood that, although Figure 4 , Figure 5 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 4 , Figure 5 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0136] Exemplary device

[0137] Based on the same inventive concept as the image data processing method, this disclosure also provides an image data processing apparatus applied to a neural network accelerator, such as... Figure 6 As shown, it includes:

[0138] Data acquisition module 601 is used to acquire initial image data;

[0139] The conversion module 602 is used to convert the data type of the initial image data into a preset data type to obtain intermediate image data; the preset data type is a data type supported by the neural network accelerator.

[0140] The computation processing module 603 is used to perform linear computation processing on the intermediate image data using a neural network accelerator, and then perform inverse data type conversion on the linearly processed image data to obtain target image data. The target image data and the initial image data have the same data type.

[0141] Optionally, the neural network accelerator supports signed integer data types, while the initial image data is of unsigned integer data type. The binary numbers of unsigned integers and signed integers occupy the same number of bytes.

[0142] Optionally, the linear operation is processed using a linear filtering algorithm.

[0143] In one embodiment, such as Figure 7 As shown, the conversion module 602 includes:

[0144] The first sign bit matrix acquisition unit 6021 is used to acquire the first sign bit matrix corresponding to the initial image data; the first sign bit matrix is ​​used to convert the data type of each pixel from unsigned integer to signed integer.

[0145] The first conversion unit 6022 is used to convert the data type of the initial image data based on the first sign bit matrix to obtain intermediate image data; each pixel of the initial image data and the intermediate image data occupies the same number of bytes.

[0146] In one embodiment, see Figure 7 As shown, the conversion module further includes: a first sign bit matrix determination unit 6023, used to overflow and saturate the pixels to obtain first image data when the binary value of a pixel in the initial image data exceeds the range of binary values ​​of a signed integer; to invert the binary value of each pixel in the first image data and subtract 1 to obtain second image data; to right-shift the second image data by a first shift value to determine the first sign bit matrix; the first shift value is the value obtained by subtracting 1 from the bit value corresponding to the byte occupied by the initial image data.

[0147] In one embodiment, see Figure 7As shown, the first conversion unit 6022 is specifically used to obtain the binary value range corresponding to the first compensation parameter, the first image data, and the initial image data; the first compensation parameter is used to represent half of the number of values ​​in the binary value range; the first sign bit matrix is ​​multiplied by the number of binary numbers in the binary value range to obtain the first sign bit transformation matrix; the number of binary numbers in the binary value range is the value obtained by left-shifting the binary digit 1 according to the bit value; the first compensation parameter, the first sign bit transformation matrix, and the first image data are added together to obtain the intermediate image data.

[0148] In one embodiment, see Figure 7 As shown, the arithmetic processing module 603 includes:

[0149] The linear processing unit 6031 is used to perform linear operations on the intermediate image data to obtain reference image data; the reference image data is signed integer data.

[0150] The second sign bit matrix acquisition unit 6032 is used to acquire the second sign bit matrix of the reference image data; the second sign bit matrix is ​​used to convert the data type of each pixel in the reference image data from signed integer to unsigned integer.

[0151] The second conversion unit 6033 is used to perform inverse conversion of the data type of the reference image data based on the second sign bit matrix to obtain the target image data. Each pixel in the reference image data and the target image data occupies the same number of bytes.

[0152] In one embodiment, see Figure 7 As shown, the arithmetic processing module 603 also includes:

[0153] The second sign bit matrix determination unit 6034 is used to invert the binary value of each pixel in the reference image data and subtract 1 to obtain the third image data; and to right-shift the third image data by a second shift value to determine the second sign bit matrix; the second shift value is the value obtained by subtracting 1 from the bit value corresponding to the byte occupied by the reference image data.

[0154] In one embodiment, see Figure 7 As shown, the second conversion unit 6033 is specifically used to obtain the second compensation parameter, the reference image data, and the binary value range corresponding to the reference image data; the second compensation parameter is used to represent half of the number of values ​​in the binary value range; the second sign bit matrix is ​​multiplied by the number of binary numbers in the binary value range to obtain the second sign bit transformation matrix; the number of binary numbers in the binary value range is the value obtained by left-shifting the binary digit 1 according to the bit value; the second compensation parameter, the second sign bit transformation matrix, and the reference image data are added together to obtain the target image data.

[0155] In one embodiment, see Figure 7 As shown, the arithmetic processing module 603 also includes:

[0156] The second compensation parameter determination unit 6035 is used to obtain the binary value range corresponding to the reference image data; and to determine the second compensation parameter by dividing the number of binary numbers in the binary value range corresponding to the reference image data by 2.

[0157] Specific limitations regarding the image data processing device can be found in the limitations of the image data processing method described above, and will not be repeated here. Each module in the aforementioned image data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0158] Exemplary electronic devices

[0159] Figure 8 This diagram illustrates a structural block diagram of an electronic device according to an exemplary embodiment, which can be used to perform the above-described combination. Figure 4 Describes the image processing methods.

[0160] like Figure 8 As shown, the electronic device may include one or more processors 310, one or more memories 320, hardware accelerators 330, and one or more input / output (I / O) interfaces 340, which are interconnected via a bus system 350.

[0161] The processor 310 can be any form of processing unit with data processing capabilities and / or instruction execution capabilities, examples of which include, but are not limited to, a central processing unit (CPU), an ARM processor, a microprocessor unit (MCU), a general-purpose processor, a controller, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The processor 310 can execute instructions in its associated memory 320 and / or exchange data with it, thereby controlling other components coupled via the bus system 350 to operate collaboratively and perform the methods, steps, or functions described above.

[0162] The memory 320 may include various forms of computer-readable and writable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include dynamic random access memory (DRAM) and / or cache memory. The non-volatile memory may, for example, include electrically erasable programmable read-only memory (EEPROM), hard disk, flash memory, etc. The readable and writable storage media may include, for example, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof.

[0163] In addition, the memory 320 may store computer-executable instructions that can be executed by the processor 310 to control other components coupled via the bus system 350 to operate in coordination, thereby performing the methods, steps or functions described above.

[0164] The hardware accelerator 330, also known as an AI accelerator or dedicated hardware, can be specialized hardware designed to perform image processing-related tasks. For example... Figure 8 As shown, the hardware accelerator 330 may include a multiply-accumulate array 3301 and an on-chip memory or on-chip cache 3302. It should be understood that the memory 320 described above may also be referred to as off-chip memory relative to the on-chip cache 3302. The multiply-accumulate array 3301 may include an array of multiple multipliers and multiple adders for feature map processing, convolution processing, filtering, and reconstruction processing. The on-chip cache 3302 may include, for example, static random access memory (SRAM) for caching the raw image data to be provided to the multiply-accumulate array 3301, such as first image data.

[0165] I / O interface 340 may include communication interfaces for connecting to various input and output devices, such as camera interfaces, display interfaces, network interfaces, and controller interfaces supporting specific communication protocols. It should be understood that various I / O interfaces 340 can be provided according to actual application needs. In this embodiment, the I / O interface 340 is used to connect to devices such as… Figure 3 The camera 20 and display screen 10 are shown.

[0166] Bus system 350 can be any bus system capable of connecting various components in an electronic device and supporting communication between them, examples of which include, but are not limited to, CAN (Regional Network Controller) bus, ISA (Industry Standard Architecture) bus, PIC (Peripheral Interconnect) or PCI-E (Rapid Peripheral Interconnect) bus, I2C (Inter-Integrated Circuit) bus, SPI (Serial Peripheral Interface) bus, UART (Universal Asynchronous Serial Port) bus, etc.

[0167] It should be understood that, Figure 8The electronic device shown in this embodiment only includes some components that are closely related to this disclosure, but other components may also be included, and this embodiment does not limit this.

[0168] Furthermore, in addition to the aforementioned electronic devices, the method disclosed herein can also be implemented using general-purpose hardware such as a CPU or GPU, and can achieve similar technical effects. For example, before filtering the image data, the unsigned first image data is converted into signed second image data using a first mapping relationship. Since the hardware platform supports the processing of signed data, the second image data can be directly filtered and restored after conversion, thereby obtaining a clear and complete image. This avoids the problems of blurred or abnormal image edges caused by forcibly processing unsigned image data.

[0169] Exemplary computer program products and computer-readable storage media

[0170] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, can control a hardware accelerator to perform the methods described in part or all of the "Exemplary Methods" section of this specification.

[0171] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java, C++, and Phyon, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0172] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the image processing methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0173] Furthermore, the computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0174] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the specific details described above. Rather, those skilled in the art will readily conceive of many variations in form and detail under the teachings of this disclosure, and all such variations should fall within the scope defined by the claims of this disclosure.

[0175] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” and “having” are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0176] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0177] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0178] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. An image data processing method, comprising: Obtain initial image data; Based on the value range of data types supported by the neural network accelerator, the data type of the initial image data is converted into a preset data type to obtain intermediate image data; the preset data type is a data type supported by the neural network accelerator. After performing linear operations on the intermediate image data using the neural network accelerator, the data type of the linearly processed image data is inversely converted to obtain target image data. The data type of the target image data is the same as that of the initial image data. The process of inversely converting the data type of the linearly processed image data to obtain the target image data includes: determining the compensation amount required for the data type inverse conversion based on the linearly processed image data and the value range of the data types supported by the neural network accelerator; and performing data compensation on the image data obtained from the linearly processed image data based on the compensation amount to obtain the target image data.

2. The method according to claim 1, wherein, The neural network accelerator supports signed integer data types, while the initial image data is of unsigned integer data type. The unsigned integer and the signed integer binary numbers occupy the same number of bytes.

3. The method according to claim 2, wherein, The step of converting the data type of the initial image data into a preset data type to obtain intermediate image data includes: Obtain the first sign bit matrix corresponding to the initial image data; the first sign bit matrix is ​​used to convert the data type of each pixel from unsigned integer to signed integer. Based on the first sign bit matrix, the data type of the initial image data is converted to obtain the intermediate image data; each pixel of the initial image data and the intermediate image data occupies the same number of bytes.

4. The method according to claim 3, wherein, The method for determining the first sign bit matrix includes: When the binary value of a pixel in the initial image data exceeds the range of binary values ​​of the signed integer, the pixel is overflow-saturated to obtain the first image data. The second image data is obtained by inverting the binary value of each pixel in the first image data and subtracting 1. The second image data is shifted right by a first shift value to determine the first sign bit matrix; the first shift value is the value obtained by subtracting 1 from the bit value corresponding to the byte occupied by the initial image data.

5. The method according to claim 4, wherein, The step of converting the data type of the initial image data based on the first sign bit matrix to obtain the intermediate image data includes: Obtain the binary value range corresponding to the first compensation parameter, the first image data, and the initial image data; the first compensation parameter is used to represent half of the number of values ​​in the binary value range; Multiply the first sign bit matrix by the number of binary numbers in the range of binary values ​​to obtain the first sign bit transformation matrix; the number of binary numbers in the range of binary values ​​is the value obtained by left-shifting the binary digit 1 according to the bit value; The first compensation parameter, the first sign bit transformation matrix, and the first image data are added together to obtain the intermediate image data.

6. The method according to any one of claims 2-5, wherein, After performing linear operations on the intermediate image data using the neural network accelerator, the data type of the linearly processed image data is inversely converted to obtain the target image data, including: The intermediate image data is processed by linear operations to obtain reference image data; the reference image data is signed integer data. Obtain the second sign bit matrix of the reference image data; the second sign bit matrix is ​​used to convert the data type of each pixel in the reference image data from signed integer to unsigned integer. Based on the second sign bit matrix, the data type of the reference image data is inversely converted to obtain the target image data. The number of bytes occupied by each pixel in the reference image data and the target image data are the same.

7. The method according to claim 6, wherein, The method for determining the second sign bit matrix includes: Invert the binary value of each pixel in the reference image data and subtract 1 to obtain the third image data; The third image data is right-shifted by a second shift value to determine the second sign bit matrix; the second shift value is the value obtained by subtracting 1 from the bit value corresponding to the byte occupied by the reference image data.

8. The method according to claim 7, wherein, The step of converting the data type of the reference image data based on the second symbol bit matrix to obtain the target image data includes: Obtain the second compensation parameter, the reference image data, and the binary value range corresponding to the reference image data; the second compensation parameter is used to represent half of the number of values ​​in the binary value range; Multiply the second sign bit matrix by the number of binary numbers in the range of binary values ​​to obtain the second sign bit transformation matrix; the number of binary numbers in the range of binary values ​​is the value obtained by left-shifting the binary digit 1 according to the bit value; The target image data is obtained by adding the second compensation parameter, the second sign bit transformation matrix, and the reference image data.

9. The method according to claim 8, wherein, The method for determining the second compensation parameter includes: Obtain the binary value range corresponding to the reference image data; The second compensation parameter is determined by dividing the number of binary numbers in the range of binary values ​​corresponding to the reference image data by 2.

10. The method according to any one of claims 2-5, wherein, The linear operation processing is a linear filtering algorithm.

11. An image data processing apparatus, applied to a neural network accelerator, comprising: The data acquisition module is used to acquire initial image data; The conversion module is used to convert the data type of the initial image data into a preset data type based on the value range of the data types supported by the neural network accelerator, so as to obtain intermediate image data; the preset data type is a data type supported by the neural network accelerator. The computational processing module is used to perform linear computation processing on the intermediate image data using the neural network accelerator, and then perform inverse data type conversion on the linearly processed image data to obtain target image data. The target image data and the initial image data have the same data type. The inverse data type conversion of the linearly processed image data to obtain the target image data includes: determining the compensation amount required for the inverse data type conversion based on the linearly processed image data and the value range of the data types supported by the neural network accelerator; and performing data compensation on the linearly processed image data based on this compensation amount to obtain the target image data.

12. An electronic device, comprising: Memory, used to store processor-executable instructions; A processor is configured to read the executable instructions from the memory and execute the instructions to implement the image data processing method according to any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, the computer program being used to perform the image data processing method according to any one of claims 1-10.

Images