An image processing method, apparatus and device

Abstract

This application provides an image processing method, apparatus, and device. The method includes: preprocessing an acquired original image to obtain first image data, wherein the preprocessing converts the original image into first image data adapted to an AI model; performing AI processing on the first image data using the AI ​​model to obtain second image data, wherein the AI ​​processing enhances high-frequency information in the first image data; storing the second image data in a designated storage medium; reading the stored image data from the designated storage medium; encoding and compressing the stored image data to obtain a target bitstream; sending the target bitstream to a backend device; and having the backend device acquire a target image based on the target bitstream, and store or display the target image. The technical solution of this application can improve image quality.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus and device. Background Technology

[0002] Front-end devices (such as smartphones, IoT devices, cameras, etc.) are also called edge devices. After acquiring the raw image, the front-end device can encode the raw image to obtain a bitstream and transmit the bitstream to the back-end device (such as storage device, display device, etc.). After receiving the bitstream, the back-end device can decode the bitstream to obtain the target image and store or display the target image.

[0003] When the front-end device encodes the original image to obtain the bitstream, the encoding process is actually a compression process of the original image. Although it can reduce the amount of data transmission, it also leads to a reduction in the image quality of the original image. Therefore, when the back-end device decodes the bitstream to obtain the target image, the image quality of the target image is lower than that of the original image, resulting in poor image quality of the target image when stored or displayed. Summary of the Invention

[0004] This application provides an image processing method applied to a front-end device, the method comprising:

[0005] The acquired raw image is preprocessed to obtain first image data, wherein the preprocessing is used to convert the raw image into first image data that is compatible with the artificial intelligence (AI) model;

[0006] The first image data is processed by the AI ​​model to obtain the second image data, and the AI ​​processing is used to enhance the high-frequency information in the first image data;

[0007] Store the second image data in the specified storage medium;

[0008] The system reads stored image data from the specified storage medium, encodes and compresses the stored image data to obtain a target bitstream, sends the target bitstream to a backend device, and the backend device acquires the target image based on the target bitstream, and stores or displays the target image.

[0009] This application provides an image processing apparatus applied to a front-end device, the apparatus comprising:

[0010] A preprocessing module is used to preprocess the acquired raw image to obtain first image data. The preprocessing is used to convert the raw image into first image data that is compatible with an artificial intelligence (AI) model.

[0011] The AI ​​processing module is used to perform AI processing on the first image data through the AI ​​model to obtain the second image data, and the AI ​​processing is used to enhance the high-frequency information in the first image data;

[0012] A storage module for storing the second image data in a specified storage medium;

[0013] The sending module is used to read stored image data from a specified storage medium, encode and compress the stored image data to obtain a target bitstream, send the target bitstream to a backend device, and have the backend device acquire, store, or display the target image based on the target bitstream.

[0014] This application provides an electronic device, including: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the image processing method of the example above in this application.

[0015] This application provides a computer program product, which includes a computer program that, when executed by a processor, implements the image processing method executorified above in this application.

[0016] This application provides a machine-readable storage medium storing machine-executable instructions that can be executed by a processor; wherein the processor is configured to execute the machine-executable instructions to implement the image processing method of the example above in this application when the machine-executable instructions are executed.

[0017] As can be seen from the above technical solutions, in this embodiment, after acquiring the original image, the front-end device performs preprocessing on the original image to obtain first image data. Then, it uses an AI model to perform AI processing on the first image data to obtain second image data, whereby the AI ​​processing enhances the high-frequency information in the first image data. Based on this, the front-end device encodes and compresses the image data to obtain a target bitstream, which is then sent to the back-end device. The back-end device decodes the target bitstream to obtain the target image. Because the high-frequency information in the image data is enhanced by the AI ​​model, even if the encoding and compression process reduces the image quality of the image data (loss of high-frequency information leads to reduced image quality), the image quality of the target image obtained by the back-end device decoding the target bitstream is equivalent to that of the original image, thereby improving the image quality of the stored or displayed target image. By using an AI model deployed on the front-end device to process the real-time data stream (i.e., video stream), the image quality of the real-time data stream can be improved, helping video image encoding and decoding achieve higher video image quality. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating an image processing method according to one embodiment of this application;

[0019] Figure 2 This is a flowchart illustrating an image processing method according to one embodiment of this application;

[0020] Figure 3 This is a schematic diagram of the pre-compression process in one embodiment of this application;

[0021] Figure 4A This is a schematic diagram of pixel filling of non-edge blocks in one embodiment of this application;

[0022] Figure 4B This is a schematic diagram of pixel filling of edge blocks in one embodiment of this application;

[0023] Figure 5A This is a schematic diagram of image transformation of the first image data in one embodiment of this application;

[0024] Figure 5B This is a schematic diagram illustrating image augmentation of the first image data in one embodiment of this application;

[0025] Figure 5C This is a schematic diagram illustrating semantic transformation of first image data in one embodiment of this application;

[0026] Figure 5D This is a schematic diagram illustrating AI processing of first image data in one embodiment of this application;

[0027] Figure 6A This is a schematic diagram of image data storage in one embodiment of this application;

[0028] Figure 6B This is a schematic diagram of the storage method of two tiles in one embodiment of this application;

[0029] Figure 6C This is a schematic diagram of image data display in one embodiment of this application;

[0030] Figure 7 This is a schematic diagram of the structure of an image processing apparatus according to one embodiment of this application;

[0031] Figure 8 This is a hardware structure diagram of an electronic device according to one embodiment of this application. Detailed Implementation

[0032] This application proposes an image processing method that can be applied to front-end devices, such as smartphones, IoT devices, and cameras, without limiting the type of front-end device. See also Figure 1 The diagram shown is a flowchart of the image processing method, which may include:

[0033] Step 101: Preprocess the acquired original image to obtain the first image data. The preprocessing is used to convert the original image into the first image data that is compatible with the AI ​​(artificial intelligence) model.

[0034] Step 102: The first image data is processed by an AI model to obtain the second image data. The AI ​​processing is used to enhance the high-frequency information in the first image data.

[0035] Step 103: Store the second image data in the specified storage medium.

[0036] Step 104: Read the stored image data from the specified storage medium, encode and compress the stored image data to obtain the target bitstream, and send the target bitstream to the backend device. The backend device then obtains the target image based on the target bitstream and stores or displays the target image.

[0037] For example, preprocessing may include, but is not limited to, at least one of color space transformation, pixel value mapping, image block partitioning, and image rearrangement. Preprocessing the acquired original image to obtain first image data may include, but is not limited to: if the preprocessing includes color space transformation, and the first color space of the original image is a color space not supported by the AI ​​model, then the original image can be subjected to color space transformation to obtain image data in a second color space, where the second color space is a color space supported by the AI ​​model.

[0038] If the preprocessing includes pixel value mapping, and the first pixel value distribution of the original image is a pixel value distribution that the AI ​​model does not support, then the original image can be pixel value mapped to obtain image data with a second pixel value distribution, which is a pixel value distribution supported by the AI ​​model.

[0039] If the preprocessing includes image block partitioning, the original image can be partitioned into multiple initial image blocks. These initial image blocks can include edge blocks and non-edge blocks. Edge blocks include the boundaries of the original image, while non-edge blocks do not include the boundaries of the original image. For each initial image block, pixel filling can be performed to obtain target image blocks. These multiple target image blocks serve as input data for the AI ​​model, and the AI ​​model processes these multiple target image blocks in parallel.

[0040] If the preprocessing includes image rearrangement, and the first arrangement of the original image is an arrangement (i.e., a data arrangement) that the AI ​​model does not support, then the original image can be rearranged to obtain image data in a second arrangement, and the second arrangement is an arrangement supported by the AI ​​model.

[0041] For example, AI processing may include, but is not limited to, at least one of image transformation, image augmentation, and semantic transformation. If AI processing includes image transformation, then obtaining second image data by performing AI processing on the first image data through an AI model may include, but is not limited to: performing image transformation on the first image data through an AI model to obtain transformed image data, wherein the resolution of the transformed image data may be equal to or less than the resolution of the first image data, and the number of channels of the transformed image data may be equal to or greater than the number of channels of the first image data; wherein, the image transformation is used to separate high-frequency and low-frequency information in the first image data. The image transformation may include, but is not limited to, at least one of interpolation transformation (such as nearest neighbor interpolation, mean transformation, bilinear interpolation, adaptive interpolation, bicubic interpolation, etc.), wavelet transform, Gaussian pyramid transform, Laplacian pyramid transform, and Sobel filtering transform. The transformed image data is input into the image processing neural network in the AI ​​model, and the image processing neural network operates on the transformed image data to obtain the second image data; wherein, the resolution of the second image data may be equal to or less than the resolution of the first image data, and the image processing neural network is used to enhance the high-frequency information in the first image data.

[0042] For example, AI processing may include, but is not limited to, at least one of image transformation, image augmentation, and semantic transformation. If AI processing includes image augmentation, then obtaining second image data by AI processing the first image data through an AI model may include, but is not limited to: inputting the first image data into an image processing neural network in the AI ​​model, operating the first image data through the image processing neural network to obtain image data to be augmented, wherein the resolution of the image data to be augmented may be equal to or less than the resolution of the first image data, and the number of channels of the image data to be augmented may be equal to or less than the number of channels of the first image data. Image augmentation is performed on the first image data through an AI model to obtain augmented image data corresponding to the image data to be augmented, wherein the resolution of the augmented image data is equal to or less than the resolution of the first image data, and the number of channels of the augmented image data is equal to or less than the number of channels of the first image data; wherein the augmented image data is used to enhance high-frequency information in the image data to be augmented; wherein image augmentation may include, but is not limited to, at least one of guided filtering, image edge enhancement, image contrast enhancement, and histogram equalization. The image data to be augmented and the augmented image data are fused to obtain second image data, wherein the resolution of the second image data is equal to or less than the resolution of the first image data.

[0043] For example, AI processing may include, but is not limited to, at least one of image transformation, image augmentation, and semantic transformation. If AI processing includes semantic transformation, then obtaining second image data by AI processing the first image data through an AI model may include, but is not limited to: inputting the first image data into an image processing neural network in the AI ​​model, operating the first image data through the image processing neural network to obtain image data to be enhanced, wherein the resolution of the image data to be enhanced is equal to or less than the resolution of the first image data, and the number of channels of the image data to be enhanced is equal to or less than the number of channels of the first image data; inputting the first image data into a semantic processing neural network in the AI ​​model, operating the first image data through the semantic processing neural network to obtain structured semantic information, wherein the structured semantic information is used to describe the semantics of the image data to be enhanced; and generating second image data based on the image data to be enhanced and the structured semantic information, i.e., the second image data may include the image data to be enhanced and the structured semantic information.

[0044] For example, AI processing may include, but is not limited to, at least one of image transformation, image augmentation, and semantic transformation. If AI processing includes image transformation, image augmentation, and semantic transformation simultaneously, then obtaining second image data by performing AI processing on the first image data using an AI model may include, but is not limited to: performing image transformation on the first image data using an AI model to obtain transformed image data, where the image transformation is used to separate high-frequency and low-frequency information in the first image data; inputting the transformed image data into an image processing neural network in the AI ​​model, where the image processing neural network operates on the transformed image data to obtain first image data to be enhanced, where the image processing neural network is used to enhance the high-frequency information in the first image data; performing image augmentation on the first image data using an AI model to obtain augmented image data, where the augmented image data is used to enhance the high-frequency information in the first image data to be enhanced; fusing the first image data to be enhanced and the augmented image data to obtain second image data to be enhanced; and inputting the first image data into a semantic processing neural network in the AI ​​model, where the semantic processing neural network operates on the first image data to obtain structured semantic information, where the structured semantic information is used to describe the semantics of the second image data to be enhanced. The second image data is generated based on the second image data to be enhanced and the structured semantic information, that is, the second image data includes the second image data to be enhanced and the structured semantic information.

[0045] For example, if the first image data includes K target image blocks, and the second image data includes K image blocks to be stored corresponding to the K target image blocks, where K is a positive integer greater than 1; for each image block to be stored, the image block to be stored corresponds to a storage start address, and the interval between the storage start addresses of two adjacent image blocks to be stored is the width of the image block to be stored; the second image data corresponds to a storage span, and the storage span is greater than or equal to the sum of the widths of the K image blocks to be stored. Based on this, storing the second image data in the specified storage medium may include, but is not limited to: for each image block to be stored, when storing the i-th row of data of the image block to be stored in the specified storage medium, storing the i-th row of data sequentially starting from the target storage location; wherein, the target storage location may be determined by the following formula: p = w + stride * (i - 1); where p can represent the target storage location, w can represent the storage start address corresponding to the image block to be stored, stride can represent the storage span, i can represent the number of rows of the image block to be stored, and i is a positive integer.

[0046] As can be seen from the above technical solutions, in this embodiment, after acquiring the original image, the front-end device performs preprocessing on the original image to obtain first image data. Then, it uses an AI model to perform AI processing on the first image data to obtain second image data, whereby the AI ​​processing enhances the high-frequency information in the first image data. Based on this, the front-end device encodes and compresses the image data to obtain a target bitstream, which is then sent to the back-end device. The back-end device decodes the target bitstream to obtain the target image. Because the high-frequency information in the image data is enhanced by the AI ​​model, even if the encoding and compression process reduces the image quality of the image data (loss of high-frequency information leads to reduced image quality), the image quality of the target image obtained by the back-end device decoding the target bitstream is equivalent to that of the original image, thereby improving the image quality of the stored or displayed target image. By using an AI model deployed on the front-end device to process the real-time data stream (i.e., video stream), the image quality of the real-time data stream can be improved, helping video image encoding and decoding achieve higher video image quality.

[0047] The technical solutions described above in the embodiments of this application will be explained below in conjunction with specific application scenarios.

[0048] This application proposes a video image encoding and decoding system, which may include a front-end device (the front-end device acts as the video encoding device of the video image encoding and decoding system) and a back-end device (the back-end device acts as the video decoding device of the video image encoding and decoding system). The front-end device may also be called an edge device. Edge devices are distinct from cloud devices. Cloud devices can use server resources to implement AI, while edge devices use local resources to implement AI and do not use server resources.

[0049] For example, edge devices can include smartphones, IoT devices, cameras, in-vehicle devices, autonomous driving devices, access control devices, conference tablets, smart interactive all-in-one machines, etc. There are no restrictions on the type of edge device; any device with image acquisition and image transmission functions can be called an edge device.

[0050] Backend devices are devices that communicate with frontend devices. They can receive images from frontend devices and then process them. Backend devices can be storage devices (devices used to store images) or display devices (devices used to display images), and there are no restrictions on the type of backend device.

[0051] In the above application scenarios, this application proposes an image processing method, see [link to relevant documentation]. Figure 2 The diagram shown is a flowchart of the image processing method, which may include:

[0052] Step 201: The front-end device acquires the original image.

[0053] For example, the front-end device can acquire real-time data streams, which can be video streams. Each frame of the real-time data stream can be used as the original image, or a portion of the images in the real-time data stream can be used as the original image. There are no restrictions on the source of the original image.

[0054] Step 202: The front-end device pre-compresses the original image based on the configuration information to obtain image data.

[0055] For example, an AI model can be configured on the front-end device to pre-compress the original image based on the configuration information to obtain image data. Pre-compressing the original image yields image data that is more conducive to subsequent encoding, compression, and compression recovery. Pre-compressing the original image enhances high-frequency information in the image data, and even if the encoding and compression process reduces the image quality (by losing high-frequency information), the image quality of the target image can still be improved when the back-end device decodes the target bitstream to obtain the target image.

[0056] By configuring AI models on front-end devices instead of back-end devices, the AI ​​models can be executed on the front-end devices to pre-compress the original images. This helps to process or make decisions on real-time data (original images) at the data source or on front-end devices that are "close" to the data, thereby reducing the transmission of data between front-end and back-end devices, reducing data transmission latency, and lowering network bandwidth requirements.

[0057] When configuring an AI model on a front-end device, the AI ​​model can be lightweight and efficient, thus adapting to the needs of front-end devices with limited computing resources. Lightweight means that the amount of computation and memory required during the AI ​​model inference process is relatively low, while still maintaining a certain level of model performance.

[0058] Step 203: The front-end device encodes and compresses the image data to obtain the target bitstream.

[0059] For example, the front-end device can be used as a video encoding device in a video image encoding and decoding system. It can be pre-configured with an encoding and compression algorithm. The front-end device can use the encoding and compression algorithm to encode and compress the image data to obtain the target bitstream. In this embodiment, there are no restrictions on this encoding and compression process.

[0060] Step 204: The front-end device sends the target bitstream to the back-end device.

[0061] Step 205: The backend device decodes the target bitstream to obtain the decoded data.

[0062] For example, the backend device can be a video decoding device in a video image encoding and decoding system. It can be pre-configured with a decoding algorithm. After receiving the target bitstream, the backend device can use the decoding algorithm to decode the target bitstream to obtain the decoded data. In this embodiment, there are no restrictions on this decoding process.

[0063] Step 206: The backend device compresses and restores the decoded data to obtain the target image.

[0064] For example, an AI model can be configured on the backend device. The AI ​​model can be used to compress and restore the decoded data to obtain the target image. Compression and restoration can be the reverse process of pre-compression. That is, the AI ​​model of the backend device is adapted to the AI ​​model of the frontend device. The AI ​​model of the frontend device is used to implement pre-compression, while the AI ​​model of the backend device is used to implement compression and restoration. By compressing and restoring the decoded data, the target image after compression and restoration is adapted to the original image, that is, the target image is the same as or approximately the same as the original image.

[0065] Step 207: The backend device stores or displays the target image. For example, if the backend device is a storage device, it can store the target image; if the backend device is a display device, it can display the target image.

[0066] In one possible implementation, for step 202, see [link to relevant documentation]. Figure 3 The diagram shows a pre-compression process. The original image can be pre-compressed to obtain image data using the following steps:

[0067] Step 301: The front-end device performs preprocessing on the original image to obtain the first image data.

[0068] For example, preprocessing is used to convert the original image into first image data adapted to the AI ​​model based on configuration information, such that the first image data can be the image data required by the AI ​​model.

[0069] For example, preprocessing of the original image refers to processing the original image before encoding and transmission, such as downsampling and denoising. Preprocessing reduces transmission bandwidth and improves image quality. In this embodiment, preprocessing the original image means converting it into first image data adapted to the AI ​​model. Preprocessing may include, but is not limited to, at least one of color space transformation, pixel value mapping, image block partitioning, and image rearrangement. The above are just a few examples of preprocessing and are not limited thereto; the goal is simply to convert the original image into first image data adapted to the AI ​​model.

[0070] For example, the original image includes image data from at least two channels, or at least three channels. For instance, the original image may include image data from a luminance (Luma) channel and image data from a chroma (Chroma) channel. Or, for example, the original image may include image data from a red (R) channel, a green (G) channel, and a blue (B) channel.

[0071] Scenario 1: If the first color space of the original image is a color space supported by the AI ​​model, then preprocessing does not include color space transformation. If the first color space of the original image is a color space not supported by the AI ​​model, then preprocessing includes color space transformation. Based on this, if preprocessing includes color space transformation, then the second color space supported by the AI ​​model is determined, and the original image is subjected to color space transformation to obtain image data in the second color space. For example, when performing color space transformation, the resolution of the image data in the second color space can be the same as the resolution of the original image in the first color space, and the number of channels in the image data in the second color space can be the same as the number of channels in the original image in the first color space.

[0072] For example, the configuration information may include a color space conversion formula, which is a conversion formula between a first color space and a second color space. This color space conversion formula is used to convert the original image in the first color space into image data in the second color space. Based on this, image data in the second color space can be obtained by performing a color space transformation on the original image using the color space conversion formula.

[0073] For example, the original image's primary color space is RGB, while the AI ​​model does not support RGB but supports YUV. Therefore, the original image can be transformed into YUV color space image data.

[0074] For example, the original image's primary color space is YUV, but the AI ​​model does not support YUV and does support RGB. Therefore, the original image can be transformed into RGB color space.

[0075] Scenario 2: If the first pixel value distribution of the original image is a pixel value distribution supported by the AI ​​model, then preprocessing does not include pixel value mapping. If the first pixel value distribution of the original image is a pixel value distribution not supported by the AI ​​model, then preprocessing includes pixel value mapping. Based on this, if preprocessing includes pixel value mapping, then the second pixel value distribution supported by the AI ​​model is determined, and pixel value mapping is performed on the original image (or image data in the second color space) with the first pixel value distribution to obtain image data with the second pixel value distribution.

[0076] For example, when performing pixel value mapping, the resolution of the image data of the second pixel value distribution can be the same as the resolution of the original image of the first pixel value distribution, and the number of channels of the image data of the second pixel value distribution can be the same as the number of channels of the original image of the first pixel value distribution.

[0077] For example, the configuration information may include a pixel value conversion formula, which is a conversion formula between a first pixel value distribution and a second pixel value distribution. This formula is used to convert the original image with the first pixel value distribution into image data with the second pixel value distribution. Based on this, the original image can be pixel-mapped using the pixel value conversion formula to obtain image data with the second pixel value distribution.

[0078] For example, the first pixel value distribution of the original image is 0~255, but the AI ​​model does not support the pixel value distribution of 0~255, and the AI ​​model supports the second pixel value distribution of 0~1. Therefore, pixel value mapping (such as pixel value scaling operation) can be performed on the original image to scale the pixel values ​​of 0~255 to pixel values ​​of 0~1. The scaling method can use the pixel value conversion formula in the configuration information, and there are no restrictions on this process.

[0079] For example, the first pixel value distribution of the original image is 0~255. The AI ​​model does not support the pixel value distribution of 0~255, but it supports the second pixel value distribution of -127~128. Therefore, pixel value mapping (such as pixel value translation operation) can be performed on the original image to translate the pixel values ​​of 0~255 to pixel values ​​of -127~128. The translation method can use the pixel value conversion formula in the configuration information, and there are no restrictions on this process.

[0080] Scenario 3: If the first arrangement of the original image is a layout supported by the AI ​​model, then preprocessing does not include image rearrangement. If the first arrangement of the original image is a layout not supported by the AI ​​model, then preprocessing includes image rearrangement. Based on this, if preprocessing includes image rearrangement, then the second arrangement supported by the AI ​​model is determined, and the original image (or image data in the second color space, or image data with the second pixel value distribution) in the first arrangement is rearranged to obtain image data in the second arrangement.

[0081] For example, when performing image rearrangement, the resolution of the image data in the second arrangement can be the same as or different from the resolution of the original image in the first arrangement, and the number of channels of the image data in the second arrangement can be the same as or different from the number of channels of the original image in the first arrangement.

[0082] For example, the configuration information may include an image rearrangement method, which is a rearrangement method between a first arrangement method and a second arrangement method, and the image rearrangement method is used to rearrange the original image of the first arrangement method into image data of the second arrangement method. Based on this, the original image of the first arrangement method can be rearranged according to the image rearrangement method to obtain image data of the second arrangement method.

[0083] For example, the first arrangement of the original image is interlaced RGB image data, but the AI ​​model does not support interlaced RGB image data. The second arrangement supported by the AI ​​model is tiled RGB image data. Therefore, the interlaced RGB image data can be transformed into tiled RGB image data according to the image rearrangement method. For example, RGBRGB…RGB can be rearranged into RR..RGG..GBB..B.

[0084] Scenario 4: If the AI ​​model does not support parallel processing, the preprocessing does not include image block partitioning. If the AI ​​model supports parallel processing, the preprocessing may or may not include image block partitioning. Based on this, if the preprocessing includes image block partitioning, the original image (or image data in the second color space, or image data with the second pixel value distribution, or image data with the second arrangement) can be partitioned into multiple initial image blocks. These initial image blocks can include edge blocks and non-edge blocks. Edge blocks can include the boundaries of the original image, while non-edge blocks do not include the boundaries of the original image.

[0085] For each initial image block, pixel padding can be performed to obtain the corresponding target image block, thus resulting in multiple target image blocks. Alternatively, for each initial image block, it can be used as the corresponding target image block.

[0086] For example, see Figure 4AThe diagram illustrates pixel filling of a non-edge block, where the non-edge block is the initial image block excluding the boundaries of the original image. For the target image block corresponding to the non-edge block, the target image block can include a tile region and an overlap region. The tile region is the initial image block itself, and the overlap region is the pixel-filled region, obtained by filling the overlap region with the pixel values ​​of the original image in that region. Thus, the initial image block and the overlap region together form the target image block.

[0087] For example, see Figure 4B The diagram illustrates pixel filling of an edge block, where the edge block is the initial image block including the boundary of the original image. For the target image block corresponding to the edge block, the target image block can include a tile region, an overlap region, and a padding region. The tile region is the initial image block itself, while the overlap and padding regions are pixel-filled regions. For the overlap region (i.e., the internal region of the original image), the overlap region is filled based on the pixel values ​​of the original image within that region. For the padding region (i.e., the external region of the original image), the padding region can be filled based on pre-configured fixed pixel values, or based on the boundary pixel values ​​of the original image, or other methods; there are no restrictions on the filling method. In this way, the initial image block, the overlap region, and the padding region can form the target image block.

[0088] See Figure 4A and Figure 4B As shown, the initial image patch has a size of h*w, and the target image patch has a size of (h+r)*(w+r). The configuration information includes the partition size and the padding size. The partition size indicates the size of the initial image patch, and the padding size indicates the size of the pixel-filled area. Based on this, the original image can be divided into initial image patches of size h*w based on the partition size. Based on the padding size, the initial image patches of size h*w can be padded with pixels to obtain the target image patch of size (h+r)*(w+r).

[0089] For example, the initial image patch or the target image patch can be a square image patch or a rectangular image patch. For each target image patch, the resolution of the target image patch is smaller than the resolution of the original image.

[0090] For example, after dividing the image into multiple target image blocks, these blocks are used as input data for an AI model, allowing the model to process them in parallel. Based on this approach, by dividing the original image into relatively small target image blocks, each block can be processed independently, thus enabling parallel processing and improving the processing efficiency of the AI ​​model.

[0091] This completes step 301, where the first image data is obtained by preprocessing the original image.

[0092] Step 302: The front-end device performs AI processing on the first image data using an AI model to obtain the second image data. The AI ​​processing is used to enhance the high-frequency information in the first image data based on the configuration information.

[0093] For example, an AI model is configured on the front-end device, and the AI ​​model performs AI processing on the first image data based on the configuration information to obtain the second image data. By performing AI processing on the first image data, the second image data is obtained that is more conducive to subsequent encoding, compression, and compression recovery. By performing AI processing on the first image data, high-frequency information in the first image data can be enhanced to obtain the second image data.

[0094] For example, high-frequency information refers to regions of dramatic change in the first image data, areas of rapid local change, as opposed to low-frequency information (smooth regions in the first image data), such as edges, textures, and details. High-frequency information is crucial for image sharpness, edge sharpness, and texture detail. Enhancing high-frequency information can improve image sharpness and help identify image edges and texture features.

[0095] Based on this, in this embodiment, the high-frequency information in the first image data is enhanced by an AI model. In this way, even if the encoding and compression process reduces the image quality of the image data (it will lose the high-frequency information in the image data, resulting in a decrease in image quality), the image quality of the target image can still be improved.

[0096] For example, the AI ​​model includes at least one convolutional layer and one activation layer. There are no restrictions on the network structure of the AI ​​model, as long as the AI ​​model can enhance the high-frequency information in the first image data.

[0097] For example, when obtaining second image data by AI processing of first image data using an AI model, the AI ​​processing can be real-time. Real-time processing means that the input data of the AI ​​model (i.e., the first image data) is streamed, and the AI ​​model immediately processes the first image data after receiving it, and outputs the inference result (i.e., the second image data) within a short time. Real-time processing requires the AI ​​model to have low latency in inference and be able to complete the processing of input data within a limited time.

[0098] For example, the first image data can be image data of RGB channels, and AI processing can be performed on at least one of the RGB channels using an AI model, such as performing AI processing on the RGB channels simultaneously.

[0099] The first image data can be image data from the YUV channels. AI processing can be performed on at least one of the YUV channels using an AI model, such as performing AI processing on the YUV channels simultaneously.

[0100] In this embodiment, AI processing of the first image data using an AI model refers to enhancing the high-frequency information in the first image data to obtain the second image data. AI processing can include, but is not limited to, at least one of image transformation, image augmentation, and semantic transformation. The above are just a few examples of AI processing and are not intended to limit the scope, as long as it can enhance the high-frequency information in the first image data.

[0101] Case 1: If the AI ​​processing includes image transformation, see [reference needed]. Figure 5A The diagram illustrates how an AI model transforms first image data to obtain second image data. For example, the input data to the AI ​​model might be the first image data and configuration information, while the output data is the second image data.

[0102] For example, the first image data is transformed by an AI model to obtain the transformed image data. The resolution of the transformed image data can be equal to or less than the resolution of the first image data, and the number of channels of the transformed image data can be equal to or greater than the number of channels of the first image data.

[0103] For example, the first image data includes at least two channels of image data, such as a green channel image data a1, a red channel image data a2, and a blue channel image data a3. After image transformation of the first image data using an AI model, the transformed image data includes at least one green channel image data b1, meaning the number of channels in the green channel image data increases or remains unchanged, and the resolution of image data b1 is equal to or less than the resolution of image data a1. The transformed image data also includes at least one red channel image data b2, meaning the number of channels in the red channel image data increases or remains unchanged, and the resolution of image data b2 is equal to or less than the resolution of image data a2. The transformed image data also includes at least one blue channel image data b3, meaning the number of channels in the blue channel image data increases or remains unchanged, and the resolution of image data b3 is equal to or less than the resolution of image data a3.

[0104] When performing image transformation on the first image data using an AI model, the configuration information can include the image transformation method, allowing the first image data to be transformed based on that method. When performing image transformation on image data from multiple channels, the image transformation methods for different channels can be the same or different. For example, image data a1, a2, and a3 can be transformed using the same image transformation method. Alternatively, image transformation method 1 can be used to transform image data a1, image transformation method 2 can be used to transform image data a2, and image transformation method 3 can be used to transform image data a3.

[0105] When performing image transformation on the first image data using an AI model to obtain transformed image data, the image transformation is used to separate high-frequency and low-frequency information in the first image data. There are no restrictions on the image transformation method, as long as it can separate high-frequency and low-frequency information in the first image data.

[0106] For example, image transformation can include, but is not limited to, at least one of the following: interpolation transformation (such as nearest neighbor interpolation, mean transformation, bilinear interpolation, adaptive interpolation, bicubic interpolation, etc.), wavelet transform, Gaussian pyramid transform, Laplacian pyramid transform, and Sobel filtering transform. There is no restriction on the image transformation method. Based on this, the configuration information can include the image transformation method, which indicates whether the image transformation is implemented using nearest neighbor interpolation, or indicates whether the image transformation is implemented using bilinear interpolation, and so on. There is no restriction on the content of this configuration information.

[0107] For example, if the image transformation method specified in the configuration information is bilinear interpolation, the transformed image data is obtained by performing bilinear interpolation on the first image data using an AI model. If the image transformation method specified in the configuration information is Sobel filtering, the transformed image data is obtained by performing Sobel filtering on the first image data using an AI model. If the image transformation method specified in the configuration information is both bilinear interpolation and Gaussian pyramid transformation (i.e., using multiple transformations simultaneously), the transformed image data is obtained by performing both bilinear interpolation and Gaussian pyramid transformation on the first image data using an AI model.

[0108] For example, after obtaining the transformed image data, it can be input into the image processing neural network in the AI ​​model. The image processing neural network then processes the transformed image data to obtain the second image data. For instance, the AI ​​model includes an image processing neural network, which contains at least one convolutional layer and one activation layer; the network structure of this image processing neural network is not limited. When processing the transformed image data through the image processing neural network, convolution and activation operations can be performed on the transformed image data; the type of operation is not limited.

[0109] For example, the input data of an image processing neural network is transformed image data, which includes at least two channels of image data. The output data of the image processing neural network is second image data, which also includes at least two channels of image data. The input data of an AI model is first image data, which also includes at least two channels of image data. Furthermore, the resolution of the second image data can be equal to or less than the resolution of the first image data.

[0110] For example, the first image data includes image data a1 in the green channel, image data a2 in the red channel, and image data a3 in the blue channel. The second image data includes image data c1 in the green channel and image data c2 in the red channel. Then, the resolution of image data c1 is equal to or less than the resolution of image data a1, and the resolution of image data c2 is equal to or less than the resolution of image data a2. Alternatively, the second image data includes image data c1 in the green channel and image data c3 in the blue channel. Then, the resolution of image data c1 is equal to or less than the resolution of image data a1, and the resolution of image data c3 is equal to or less than the resolution of image data a3. Or, the second image data includes image data c2 in the red channel and image data c3 in the blue channel. Then, the resolution of image data c2 is equal to or less than the resolution of image data a2, and the resolution of image data c3 is equal to or less than the resolution of image data a3.

[0111] For example, when operating on transformed image data using an image processing neural network, the image processing neural network is used to enhance the high-frequency information in the first image data. For instance, the image transformation has already separated high-frequency and low-frequency information in the first image data, such that the transformed image data includes both the high-frequency and low-frequency information from the first image data. In this way, the high-frequency information in the first image data can be enhanced using the image processing neural network; that is, when operating on the transformed image data using the image processing neural network, the high-frequency information in the first image data can be enhanced.

[0112] Scenario 2: If AI processing includes image augmentation, see [reference needed]. Figure 5B The diagram illustrates how an AI model augments first image data to obtain second image data. For example, the input data to the AI ​​model might be the first image data and configuration information, while the output data is the second image data.

[0113] For example, the first image data can be input into the image processing neural network in the AI ​​model, and the image processing neural network can operate on the first image data to obtain the image data to be enhanced. For instance, the AI ​​model may include an image processing neural network, which includes at least one convolutional layer and one activation layer. When operating on the first image data through the image processing neural network, convolutional operations and activation operations can be performed on the first image data, and there is no restriction on the type of these operations.

[0114] The image data to be enhanced includes at least two channels of image data, and the first image data also includes at least two channels of image data. The resolution of the image data to be enhanced is equal to or less than the resolution of the first image data, and the number of channels in the image data to be enhanced is equal to or less than the number of channels in the first image data.

[0115] For example, the input data of an image processing neural network is first image data, and the output data of the image processing neural network is image data to be enhanced. The first image data includes image data a1 (green channel), image data a2 (red channel), and image data a3 (blue channel). The image data to be enhanced includes image data d1 (green channel) and image data d2 (red channel), where the resolution of image data d1 is equal to or less than the resolution of image data a1, and the resolution of image data d2 is equal to or less than the resolution of image data a2. Alternatively, the image data to be enhanced may include image data d1 (green channel) and image data d3 (blue channel), where the resolution of image data d1 is equal to or less than the resolution of image data a1, and the resolution of image data d3 is equal to or less than the resolution of image data a3. Or, the image data to be enhanced may include image data d2 (red channel) and image data d3 (blue channel), where the resolution of image data d2 is equal to or less than the resolution of image data a2, and the resolution of image data d3 is equal to or less than the resolution of image data a3.

[0116] For example, the first image data is augmented using an AI model to obtain augmented image data. The resolution of the augmented image data can be equal to or less than the resolution of the first image data, and the number of channels in the augmented image data can be equal to or less than the number of channels in the first image data.

[0117] For example, the first image data includes at least two channels of image data, such as image data a1 in the green channel, image data a2 in the red channel, and image data a3 in the blue channel. The expanded image data includes at least one channel of image data, such as image data e1 in the green channel (the resolution of image data e1 is equal to or less than the resolution of image data a1), or image data e2 in the red channel (the resolution of image data e2 is equal to or less than the resolution of image data a2), or image data e3 in the blue channel (the resolution of image data e3 is equal to or less than the resolution of image data a3), or image data e4 in the feature channel (the resolution of image data e4 is equal to or less than the resolution of any of the image data).

[0118] When performing image augmentation on the first image data using an AI model, the configuration information may include the image augmentation method, so that the first image data can be augmented based on the image augmentation method.

[0119] When the first image data is expanded using an AI model to obtain expanded image data, the image expansion is used to enhance the high-frequency information in the first image data. Thus, the expanded image data is used to enhance the high-frequency information in the image data to be enhanced. In other words, the expanded image data is the enhanced data for the high-frequency information in the first image data. In this embodiment, there is no limitation on the image expansion method, as long as it can enhance the high-frequency information in the first image data.

[0120] For example, image augmentation may include, but is not limited to, at least one of guided filtering, image edge enhancement, image contrast enhancement, and histogram equalization; there is no limitation on the image augmentation method. Based on this, the configuration information may include the image augmentation method, which indicates whether guided filtering or image edge enhancement is used for image augmentation, and so on.

[0121] For example, if the image augmentation method in the configuration information indicates the use of histogram equalization, then the augmented image data can be obtained by performing histogram equalization on the first image data using an AI model. If the image augmentation method in the configuration information indicates the use of image edge enhancement, then the augmented image data can be obtained by performing image edge enhancement on the first image data using an AI model, and so on.

[0122] Taking the guided filtering of the first image data using an AI model to obtain expanded image data as an example, guided filtering based on image data a1 from the green channel and image data a2 from the red channel yields expanded image data for the blue channel. This expanded image data is used to enhance the high-frequency information in the blue channel image data to be enhanced, thereby introducing information from the green and red channels into the blue channel and enhancing its high-frequency information. Similarly, guided filtering based on image data a1 from the green channel and image data a3 from the blue channel yields expanded image data for the red channel. This expanded image data is used to enhance the high-frequency information in the red channel image data to be enhanced, and so on.

[0123] For example, after obtaining the image data to be enhanced and the expanded image data, the two images can be fused to obtain a second image data. For instance, the configuration information may include an image fusion method, allowing the image data to be enhanced and the expanded image data to be fused based on this method. For example, if there is image data to be enhanced and expanded image data with a blue channel, the two images can be fused based on this image fusion method to obtain a second image data with a blue channel, and so on.

[0124] Obviously, since the expanded image data of the blue channel can enhance the high-frequency information in the image data to be enhanced in the blue channel, the high-frequency information can be enhanced by fusing the image data to be enhanced and the expanded image data. That is, the second image data is the enhanced image data.

[0125] For example, image fusion can include, but is not limited to, at least one of weighted fusion, stitching fusion, multiplicative fusion, attention fusion, additive fusion, and convolutional fusion. Based on this, the configuration information can include the image fusion method, which indicates whether weighted fusion or attention fusion is used, and so on.

[0126] For example, if the image fusion method in the configuration information indicates the use of weighted fusion, then the image data to be enhanced and the expanded image data can be weighted and fused to obtain the second image data. If the image fusion method in the configuration information indicates the use of attention fusion, then the image data to be enhanced and the expanded image data can be attention-fused to obtain the second image data, and so on.

[0127] For example, after obtaining the second image data, the second image data includes image data from at least two channels, and the resolution of the second image data may be equal to or less than the resolution of the first image data.

[0128] Scenario 3: If the AI ​​processing includes semantic transformation, see [reference needed]. Figure 5C The diagram illustrates how an AI model performs semantic transformation on first image data to obtain second image data. For example, the input data to the AI ​​model can be the first image data, and the output data can be the second image data.

[0129] For example, the first image data can be input into the image processing neural network in the AI ​​model, and the image processing neural network can operate on the first image data to obtain the image data to be enhanced. For instance, the AI ​​model may include an image processing neural network, which includes at least one convolutional layer and one activation layer. When operating on the first image data through the image processing neural network, convolutional operations and activation operations can be performed on the first image data, and there is no restriction on the type of these operations.

[0130] The image data to be enhanced includes at least two channels of image data, and the first image data also includes at least two channels of image data. The resolution of the image data to be enhanced is equal to or less than the resolution of the first image data, and the number of channels in the image data to be enhanced is equal to or less than the number of channels in the first image data.

[0131] For example, the first image data can be input into a semantic processing neural network in an AI model. The semantic processing neural network operates on the first image data to obtain structured semantic information. For instance, the AI ​​model may include a semantic processing neural network, which includes at least one convolutional layer and one activation layer; the network structure is not limited. When operating on the first image data through the semantic processing neural network, convolutional and activation operations can be performed on the first image data.

[0132] For example, a semantic processing neural network can be pre-trained. The input data of the semantic processing neural network is the first image data, and the output data of the semantic processing neural network is structured semantic information. Therefore, after the first image data is input into the semantic processing neural network, the semantic processing neural network can process the first image data to obtain the structured semantic information corresponding to the first image data.

[0133] For example, structured semantic information is used to describe the semantics of the image data to be enhanced (or the first image data), and high-frequency information of the image data to be enhanced can be recovered through structured semantic information.

[0134] Structured semantic information, also known as image semantic information, refers to the content and meaning contained in the image data to be enhanced (or the first image data), such as the type of object, the object's action or state, the theme or context expressed by the image, and the main subject in the image (such as people, animals, buildings, or natural landscapes). Structured semantic information can be expressed through language, including natural language and symbolic language (mathematical language).

[0135] For example, after obtaining the image data to be enhanced and the structured semantic information, a second image data can be generated based on the image data to be enhanced and the structured semantic information. That is, the second image data may include the image data to be enhanced and the structured semantic information. The structured semantic information can be placed before the image data to be enhanced or after the image data to be enhanced.

[0136] Scenario 4: If the AI ​​processing simultaneously includes image transformation, image augmentation, and semantic transformation, see [reference needed]. Figure 5D The diagram shows how an AI model processes the first image data to obtain the second image data.

[0137] The first image data is transformed using an AI model to obtain transformed image data. This image transformation separates high-frequency and low-frequency information in the first image data. The transformed image data is then input into the image processing neural network within the AI ​​model. The image processing neural network operates on the transformed image data to obtain the first image data to be enhanced. This image processing neural network is used to enhance the high-frequency information in the first image data. This process is similar to Case 1 and will not be elaborated further here.

[0138] The first image data is augmented using an AI model to obtain augmented image data, which is then used to enhance high-frequency information in the first image data to be augmented. The first image data to be augmented (i.e., the image data output by the image processing neural network) and the augmented image data are then fused to obtain the second image data to be augmented. This process can be referred to in Case 2 and will not be elaborated here.

[0139] The first image data is input into the semantic processing neural network in the AI ​​model. The semantic processing neural network operates on the first image data to obtain structured semantic information. The structured semantic information is used to describe the semantics of the second image data to be enhanced. This process can be referred to in Case 3, and will not be repeated here.

[0140] The second image data is generated based on the second image data to be enhanced and the structured semantic information. That is, the second image data may include the second image data to be enhanced and the structured semantic information.

[0141] At this point, step 302 is complete, and the second image data is obtained by performing AI processing on the first image data.

[0142] Step 303: The front-end device performs post-processing on the second image data.

[0143] For example, post-processing of image data refers to further processing of image data after encoding, transmission, and decoding, such as decompression distortion, noise reduction, and super-resolution, to improve the subjective quality of the image. In this embodiment, post-processing of the second image data means storing the second image data in a specified storage medium (such as memory or hard disk). For example, the second image data is a serial image data stream, which is rearranged according to certain rules before storage.

[0144] For example, in the storage process of the second image data, the second image data (i.e., the output data of the AI ​​model) is a serial image data stream. The serial image data stream can be rearranged according to certain rules and stored in a specified storage medium (such as memory or hard disk), thereby facilitating access and reading in subsequent processes. In this way, the stored data is image data arranged according to certain rules.

[0145] For example, if image block division is not performed during preprocessing, the first image data is a complete image. When AI processing is performed on the first image data to obtain the second image data, the second image data is also a complete image. Based on this, when storing the second image data in a specified storage medium, each row of data can be stored sequentially. For example, all the data in the first row can be stored first, then all the data in the second row, and so on, until all the data in the last row is stored.

[0146] For example, during preprocessing, if image blocks are divided, the first image data includes K target image blocks, where K is a positive integer greater than 1. When the first image data is processed by AI to obtain the second image data, the second image data includes K image blocks to be stored corresponding to the K target image blocks. For instance, the K target image blocks are processed by AI in parallel, as shown in step 302. When each target image block is processed by AI, the corresponding image block to be stored is obtained. Thus, all the image blocks to be stored constitute the second image data, that is, the second image data includes K image blocks to be stored.

[0147] When storing the second image data in the specified storage medium, for each image block to be stored, there is a corresponding storage start address, and the interval between the storage start addresses of two adjacent image blocks to be stored is the width of the image block to be stored. Furthermore, the second image data corresponds to a storage span, and this storage span is greater than or equal to the sum of the widths of the K image blocks to be stored. Based on this, for each image block to be stored, when storing the i-th row of data in the specified storage medium, the i-th row of data is stored sequentially starting from the target storage location; where the target storage location can be determined using the following formula: p = w + stride * (i-1); where p represents the target storage location, w represents the storage start address corresponding to the image block to be stored, stride represents the storage span, and i can represent the number of rows in the image block to be stored, i being a positive integer.

[0148] See Figure 6AThe diagram illustrates the storage of multiple image blocks (denoted as image blocks A, B, ..., N) in a specified storage medium. First, the serial data of image block A is stored, such as storing the first row of data (A1A1A1), the second row (A2A2A2), ..., the h-th row (AhAhAh) sequentially. Then, the serial data of image block B is stored, such as storing the first row of data (B1B1B1), the second row (B2B2B2), ..., the h-th row (BhBhBh) sequentially. This process continues until the last image block N is stored, such as storing the first row of data (N1N1N1), the second row (N2N2N2), ..., the h-th row (NhNhNh) sequentially. During the encoding and compression process, the first row of data (A1A1A1) of the image block to be stored is read first, followed by the first row of data (B1B1B1), ..., and then the first row of data (N1N1N1) of the image block to be stored. This yields the complete first row of data. Then, the second row of data (A2A2A2), followed by the second row of data (B2B2B2), ..., and then the second row of data (N2N2N2) of the image block to be stored is read. This process continues until the h-th row of data (AhAhAh) of the image block to be stored, the h-th row of data (BhBhBh), ..., and then the h-th row of data (NhNhNh) of the image block to be stored is read, yielding the complete h-th row of data.

[0149] Taking N image blocks to be stored as examples where there are two image blocks to be stored (denoted as tile A and tile B), see [link to example]. Figure 6B The diagram illustrates the storage method for two tiles. For example, the configuration information includes the starting address of each image block to be stored, such as the starting address of tile A (addA) and the starting address of tile B (addB). The interval between the starting addresses of two adjacent image blocks to be stored can be the width of the previous image block. For example, the interval between addB and addA can be the width wA of tile A. If tile C follows tile B, the interval between the starting address of tile C (addC) and addB can be the width wB of tile B, and so on. The configuration information also includes the storage span (stride), which is greater than or equal to the sum of the widths of all image blocks to be stored. For example, the storage span (stride) is greater than or equal to the sum of the widths wA of tile A and wB of tile B, i.e., stride ≥ wA + wB.

[0150] See Figure 4A and Figure 4B As shown, the target image block includes a tile region and a pixel-filled region. Therefore, the image block to be stored also includes a tile region and a pixel-filled region. When storing multiple image blocks to be stored, only the tile region of the image block to be stored is stored, and the pixel-filled region of the image block to be stored is not stored.

[0151] Target image blocks A and B can be processed in parallel. The processing of target image block A is denoted as AI model processing 1, resulting in the serial data A1A1A1A2A2A2…AhAhAh of tile A, which is then stored. For example, the target storage location for each row is determined, such as p = w + stride * (i-1), where p represents the target storage location, w represents the starting storage address addA of tile A, and i represents the row number. Starting from the target storage location (addA) of the first row, the first row of data A1A1A1 of the tile region (excluding pixel fill areas) of tile A is stored. Starting from the target storage location (addA + stride) of the second row, the second row of data A2A2A2 of the tile region of tile A is stored. Starting from the target storage location (addA + stride * 2) of the third row, the third row of data A2A2A2 of the tile region of tile A is stored, and so on.

[0152] Furthermore, the processing of target image block B is denoted as AI model processing 2, which can obtain the serial data B1B1B1B2B2B2…BhBhBh of tile B and store the serial data of tile B. For example, the target storage location of each row is determined, such as p=w+ stride*(i-1), where w represents the storage starting address addB of tile B. Starting from the target storage location (addB) of the first row, the first row of data B1B1B1 of the tile area of ​​tile B (excluding pixel filling areas) is stored; starting from the target storage location (addB+ stride) of the second row, the second row of data B2B2B2 of the tile area of ​​tile B is stored, and so on.

[0153] Using the above storage method, the two image blocks divided in the preprocessing can be reassembled into the image before division (the size and shape of the image are restored to the state before division, but the image data has been processed by the AI ​​model). Figure 6B The example only illustrates the storage method for two image blocks, but by configuring the starting address and storage span of each image block to be stored, it can be extended to the case of storing multiple image blocks to be stored, which will not be elaborated here.

[0154] In one possible implementation, image display can also be performed during post-processing of the second image data, see [link to relevant documentation]. Figure 6C The diagram illustrates an image display process. To achieve image display, the configuration information can include information on changing the image pixel value scale and shifting the image pixel value offset. During image display, the input data (i.e., the output data of the AI ​​model) includes at least one type of image data, and the output data is image data used for encoding and compression (such as an 8-bit image). The processing method is constrained by the configuration information; different types of image data can use different processing methods. The configuration information includes constraints on image display processing. For example, if the output data of a convolutional neural network model ranges from -0.5 to 0.5, the configuration information includes two parameters: Scale=255 and Offset=128.

[0155] In one possible implementation, for steps 203-205, the front-end device can read the stored image data from the specified storage medium, for example, see [link to relevant documentation]. Figure 6A As shown, first read the first row of data (A1A1A1) of the image block to be stored, then read the first row of data (B1B1B1) of the image block to be stored, ..., read the first row of data (N1N1N1) of the image block to be stored, then read the second row of data (A2A2A2) of the image block to be stored, then read the second row of data (B2B2B2) of the image block to be stored, ..., read the second row of data (N2N2N2) of the image block to be stored, and so on.

[0156] The front-end device can encode and compress the stored image data to obtain the target bitstream. The front-end device can then send this target bitstream to the back-end device. After receiving the target bitstream, the back-end device can decode it to obtain the decoded data, i.e., the decoded image data.

[0157] In one possible implementation, for step 206, the backend device can compress and restore the decoded data to obtain the target image. The compression and restoration process can be understood as the reverse process of pre-compression.

[0158] For example, when a front-end device processes the first image data using an AI model to obtain the second image data, if the AI ​​processing includes image transformation (corresponding to case 1), then the back-end device, after receiving the decoded data, can perform an inverse image transformation on the decoded data to obtain the target image. For instance, if the front-end device performs a Gaussian pyramid transformation on the first image data using an AI model, then the back-end device performs an inverse Gaussian pyramid transformation on the decoded data to obtain the target image. Alternatively, if the front-end device performs an interpolation transformation on the first image data using an AI model, then the back-end device performs an inverse interpolation transformation on the decoded data to obtain the target image.

[0159] For example, when the front-end device uses an AI model to process the first image data to obtain the second image data, if the AI ​​processing includes image augmentation (corresponding to case 2), then the back-end device can use the decoded data as the target image after receiving the decoded data, and no longer needs to restore the decoded data.

[0160] For example, when the front-end device uses an AI model to process the first image data to obtain the second image data, if the AI ​​processing includes semantic transformation (corresponding to case 2), then the back-end device can process the decoded data based on the AI ​​model after receiving the decoded data to obtain the target image.

[0161] The AI ​​model can include an image processing neural network and a semantic recovery neural network. The decoded data can include decoded image data and structured semantic information. Based on this, the decoded image data can be input into the image processing neural network within the AI ​​model. The image processing neural network then processes the decoded image data to obtain the image data to be enhanced. For example, the image processing neural network includes at least one convolutional layer and one activation layer, which can perform convolution and activation operations on the decoded image data.

[0162] Furthermore, structured semantic information can be input into a semantic recovery neural network within an AI model. This network can then manipulate the structured semantic information to obtain semantically enhanced data. For example, a semantic recovery neural network can include at least one convolutional layer and one activation layer; there are no restrictions on the network structure. This allows for convolutional and activation operations to be performed on the structured semantic information.

[0163] A semantic recovery neural network can be pre-trained. The input data to the semantic recovery neural network is structured semantic information, and the output data is semantically enhanced data. This enhanced data can be image data, and its dimension is the same as the dimension of the image data to be enhanced. For example, if the image data to be enhanced is an M*N image, then the semantically enhanced data can also be an M*N image. Based on this, after inputting the structured semantic information into the semantic recovery neural network, semantically enhanced data can be obtained.

[0164] After obtaining the image data to be enhanced and the semantic enhancement data, the two data can be fused to obtain the target image. This can be achieved using at least one fusion method, such as weighted fusion, stitching fusion, multiplicative fusion, attention fusion, additive fusion, or convolutional fusion. For example, by fusing the image data to be enhanced and the semantic enhancement data, structured semantic information can be incorporated into the image data to be enhanced. Since structured semantic information describes the semantics of the image data to be enhanced, it can recover the high-frequency information of the image data to be enhanced, thereby enhancing the high-frequency information of the target image.

[0165] For example, when a front-end device processes first image data using an AI model to obtain second image data, if the AI ​​processing includes image transformation, image augmentation, and semantic transformation (corresponding to case 4), then the back-end device, after receiving the decoded data, can perform an inverse image transformation on the decoded data to obtain the first image data to be enhanced. Alternatively, the decoded image data can be input into the image processing neural network in the AI ​​model, and the image processing neural network can operate on the decoded image data to obtain the second image data to be enhanced. Structured semantic information can be input into the semantic recovery neural network in the AI ​​model, and the semantic recovery neural network can operate on the structured semantic information to obtain semantically enhanced data. Then, the first and second image data to be enhanced are fused to obtain the target image data to be enhanced, and the target image data to be enhanced and the semantically enhanced data are fused to obtain the target image.

[0166] In one possible implementation, regarding step 207, after obtaining the target image, the backend device can directly store or display the target image. Alternatively, the backend device can perform post-processing on the target image (such as distortion removal, noise reduction, super-resolution, etc.) and store or display the post-processed target image.

[0167] As can be seen from the above technical solutions, in this embodiment, because the high-frequency information in the image data is enhanced by the AI ​​model, even if the encoding and compression process reduces the image quality of the image data (resulting in the loss of high-frequency information and thus image quality degradation), when the backend device decodes the target bitstream to obtain the target image, the image quality of the target image is equivalent to that of the original image, thereby improving the image quality of the stored or displayed target image. By processing the real-time data stream (i.e., video stream) using an AI model deployed on the front-end device, the image quality of the real-time data stream can be improved, helping the video image encoding and decoding system achieve higher video image quality.

[0168] Based on the same concept as the above method, this application proposes an image processing apparatus for use in a front-end device. See [link to relevant documentation]. Figure 7 The diagram shown is a structural schematic of the device, which includes:

[0169] The preprocessing module 71 is used to preprocess the acquired original image to obtain first image data, wherein the preprocessing is used to convert the original image into first image data that is compatible with the artificial intelligence AI model;

[0170] AI processing module 72 is used to perform AI processing on the first image data through the AI ​​model to obtain second image data, wherein the AI ​​processing is used to enhance the high-frequency information in the first image data;

[0171] Storage module 73 is used to store the second image data in a specified storage medium;

[0172] The sending module 74 is used to read stored image data from a specified storage medium, encode and compress the stored image data to obtain a target bitstream, send the target bitstream to a backend device, and have the backend device acquire, store or display the target image based on the target bitstream.

[0173] For example, the preprocessing includes at least one of color space transformation, pixel value mapping, image block partitioning, and image rearrangement; when the preprocessing module 71 preprocesses the acquired original image to obtain first image data, it is specifically used for: if the preprocessing includes color space transformation, and the first color space of the original image is a color space not supported by the AI ​​model, then the original image is subjected to color space transformation to obtain image data in a second color space, wherein the second color space is a color space supported by the AI ​​model; if the preprocessing includes pixel value mapping, and the first pixel value distribution of the original image is a pixel value distribution not supported by the AI ​​model, then the original image is subjected to pixel value mapping to obtain image data in a second pixel value distribution, wherein the second pixel value distribution is supported by the AI ​​model. Pixel value distribution; if the preprocessing includes image block division, the original image is divided into multiple initial image blocks, the multiple initial image blocks include edge blocks and non-edge blocks, the edge blocks include the boundary of the original image, and the non-edge blocks do not include the boundary of the original image; for each initial image block, pixel filling is performed on the initial image block to obtain target image blocks; wherein, the multiple target image blocks are used as input data of the AI ​​model, and the AI ​​model performs parallel processing on the multiple target image blocks; if the preprocessing includes image rearrangement, and the first arrangement of the original image is an arrangement not supported by the AI ​​model, the original image is rearranged to obtain image data with a second arrangement, the second arrangement being an arrangement supported by the AI ​​model.

[0174] For example, the AI ​​processing includes at least one of image transformation, image augmentation, and semantic transformation. When the AI ​​processing module 72 performs AI processing on the first image data using the AI ​​model to obtain the second image data, it specifically performs the following: if the AI ​​processing includes image transformation, it performs image transformation on the first image data using the AI ​​model to obtain transformed image data. The resolution of the transformed image data is equal to or less than the resolution of the first image data, and the number of channels in the transformed image data is equal to or greater than the number of channels in the first image data. The image transformation is used to separate high-frequency and low-frequency information in the first image data. The image transformation includes at least one of interpolation transformation, wavelet transformation, Gaussian pyramid transformation, Laplacian pyramid transformation, and Sobel filtering transformation. The transformed image data is input to the image processing neural network in the AI ​​model, and the image processing neural network operates on the transformed image data to obtain the second image data. The resolution of the second image data is equal to or less than the resolution of the first image data, and the image processing neural network is used to enhance the high-frequency information in the first image data.

[0175] For example, when the AI ​​processing module 72 performs AI processing on the first image data through the AI ​​model to obtain the second image data, it specifically performs the following steps: if the AI ​​processing includes image augmentation, the first image data is input to the image processing neural network in the AI ​​model, and the first image data is processed by the image processing neural network to obtain image data to be augmented, wherein the resolution of the image data to be augmented is equal to or less than the resolution of the first image data, and the number of channels of the image data to be augmented is equal to or less than the number of channels of the first image data; the first image data is augmented by the AI ​​model to obtain augmented image data corresponding to the image data to be augmented, wherein the resolution of the augmented image data is equal to or less than the resolution of the first image data, and the number of channels of the augmented image data is equal to or less than the number of channels of the first image data; wherein the augmented image data is used to enhance the high-frequency information in the image data to be augmented; wherein the image augmentation includes at least one of guided filtering, image edge enhancement, image contrast enhancement, and histogram equalization; the image data to be augmented and the augmented image data are fused to obtain the second image data, wherein the resolution of the second image data is equal to or less than the resolution of the first image data.

[0176] For example, when the AI ​​processing module 72 performs AI processing on the first image data through the AI ​​model to obtain the second image data, it specifically performs the following steps: if the AI ​​processing includes semantic transformation, the first image data is input to the image processing neural network in the AI ​​model, and the first image data is processed by the image processing neural network to obtain image data to be enhanced, wherein the resolution of the image data to be enhanced is equal to or less than the resolution of the first image data, and the number of channels of the image data to be enhanced is equal to or less than the number of channels of the first image data; the first image data is input to the semantic processing neural network in the AI ​​model, and the first image data is processed by the semantic processing neural network to obtain structured semantic information, wherein the structured semantic information is used to describe the semantics of the image data to be enhanced; and the second image data is generated based on the image data to be enhanced and the structured semantic information.

[0177] For example, when the AI ​​processing module 72 performs AI processing on the first image data using the AI ​​model to obtain the second image data, it specifically performs the following: if the AI ​​processing includes image transformation, image expansion, and semantic transformation, the first image data is transformed using the AI ​​model to obtain transformed image data, wherein the image transformation is used to separate high-frequency and low-frequency information in the first image data; the transformed image data is input into the image processing neural network in the AI ​​model, and the transformed image data is processed by the image processing neural network to obtain first image data to be enhanced, wherein the image processing neural network is used to enhance the high-frequency information in the first image data; the first image data is expanded using the AI ​​model to obtain expanded image data, wherein the expanded image data is used to enhance the high-frequency information in the first image data to be enhanced; the first image data to be enhanced and the expanded image data are fused to obtain second image data to be enhanced; the first image data is input into the semantic processing neural network in the AI ​​model, and the first image data is processed by the semantic processing neural network to obtain structured semantic information, wherein the structured semantic information is used to describe the semantics of the second image data to be enhanced; and the second image data is generated based on the second image data to be enhanced and the structured semantic information.

[0178] For example, if the first image data includes K target image blocks, the second image data includes K image blocks to be stored corresponding to the K target image blocks, where K is a positive integer greater than 1; for each image block to be stored, the image block to be stored corresponds to a storage start address, and the interval between the storage start addresses of two adjacent image blocks to be stored is the width of the image block to be stored; the second image data corresponds to a storage span, and the storage span is greater than or equal to the sum of the widths of the K image blocks to be stored;

[0179] When the storage module 73 stores the second image data in the specified storage medium, it is specifically used to: for each image block to be stored, when storing the i-th row of data of the image block to be stored in the specified storage medium, store the i-th row of data sequentially starting from the target storage location; wherein, the target storage location is determined by the following formula: p=w+ stride*(i-1); p represents the target storage location, w represents the storage starting address corresponding to the image block to be stored, stride represents the storage span, i represents the number of rows of the image block to be stored, and i is a positive integer.

[0180] Based on the same concept as the above method, this application proposes an electronic device (such as a front-end device), see [link]. Figure 8As shown, it includes: a processor 81 and a machine-readable storage medium 82, the machine-readable storage medium 82 storing machine-executable instructions that can be executed by the processor 81; the processor 81 is used to execute the machine-executable instructions to implement the image processing method disclosed in the above example of this application.

[0181] Based on the same concept as the above method, this application also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the image processing method disclosed in the above examples of this application.

[0182] The aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0183] Based on the same application concept as the above method, this application embodiment also provides a computer program product, which may include a computer program. When the computer program is executed by a processor, it implements the image processing method disclosed in the above examples of this application.

[0184] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The above descriptions are merely embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An image processing method, characterized in that, The method is applied to a front-end device, wherein the front-end device is configured with an AI model, and the AI ​​model is a lightweight model. Lightweight models require less computation and memory during AI model inference. The method includes: The acquired original image is preprocessed to obtain first image data. The preprocessing is used to convert the original image into first image data that is compatible with an artificial intelligence (AI) model. The preprocessing includes at least one of color space transformation, pixel value mapping, image block division, and image rearrangement. The first image data is processed by the AI ​​model to obtain the second image data, and the AI ​​processing is used to enhance the high-frequency information in the first image data; Store the second image data in the specified storage medium; The system reads stored image data from the specified storage medium, encodes and compresses the stored image data to obtain a target bitstream, sends the target bitstream to a backend device, and the backend device acquires the target image based on the target bitstream, and stores or displays the target image.

2. The method according to claim 1, characterized in that, The step of preprocessing the acquired raw image to obtain the first image data includes: If the preprocessing includes color space transformation, and the first color space of the original image is a color space not supported by the AI ​​model, then the original image is subjected to color space transformation to obtain image data in a second color space, where the second color space is a color space supported by the AI ​​model. If the preprocessing includes pixel value mapping, and the first pixel value distribution of the original image is a pixel value distribution that the AI ​​model does not support, then the original image is pixel value mapped to obtain image data with a second pixel value distribution, where the second pixel value distribution is a pixel value distribution supported by the AI ​​model. If the preprocessing includes image block segmentation, the original image is segmented into multiple initial image blocks, which include edge blocks and non-edge blocks. Edge blocks include the boundaries of the original image, while non-edge blocks do not include the boundaries of the original image. For each initial image block, pixel filling is performed to obtain target image blocks. The multiple target image blocks serve as input data for the AI ​​model, and the AI ​​model processes the multiple target image blocks in parallel. If the preprocessing includes image rearrangement, and the first arrangement of the original image is an arrangement not supported by the AI ​​model, then the original image is rearranged to obtain image data with a second arrangement, which is an arrangement supported by the AI ​​model.

3. The method according to claim 1, characterized in that, The AI ​​processing includes at least one of image transformation, image augmentation, and semantic transformation. If the AI ​​processing includes image transformation, the step of obtaining the second image data by performing AI processing on the first image data using the AI ​​model includes: The AI ​​model performs image transformation on the first image data to obtain transformed image data. The resolution of the transformed image data is equal to or less than the resolution of the first image data, and the number of channels in the transformed image data is equal to or greater than the number of channels in the first image data. The image transformation is used to separate high-frequency and low-frequency information in the first image data. The image transformation includes at least one of interpolation transformation, wavelet transform, Gaussian pyramid transform, Laplace pyramid transform, and Sobel filter transform. The transformed image data is input into the image processing neural network in the AI ​​model, and the image processing neural network operates on the transformed image data to obtain second image data; wherein, the resolution of the second image data is equal to or less than the resolution of the first image data, and the image processing neural network is used to enhance the high-frequency information in the first image data.

4. The method according to claim 1, characterized in that, The AI ​​processing includes at least one of image transformation, image augmentation, and semantic transformation. If the AI ​​processing includes image augmentation, the step of obtaining the second image data by performing AI processing on the first image data using the AI ​​model includes: The first image data is input to the image processing neural network in the AI ​​model. The image processing neural network operates on the first image data to obtain image data to be enhanced. The resolution of the image data to be enhanced is equal to or less than the resolution of the first image data, and the number of channels of the image data to be enhanced is equal to or less than the number of channels of the first image data. The AI ​​model is used to perform image augmentation on the first image data to obtain augmented image data corresponding to the image data to be enhanced. The resolution of the augmented image data is equal to or less than the resolution of the first image data, and the number of channels of the augmented image data is equal to or less than the number of channels of the first image data. The augmented image data is used to enhance high-frequency information in the image data to be enhanced. The image augmentation includes at least one of guided filtering, image edge enhancement, image contrast enhancement, and histogram equalization. The image data to be enhanced and the expanded image data are fused to obtain a second image data, the resolution of the second image data being equal to or less than the resolution of the first image data.

5. The method according to claim 1, characterized in that, The AI ​​processing includes at least one of image transformation, image augmentation, and semantic transformation. If the AI ​​processing includes semantic transformation, the step of obtaining the second image data by performing AI processing on the first image data using the AI ​​model includes: The first image data is input to the image processing neural network in the AI ​​model. The image processing neural network operates on the first image data to obtain image data to be enhanced. The resolution of the image data to be enhanced is equal to or less than the resolution of the first image data, and the number of channels of the image data to be enhanced is equal to or less than the number of channels of the first image data. The first image data is input into the semantic processing neural network in the AI ​​model. The semantic processing neural network operates on the first image data to obtain structured semantic information, and the structured semantic information is used to describe the semantics of the image data to be enhanced. A second image data is generated based on the image data to be enhanced and the structured semantic information.

6. The method according to claim 1, characterized in that, If the AI ​​processing includes image transformation, image augmentation, and semantic transformation, the step of obtaining the second image data by performing AI processing on the first image data using the AI ​​model includes: The AI ​​model performs image transformation on the first image data to obtain transformed image data. The image transformation is used to separate high-frequency and low-frequency information in the first image data. The transformed image data is then input into the image processing neural network in the AI ​​model. The image processing neural network operates on the transformed image data to obtain first image data to be enhanced. The image processing neural network is used to enhance the high-frequency information in the first image data. The AI ​​model is used to perform image augmentation on the first image data to obtain augmented image data. The augmented image data is used to enhance the high-frequency information in the first image data to be enhanced. The first image data to be enhanced and the augmented image data are fused to obtain the second image data to be enhanced. The first image data is input into the semantic processing neural network in the AI ​​model. The semantic processing neural network operates on the first image data to obtain structured semantic information, and the structured semantic information is used to describe the semantics of the second image data to be enhanced. The second image data is generated based on the second image data to be enhanced and the structured semantic information.

7. The method according to claim 1, characterized in that, If the first image data includes K target image blocks, the second image data includes K image blocks to be stored corresponding to the K target image blocks, where K is a positive integer greater than 1; for each image block to be stored, the image block to be stored corresponds to a storage start address, and the interval between the storage start addresses of two adjacent image blocks to be stored is the width of the image block to be stored; the second image data corresponds to a storage span, and the storage span is greater than or equal to the sum of the widths of the K image blocks to be stored; Storing the second image data in the specified storage medium includes: For each image block to be stored, when storing the i-th row of data of the image block to be stored in the specified storage medium, the i-th row of data is stored sequentially starting from the target storage location; The target storage location is determined using the following formula: p = w + stride * (i - 1); Where p represents the target storage location, w represents the storage starting address corresponding to the image block to be stored, stride represents the storage span, and i represents the row number of the image block to be stored, where i is a positive integer.

8. An image processing apparatus, characterized in that, The device is applied to a front-end device, which is configured with an AI model. The AI ​​model is a lightweight model, requiring less computation and memory during inference. The device includes: A preprocessing module is used to preprocess the acquired raw image to obtain first image data. The preprocessing is used to convert the raw image into first image data that is compatible with an artificial intelligence (AI) model. The preprocessing includes at least one of color space transformation, pixel value mapping, image block division, and image rearrangement. The AI ​​processing module is used to perform AI processing on the first image data through the AI ​​model to obtain the second image data, and the AI ​​processing is used to enhance the high-frequency information in the first image data; A storage module for storing the second image data in a specified storage medium; The sending module is used to read stored image data from a specified storage medium, encode and compress the stored image data to obtain a target bitstream, send the target bitstream to a backend device, and have the backend device acquire, store, or display the target image based on the target bitstream.

9. The apparatus according to claim 8, characterized in that, The preprocessing module, when preprocessing the acquired original image to obtain first image data, specifically performs the following: if the preprocessing includes color space transformation, and the first color space of the original image is a color space not supported by the AI ​​model, then the original image undergoes color space transformation to obtain image data in a second color space, where the second color space is a color space supported by the AI ​​model; if the preprocessing includes pixel value mapping, and the first pixel value distribution of the original image is a pixel value distribution not supported by the AI ​​model, then the original image undergoes pixel value mapping to obtain image data with a second pixel value distribution, where the second pixel value distribution is a pixel value distribution supported by the AI ​​model; if the preprocessing includes image block division, then... The original image is divided into multiple initial image blocks, including edge blocks and non-edge blocks. The edge blocks include the boundary of the original image, and the non-edge blocks do not include the boundary of the original image. For each initial image block, pixel filling is performed to obtain target image blocks. The multiple target image blocks serve as input data for the AI ​​model, and the AI ​​model processes the multiple target image blocks in parallel. If the preprocessing includes image rearrangement, and the first arrangement of the original image is an arrangement not supported by the AI ​​model, then the original image is rearranged to obtain image data with a second arrangement, which is an arrangement supported by the AI ​​model.

10. The apparatus according to claim 8, characterized in that, The AI ​​processing includes at least one of image transformation, image augmentation, and semantic transformation. When the AI ​​processing module processes the first image data using the AI ​​model to obtain the second image data, it specifically performs the following: if the AI ​​processing includes image transformation, it performs image transformation on the first image data using the AI ​​model to obtain transformed image data. The resolution of the transformed image data is equal to or less than the resolution of the first image data, and the number of channels in the transformed image data is equal to or greater than the number of channels in the first image data. The image transformation is used to separate high-frequency and low-frequency information in the first image data. The image transformation includes at least one of interpolation transformation, wavelet transformation, Gaussian pyramid transformation, Laplacian pyramid transformation, and Sobel filtering transformation. The transformed image data is input to the image processing neural network in the AI ​​model, and the image processing neural network operates on the transformed image data to obtain the second image data. The resolution of the second image data is equal to or less than the resolution of the first image data, and the image processing neural network is used to enhance the high-frequency information in the first image data.

11. The apparatus according to claim 8, characterized in that, When the AI ​​processing module obtains the second image data by performing AI processing on the first image data through the AI ​​model, it is specifically used as follows: if the AI ​​processing includes image augmentation, the first image data is input to the image processing neural network in the AI ​​model, and the first image data is processed by the image processing neural network to obtain image data to be enhanced. The resolution of the image data to be enhanced is equal to or less than the resolution of the first image data, and the number of channels of the image data to be enhanced is equal to or less than the number of channels of the first image data. The AI ​​model is used to perform image augmentation on the first image data to obtain augmented image data corresponding to the image data to be enhanced. The resolution of the augmented image data is equal to or less than the resolution of the first image data, and the number of channels in the augmented image data is equal to or less than the number of channels in the first image data. The augmented image data is used to enhance high-frequency information in the image data to be enhanced. The image augmentation includes at least one of guided filtering, image edge enhancement, image contrast enhancement, and histogram equalization. The image data to be enhanced and the augmented image data are fused to obtain second image data, the resolution of which is equal to or less than the resolution of the first image data.

12. The apparatus according to claim 8, characterized in that, When the AI ​​processing module obtains the second image data by performing AI processing on the first image data through the AI ​​model, it is specifically used as follows: if the AI ​​processing includes semantic transformation, the first image data is input to the image processing neural network in the AI ​​model, and the first image data is processed by the image processing neural network to obtain the image data to be enhanced. The resolution of the image data to be enhanced is equal to or less than the resolution of the first image data, and the number of channels of the image data to be enhanced is equal to or less than the number of channels of the first image data. The first image data is input into the semantic processing neural network in the AI ​​model. The semantic processing neural network operates on the first image data to obtain structured semantic information, and the structured semantic information is used to describe the semantics of the image data to be enhanced. A second image data is generated based on the image data to be enhanced and the structured semantic information.

13. The apparatus according to claim 8, characterized in that, When the AI ​​processing module processes the first image data using the AI ​​model to obtain the second image data, it specifically performs the following steps: If the AI ​​processing includes image transformation, image augmentation, and semantic transformation, the first image data is transformed using the AI ​​model to obtain transformed image data, whereby the image transformation is used to separate high-frequency and low-frequency information in the first image data; the transformed image data is input into the image processing neural network in the AI ​​model, and the transformed image data is processed by the image processing neural network to obtain first image data to be enhanced, whereby the image processing neural network is used to enhance the high-frequency information in the first image data; the first image data is augmented using the AI ​​model to obtain augmented image data, whereby the augmented image data is used to enhance the high-frequency information in the first image data to be enhanced; the first image data to be enhanced and the augmented image data are fused to obtain second image data to be enhanced; the first image data is input into the semantic processing neural network in the AI ​​model, and the first image data is processed by the semantic processing neural network to obtain structured semantic information, whereby the structured semantic information is used to describe the semantics of the second image data to be enhanced; and the second image data is generated based on the second image data to be enhanced and the structured semantic information.

14. The apparatus according to claim 8, characterized in that, If the first image data includes K target image blocks, the second image data includes K image blocks to be stored corresponding to the K target image blocks, where K is a positive integer greater than 1; for each image block to be stored, the image block to be stored corresponds to a storage start address, and the interval between the storage start addresses of two adjacent image blocks to be stored is the width of the image block to be stored; the second image data corresponds to a storage span, and the storage span is greater than or equal to the sum of the widths of the K image blocks to be stored; When storing the second image data in the specified storage medium, the storage module is specifically used to: for each image block to be stored, when storing the i-th row of data of the image block to be stored in the specified storage medium, store the i-th row of data sequentially starting from the target storage location; wherein, the target storage location is determined by the following formula: p=w+ stride*(i-1); p represents the target storage location, w represents the storage starting address corresponding to the image block to be stored, stride represents the storage span, i represents the number of rows of the image block to be stored, and i is a positive integer.

15. An electronic device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method of any one of claims 1-7.

Images