Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method, and relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, computer vision and image decoding. The specific implementation scheme is as follows: in response to acquisition of partially decoded image data with a preset scale, to-be-processed image block data is determined according to the partially decoded image data, and the partially decoded image data is obtained by performing image decoding on a part of a to-be-processed image; and performing feature extraction on the to-be-processed image block data to obtain image block features. The invention further provides an image processing device, electronic equipment and a storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of deep learning, computer vision technology and image decoding technology, and can be applied to image processing scenarios. More specifically, the present disclosure provides an image processing method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, the application scenarios of deep learning technology are increasing. In the image processing scenario, the image to be processed can be encoded based on a variety of image encoding methods. The image to be processed can be decoded to obtain decoded image data. Next, the decoded image data can be processed based on deep learning technology. Summary of the invention

[0003] The present disclosure provides an image processing method, apparatus, device and storage medium.

[0004] According to one aspect of the present disclosure, there is provided an image processing method, the method comprising: in response to obtaining partially decoded image data having a preset size, determining image block data to be processed according to the partially decoded image data, wherein the partially decoded image data is obtained by performing image decoding on a portion of the image to be processed; and performing feature extraction on the image block data to be processed to obtain image block features.

[0005] According to another aspect of the present disclosure, an image processing device is provided, the device comprising: a storage unit; a plurality of processor cores, configured to: in response to determining that the storage unit has stored partially decoded image data having a preset size, determine image block data to be processed according to the partially decoded image data, wherein the partially decoded image data is obtained by performing image decoding on a portion of the image to be processed; and perform feature extraction on the image block data to be processed to obtain image block features.

[0006] According to another aspect of the present disclosure, an image processing device is provided, including the image processing apparatus provided by the present disclosure.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0009] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method provided according to the present disclosure is implemented.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 is a schematic diagram of an exemplary system architecture to which an image processing method and apparatus can be applied according to an embodiment of the present disclosure;

[0013] Figure 2 is a flowchart of an image processing method according to an embodiment of the present disclosure;

[0014] Figure 3 is a schematic diagram of an image to be processed according to an embodiment of the present disclosure;

[0015] Figure 4A is a schematic diagram of intermediate image block data according to an embodiment of the present disclosure;

[0016] Figure 4B is a schematic diagram of multiple initial address spaces according to an embodiment of the present disclosure;

[0017] Figure 4C is a schematic diagram of image block data to be processed according to an embodiment of the present disclosure;

[0018] Figure 4D is a schematic diagram of multiple target address spaces according to one embodiment of the present disclosure;

[0019] Figure 5 is a schematic diagram of a flattening process according to another embodiment of the present disclosure;

[0020] Figure 6 is a block diagram of an image processing apparatus according to an embodiment of the present disclosure;

[0021] Figure 7 is a block diagram of an image processing apparatus according to an embodiment of the present disclosure; and

[0022] Figure 8 is a block diagram of an electronic device to which an image processing method can be applied according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] The Transformer model can process input data based on a multi-head self-attention mechanism and is widely used in the field of natural language understanding. In the field of computer vision technology, the application of the Vision Transformer (ViT) model is also increasing. The Vision Transformer model can divide the input image into multiple image patches. The size of the image patch can be, for example, 16×16. Next, the Vision Transformer model can flatten the image patch to obtain low-dimensional image features. The Vision Transformer model can also perform linear projection on the low-dimensional image features to obtain high-dimensional image features. After position embedding is performed on the high-dimensional image features, at least one of the transformer encoder and the transformer decoder can be used to process them to obtain the processing results.

[0025] The format of the input image can be various image formats such as Joint Photographic Experts Group (JPEG) and Portable Network Graphics (PNG). As the resolution of the input image continues to increase, the computing power and memory access overhead required to determine the image blocks of the input image and flatten the image are increasing. In addition, taking the Joint Photographic Experts Group image as an example, before using the transformer model for processing, one or more image blocks can be determined after the image decoding of the entire input image is completed. As the resolution of the input image continues to increase, the time cost of image decoding continues to increase, making it difficult to effectively improve the efficiency of image processing.

[0026] After the image is decoded, multiple preset processing can be performed on the decoded image data, and then the transformer model can be used for processing. When the artificial intelligence computing unit is used for image decoding and preset processing, different processor cores of the artificial intelligence computing unit can be used to implement image decoding and preset processing respectively. One processor core can decode the image and store the decoded image data in a global memory. Another processor core can read the decoded image data from the global memory, and then perform multiple preset processing on the decoded image data to obtain intermediate image data. Multiple preset processing can include image format conversion processing, image size conversion processing, etc. The intermediate image data can be written to the global memory unit. Different processor cores perform image decoding and preset processing respectively, and multiple memory accesses are required, resulting in high bandwidth resource overhead and storage resource overhead.

[0027] In the above flattening process, the image data in the storage unit needs to be adjusted, which is time-consuming and has a high time cost.

[0028] Therefore, in order to improve the efficiency of image processing, the present disclosure provides an image processing method, which will be described below.

[0029] Figure 1 is a schematic diagram of an exemplary system architecture to which an image processing method and apparatus can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0030] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0031] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptops, desktop computers, etc.

[0032] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0033] It should be noted that the image processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the image processing apparatus provided in the embodiment of the present disclosure can generally be set in the server 105. The image processing method provided in the embodiment of the present disclosure can also generally be executed by the terminal devices 101, 102, 103. Accordingly, the image processing apparatus provided in the embodiment of the present disclosure can generally be set in the terminal devices 101, 102, 103. The image processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the image processing apparatus provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0034] It can be understood that the above describes the system architecture of the present disclosure, and the following will describe the method of the present disclosure.

[0035] Figure 2 is a flowchart of an image processing method according to an embodiment of the present disclosure.

[0036] like Figure 2 As shown, the method 200 may include operations S210 to S220.

[0037] In operation S210, in response to obtaining partially decoded image data having a preset size, image block data to be processed is determined according to the partially decoded image data.

[0038] In the disclosed embodiment, the partially decoded image data is obtained by decoding a portion of the image to be processed. For example, the scale of the image to be processed may be 224×224. A portion of the original image data of 16×16 of the image to be processed may be image decoded to obtain partially decoded data. The preset scale may be 16×16. For another example, the image to be processed may be row-by-row image decoding to obtain 16×16 partially decoded image data. It can be understood that the scale of the image to be processed may be any scale, and the preset scale may also be any scale. The preset scale may be less than or equal to the scale of the image to be processed.

[0039] In the disclosed embodiment, part of the decoded image data may be used as the image block data to be processed. For example, the image block data to be processed may correspond to the data of one of the above-mentioned multiple image patches. It is understood that in the storage unit, the storage method of the image block data to be processed may be linear, which is consistent with the storage method of the data of the flattened image block.

[0040] In operation S220, feature extraction is performed on the image block data to be processed to obtain image block features.

[0041] In the disclosed embodiment, the visual transformer model can be used to perform linear projection on the image block data to be processed to extract features and obtain image block features.

[0042] Through the disclosed embodiment, after obtaining the partially decoded image data of the preset model, the image block data is determined. The computing resources of the artificial intelligence computing unit can be fully utilized, the idle time of the processor core for reasoning or training based on the visual transformer model can be reduced, and it is helpful to reduce pipeline bubbles. The image block data is determined after obtaining the partially decoded image data, which helps to make full use of the partially decoded image data in the storage unit, save the time required for subsequent flattening processing, and improve image processing efficiency.

[0043] It can be understood that the above describes the method of the present disclosure, and some ways of image decoding will be described below.

[0044] In some embodiments, the image to be processed may include multiple image block data to be decoded. Partially decoded image data may include one or more decoded image block data. The number of image block data to be decoded included in the image to be processed is greater than or equal to the number of decoded image block data included in the partially decoded data. For example, the partially decoded image data may include K decoded image block data. The decoded image block data is obtained by image decoding of the image block data to be decoded. K may be an integer greater than 1. The number of image block data to be decoded in the image to be processed is greater than or equal to K. For another example, the K decoded image block data may be 2. The 2 decoded image block data may be two adjacent decoded image block data. Through the embodiments of the present disclosure, decoding is performed with image block data as the granularity, so that the decoding of the Joint Photographic Experts Group image and the processing of the visual transformer model can be effectively integrated, and the image efficiency can be effectively improved.

[0045] In some embodiments, in some implementations of the above operation S210, in response to obtaining partially decoded image data having a preset size, determining the image block data to be processed according to the partially decoded image data includes: in response to obtaining K decoded image block data, determining the image block data to be processed according to the K decoded image block data. Figure 3 Provide explanation.

[0046] Figure 3 is a schematic diagram of an image to be processed according to an embodiment of the present disclosure.

[0047] like Figure 3 As shown, the size of the image to be processed image30 may be, for example, 224×224. The image to be processed image30 may be, for example, a Joint Photographic Experts Group image.

[0048] In the embodiment of the present disclosure, the image to be processed may include N×M image blocks to be decoded. Figure 3 As shown, N may be 14, and M may be 14. The scale of the image block data to be decoded may be 8×8. The N×M image block data to be decoded may include image block data to be decoded B311, image block data to be decoded B312, ... image block data to be decoded B3114, image block data to be decoded B321, image block data to be decoded B322, ... image block data to be decoded B3214.

[0049] In the disclosed embodiment, the decoded image block data may be obtained by performing the following operations on the image block data to be decoded of the image to be processed: performing image decoding on the plurality of image block data to be decoded in sequence according to a preset direction to obtain a plurality of decoded image block data. The preset direction may be a row direction or a column direction. The plurality of image block data to be decoded may correspond one to one to the plurality of decoded image block data. Figure 3 As shown, taking the preset direction as the row direction as an example, the image block data to be decoded B311, the image block data to be decoded B312, ... the image block data to be decoded B3114 can be sequentially decoded to obtain 14 decoded image block data. Next, the image block data to be decoded B321, the image block data to be decoded B322, ... the image block data to be decoded B3214 are decoded to obtain 14 decoded image block data.

[0050] For example, a counter may be set, and after the image decoding of a to-be-decoded image block data is completed, the counter may be accumulated to determine the number of decoded image block data. Figure 3As shown, after the image decoding of the image block data B311 to be decoded is completed, the number of decoded image block data determined by the counter may be 1. After the image decoding of the image block data B3114 to be decoded is completed, the number of decoded image block data determined by the counter may be 14.

[0051] In an embodiment of the present disclosure, the K decoded image block data are I×J decoded image block data. In response to obtaining the K decoded image block data, determining the image block data to be processed based on the K decoded image block data includes: in response to determining that the number of decoded image block data is greater than or equal to a preset number threshold, determining the image block data to be processed based on the I×J decoded image block data. I can be an integer greater than or equal to 1 and less than or equal to N, and J can be an integer greater than or equal to 1 and less than or equal to M. In the case where J is an integer greater than 1 and the preset direction is the row direction of the image to be processed, the preset number threshold may be M+I. Taking I=2 and J=2 as an example, the preset number threshold may be 16. If Figure 3 As shown, after obtaining 16 decoded image block data corresponding to the image block data to be decoded B311, the image block data to be decoded B312, ... the image block data to be decoded B3114, the image block data to be decoded B321, and the image block data to be decoded B322, I×J decoded image block data can be obtained. The I×J decoded image block data may include 4 decoded image block data corresponding to the image block data to be decoded B311, the image block data to be decoded B312, the image block data to be decoded B321, and the image block data to be decoded B322, that is, K=4. Based on these 4 decoded image block data, one image block data to be processed can be determined. Through the embodiment of the present disclosure, the image to be processed can be decoded according to the decoding method of the Joint Photographic Experts Group image, and the image block to be processed can be determined during the decoding process, which can improve the parallelism of image processing and improve processing efficiency.

[0052] It can be understood that the above description uses row-by-row decoding as an example to explain the preset number threshold. However, the present disclosure is not limited thereto. When I is an integer greater than 1 and the preset direction is the column direction of the image to be processed, the preset number threshold may be N+J. Taking I=2 and J=2 as an example, the preset number threshold may be 16.

[0053] It can be understood that the above describes the method for determining the partial decoded image data, and the following will further describe the method for determining the image block data to be processed.

[0054] In some embodiments, in some implementations of the above operation S210, determining the image block data to be processed according to the partially decoded image data includes: performing at least one of a plurality of preset processes on the partially decoded image data to obtain intermediate image data. The plurality of preset processes include image format conversion processing and image block size conversion (resize) processing. As described above, the partially decoded image data may include K decoded image block data. Intermediate image data may be obtained by performing at least one of a plurality of preset processes on the K decoded image block data. The intermediate image data may include K intermediate image block data. For example, the image format of the K decoded image block data may be a luminance-chrominance (YUV) image format. Y represents the brightness of the image, and U and V represent the chrominance of the image. If a visual transformer model is subsequently used for image processing, the luminance-chrominance image format may be converted into a red, green, and blue (RGB) image format. Thus, the image format conversion process may be performed on the K decoded image block data to obtain K intermediate image block data.

[0055] In some embodiments, in some implementations of the above operation S210, determining the image block data to be processed according to the partially decoded image data includes: converting the intermediate image data into the image block data to be processed. As described above, the intermediate image data may include K intermediate image block data. The K intermediate image block data may be converted into the image block data to be processed. For example, the size of the intermediate image block data may be 8×8. The 4 intermediate image block data may be converted into 16×16 image block data to be processed. FIG. 4A to FIG. 4C Provide explanation.

[0056] Figure 4A is a schematic diagram of intermediate image block data according to an embodiment of the present disclosure.

[0057] like Figure 4AAs shown, the intermediate image block data m411 may correspond to the above-mentioned image block data to be decoded B311. After the image block data to be decoded B311 is subjected to image decoding and image format conversion processing, the intermediate image block data m411 may be obtained. The scale of the intermediate image block data m411 is 8×8. The first row of the intermediate image block data m411 includes intermediate image sub-data md41111, ... intermediate image sub-data md41118. The second row of the intermediate image block data m411 includes intermediate image sub-data md41121, ... intermediate image sub-data md41128. The third row of the intermediate image block data m411 includes intermediate image sub-data md41131, ... intermediate image sub-data md41138. The fourth row of the intermediate image block data m411 includes intermediate image sub-data md41141, ... intermediate image sub-data md41148. The fifth row of intermediate image block data m411 includes intermediate image sub-data md41151, ... intermediate image sub-data md41158. The sixth row of intermediate image block data m411 includes intermediate image sub-data md41161, ... intermediate image sub-data md41168. The seventh row of intermediate image block data m411 includes intermediate image sub-data md41171, ... intermediate image sub-data md41178. The eighth row of intermediate image block data m411 includes intermediate image sub-data md41181, ... intermediate image sub-data md41188.

[0058] like Figure 4A As shown in FIG. 1 , if the intermediate image block data m411 is input and stored in the global storage unit, the continuous multiple rows of the intermediate image block data can be stored in multiple continuous address spaces of the global storage unit in sequence. The 8 rows of the intermediate image block data can be stored in 8 continuous address spaces in sequence. Figure 4B Provide explanation.

[0059] Figure 4B is a schematic diagram of multiple initial address spaces according to an embodiment of the present disclosure.

[0060] like Figure 4B As shown, the multiple address spaces may include consecutive initial address space addr4111, initial address space addr4112, initial address space addr4113, initial address space addr4114, initial address space addr4115, initial address space addr4116, initial address space addr4117 and initial address space addr4118.

[0061] like Figure 4BAs shown, if the intermediate image block data m411 is stored in the global storage unit, multiple consecutive rows of the intermediate image block data m411 can be stored in multiple consecutive address spaces of the global storage unit in sequence. For example, the intermediate image sub-data md41111, ... intermediate image sub-data md41118 of the first row can be stored in the initial address space addr4111. The intermediate image sub-data md41121, ... intermediate image sub-data md41128 of the second row can be stored in the initial address space addr4112. The intermediate image sub-data md41131, ... intermediate image sub-data md41138 of the third row can be stored in the initial address space addr4113. The intermediate image sub-data md41141, ... intermediate image sub-data md41148 of the fourth row can be stored in the initial address space addr4114. The intermediate image sub-data md41151, ..., intermediate image sub-data md41158 of the 5th row can be stored in the initial address space addr4115. The intermediate image sub-data md41161, ..., intermediate image sub-data md41168 of the 6th row can be stored in the initial address space addr4116. The intermediate image sub-data md41171, ..., intermediate image sub-data md41178 of the 7th row can be stored in the initial address space addr4117. The intermediate image sub-data md41181, ..., intermediate image sub-data md41188 of the 8th row can be stored in the initial address space addr4118. It can be understood that the intermediate image block data m411 is stored in the global storage unit in a tile format. The image block data to be processed converted according to the K intermediate image block data can also be stored in the global storage unit in a tile format, which will be combined below. Figure 4C and Figure 4D Provide explanation.

[0062] Figure 4C is a schematic diagram of image block data to be processed according to an embodiment of the present disclosure.

[0063] After obtaining the four decoded image block data corresponding to the above-mentioned image block data to be decoded B311, image block data to be decoded B312, image block data to be decoded B321 and image block data to be decoded B322, image format conversion processing can be performed to obtain the following: Figure 4C The image block data to be processed is shown as patch41.

[0064] like Figure 4CAs shown, the size of the image block data patch41 to be processed may be 16×16. The first row of the image block data patch41 to be processed may include the intermediate image sub-data md41111, ... the intermediate image sub-data md41118, intermediate image sub-data md41211, ..., intermediate image sub-data md41218. The second row of the image block data patch41 to be processed may include the intermediate image sub-data md41121, ..., intermediate image sub-data md41228. The 16th row of the image block data patch41 to be processed may include the intermediate image sub-data md42181, ..., intermediate image sub-data md42188, intermediate image sub-data md42281, ..., intermediate image sub-data md42288.

[0065] In the embodiment of the present disclosure, multiple rows of image block data to be processed are sequentially stored in multiple continuous address spaces. Figure 4D Provide explanation.

[0066] Figure 4D is a schematic diagram of multiple target address spaces according to an embodiment of the present disclosure.

[0067] like Figure 4D As shown, the multiple address spaces may include a continuous target address space addr4011, a target address space addr4012, ..., a target address space addr40116. The intermediate image sub-data md41111, ..., intermediate image sub-data md41118, intermediate image sub-data md41211, ..., intermediate image sub-data md41218 of the first row of the image block data patch41 to be processed may be stored in the target address space addr4011. The intermediate image sub-data md41121, ..., intermediate image sub-data md412288 of the second row of the image block data patch41 to be processed may be stored in the target address space addr4012. The intermediate image sub-data md42181, ..., intermediate image sub-data md42188, intermediate image sub-data md42281, ..., intermediate image sub-data md42288 of the 16th row of the image block data patch41 to be processed can be stored in the target address space addr40116. It can be understood that for simplicity, Figure 4D Only the storage method of the first row data and the sixteenth row data of the image block data to be processed is described. The storage method of the second row data to the fifteenth row data of the image block data to be processed is the same or similar to the storage method of the first row data and the sixteenth row data, and will not be described in detail in this disclosure.

[0068] It can be understood that the above describes the storage method of the image block data to be processed in the storage unit, and the flattening process of the visual transformer model will be described below.

[0069] Figure 5 is a schematic diagram of a flattening process according to another embodiment of the present disclosure.

[0070] like Figure 5 As shown, the size of the image block data patch51' may be 4 x 4. It can be understood that the size of the image block data is only an example, and the present disclosure does not limit this.

[0071] like Figure 5 As shown, the first row data of the image block data patch51' may include image sub-data 511, image sub-data 512, image sub-data 513, and image sub-data 514. The second row data of the image block data patch51' may include image sub-data 521, image sub-data 522, image sub-data 523, and image sub-data 524. The third row data of the image block data patch51' may include image sub-data 531, image sub-data 532, image sub-data 533, and image sub-data 534. The fourth row data of the image block data patch51' may include image sub-data 541, image sub-data 542, image sub-data 543, and image sub-data 544.

[0072] like Figure 5 As shown, by flattening the image block data patch51', a low-dimensional image feature f51' can be obtained. The low-dimensional image feature f51' may include image sub-data 511, image sub-data 512, image sub-data 513, image sub-data 514, image sub-data 521, image sub-data 522, image sub-data 523, image sub-data 524, image sub-data 531, image sub-data 532, image sub-data 533, image sub-data 534, image sub-data 541, image sub-data 542, image sub-data 543 and image sub-data 544 in sequence.

[0073] like Figure 5 and Figure 4D As shown, the low-dimensional image feature f51' is linear, and is also linear with the image block data to be processed patch41 stored in multiple continuous address spaces. Therefore, when multiple rows of image block data to be processed are sequentially stored in multiple continuous address spaces, after the image block data to be processed is provided to the visual Transformer model, the flattening process can be skipped and feature extraction (i.e., the above-mentioned operation S220) can be performed directly, which can effectively improve the image processing efficiency. Some methods of extracting features from the image block to be processed are described below.

[0074] In some embodiments, in some implementations of the above operation S220, extracting features from the image block data to be processed to obtain image block features includes: performing linear projection on the image block data to be processed to obtain image block features. For example, linear projection can be performed directly on the image block data patch41 to obtain image block features.

[0075] It can be understood that the present disclosure is described above in conjunction with the to-be-processed image block data patch 41. Next, the above operations S210 to S220 may be repeatedly performed to obtain 14×14 image block features.

[0076] In some embodiments, the method may further include: processing at least one image block feature based on an attention mechanism to obtain an image processing result. For example, 14×14 image block features may be input into at least one of an encoder and a decoder of a visual transformer to obtain an image processing result.

[0077] It can be understood that operation S210, preset processing and operation S220 can be performed by multiple processor cores of the artificial intelligence computing unit, which will be further explained below.

[0078] In some embodiments, performing at least one of a plurality of preset processes on the partially decoded image data includes: performing at least one of a plurality of preset processes on the partially decoded image data using a first processor core. The partially decoded image data is obtained by performing image decoding on a portion of the image to be decoded using the first processor core. For example, the first processor core may be used to perform image decoding on a portion of the image to be decoded, or the first processor core among a plurality of processor cores may be used to perform one or more preset processes on the partially decoded image data. Through the embodiments of the present disclosure, using the same processor core to perform operations such as image decoding and image format conversion processing can reduce memory access overhead, improve memory access efficiency, and thereby improve image processing efficiency.

[0079] In some embodiments, extracting features from the image block data to be processed to obtain image block features includes: extracting features from the image block data to be processed using a second processor core to obtain image block features. For example, feature extraction can be performed based on a visual transformer model using one or more second processor cores different from the first processor core, encoding can be performed based on an encoder of the visual transformer model using one or more second processor cores, and decoding can be performed based on a decoder of the visual transformer model using one or more second processor cores.

[0080] It can be understood that the above description is made in conjunction with the Joint Photographic Experts Group images to illustrate the method of the present disclosure, but the present disclosure is not limited thereto, and the images to be processed may be images of various formats.

[0081] It can be understood that the method of the present disclosure is described above, and the device of the present disclosure will be described below.

[0082] Figure 6 is a block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0083] like Figure 6 As shown, the apparatus 600 may include a storage unit 610 and a plurality of processor cores 620 .

[0084] The storage unit 610 may be the above-mentioned global storage unit.

[0085] The plurality of processor cores 620 may be configured to: in response to determining that the storage unit has stored partially decoded image data having a preset size, determine the image block data to be processed according to the partially decoded image data. The partially decoded image data is obtained by image decoding a portion of the image to be processed. Feature extraction is performed on the image block data to be processed to obtain image block features. For example, the plurality of processor cores may execute the above method 200.

[0086] In some embodiments, the partially decoded image data includes K decoded image block data. The plurality of processor cores are further configured to perform the following operations in response to determining that the storage unit has stored partially decoded image data of a preset size, and determining the image block data to be processed according to the partially decoded image data: in response to determining that the storage unit has stored K decoded image block data, determining the image block data to be processed according to the K decoded image block data. The decoded image block data is obtained by image decoding the image block data to be decoded of the image to be processed, and the number of the image block data to be decoded in the image to be processed is greater than or equal to K, and K is an integer greater than or equal to 1.

[0087] In some embodiments, the image to be processed includes N×M image block data to be decoded, and the multiple image block data to be decoded correspond to the multiple decoded image block data one by one. The K decoded image block data are I×J decoded image block data, where I is an integer greater than or equal to 1 and less than or equal to N, and J is an integer greater than or equal to 1 and less than or equal to M.

[0088] In some embodiments, the decoded image block data is obtained by performing image decoding on the image block data to be decoded of the image to be processed by the following operation: image decoding is performed on a plurality of image block data to be decoded in sequence according to a preset direction to obtain a plurality of decoded image block data. The preset direction includes: one of a row direction and a column direction of the image to be processed.

[0089] In some embodiments, the image to be processed includes N×M image block data to be decoded, and the K decoded image block data are I×J decoded image block data. The multiple processor cores are further configured to perform the following operations in response to determining that the storage unit has stored K decoded image block data, and determine the image block data to be processed according to the K decoded image block data: in response to determining that the number of decoded image block data in the storage unit is greater than or equal to a preset number threshold, determine the image block data to be processed according to the I×J decoded image block data. I is an integer greater than or equal to 1 and less than or equal to N, and J is an integer greater than or equal to 1 and less than or equal to M.

[0090] In some embodiments, when J is an integer greater than 1 and the preset direction is the row direction of the image to be processed, the preset number threshold is M + I. When I is an integer greater than 1 and the preset direction is the column direction of the image to be processed, the preset number threshold is N + J.

[0091] In some embodiments, the plurality of processor cores are further configured to perform the following operations to determine the image block data to be processed based on the partially decoded image data: performing at least one of a plurality of preset processes on the partially decoded image data to obtain intermediate image data. The plurality of preset processes include image format conversion processing and image block size conversion processing. The intermediate image data is converted into the image block data to be processed.

[0092] In some embodiments, a plurality of consecutive rows of image block data to be processed are sequentially stored in a plurality of consecutive address spaces.

[0093] In some embodiments, the plurality of processor cores are further configured to perform the following operations to extract features from the image block data to be processed to obtain image block features: perform linear projection on the image block data to be processed to obtain image block features.

[0094] In some embodiments, the multiple processor cores are further configured to: process at least one image block feature corresponding to the image to be processed based on the attention mechanism to obtain an image processing result.

[0095] In some embodiments, the plurality of processor cores include a first processor core and a second processor core, and the first processor core is configured to: perform at least one of a plurality of preset processes on the partially decoded image data. The partially decoded image data is obtained by using the first processor core to perform image decoding on a portion of the image to be decoded. The second processor core is configured to: perform feature extraction on the image block data to be processed to obtain image block features.

[0096] It can be understood that the apparatus of the present disclosure is described above, and the equipment including the apparatus will be described below.

[0097] Figure 7is a block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0098] like Figure 7 As shown, the device 70 may include an image processing apparatus 700. For example, the image processing apparatus 700 may be the apparatus 600 described above.

[0099] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0100] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0101] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0102] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0103] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0104] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the image processing method in any other appropriate manner (eg, by means of firmware).

[0105] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0106] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0107] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories, read-only memories, erasable programmable read-only memories (EPROM) or flash memories, optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0108] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) display or a liquid crystal display (LCD)) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0109] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a Local Area Network (LAN), a Wide Area Network (WAN), and the Internet.

[0110] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0111] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0112] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image processing method, comprising: In response to acquiring partially decoded image data having a preset size, determining image block data to be processed according to the partially decoded image data, wherein the partially decoded image data is obtained by image decoding a portion of the image to be processed; and Feature extraction is performed on the image block data to be processed to obtain image block features.

2. The method according to claim 1, wherein: The partially decoded image data includes K decoded image block data, In response to acquiring partially decoded image data having a preset size, determining the image block data to be processed according to the partially decoded image data comprises: In response to obtaining K decoded image block data, the image block data to be processed is determined according to the K decoded image block data, wherein the decoded image block data is obtained by performing image decoding on the image block data to be decoded of the image to be processed, and the number of the image block data to be decoded in the image to be processed is greater than or equal to K, and K is an integer greater than or equal to 1.

3. The method according to claim 2, wherein: The image to be processed includes N×M image block data to be decoded, and the plurality of image block data to be decoded corresponds one to one with the plurality of decoded image block data. The K decoded image block data are I×J decoded image block data, I is an integer greater than or equal to 1 and less than or equal to N, and J is an integer greater than or equal to 1 and less than or equal to M.

4. The method according to claim 2, wherein: The decoded image block data is obtained by performing image decoding on the image block data to be decoded of the image to be processed by the following operations: The plurality of image block data to be decoded are sequentially decoded according to a preset direction to obtain a plurality of decoded image block data, wherein the preset direction includes: one of a row direction and a column direction of the image to be processed.

5. The method according to claim 4, wherein: The image to be processed includes N×M image block data to be decoded, and the K decoded image block data are I×J decoded image block data. In response to acquiring K decoded image block data, determining the image block data to be processed according to the K decoded image block data comprises: In response to determining that the number of the decoded image block data is greater than or equal to a preset number threshold, the image block data to be processed is determined based on I×J decoded image block data, where I is an integer greater than or equal to 1 and less than or equal to N, and J is an integer greater than or equal to 1 and less than or equal to M.

6. The method according to claim 5, wherein: When J is an integer greater than 1 and the preset direction is the row direction of the image to be processed, the preset number threshold is M+I; When I is an integer greater than 1 and the preset direction is the column direction of the image to be processed, the preset number threshold is N+J.

7. The method according to claim 1, wherein: The determining, according to the partially decoded image data, image block data to be processed comprises: Perform at least one of a plurality of preset processes on the partially decoded image data to obtain intermediate image data, wherein the plurality of preset processes include image format conversion processing and image block size conversion processing; The intermediate image data is converted into the image block data to be processed.

8. The method according to claim 1, wherein: The plurality of continuous rows of the image block data to be processed are sequentially stored in a plurality of continuous address spaces.

9. The method according to claim 1, wherein: The extracting features of the image block data to be processed to obtain image block features comprises: Linear projection is performed on the image block data to be processed to obtain the image block features.

10. The method according to claim 1, further comprising: At least one image block feature corresponding to the image to be processed is processed based on the attention mechanism to obtain an image processing result.

11. The method according to claim 7, wherein: The performing at least one of a plurality of preset processes on the partially decoded image data comprises: Using the first processor core to perform at least one of a plurality of preset processes on the portion of the decoded image data; The partially decoded image data is obtained by using the first processor core to perform image decoding on a part of the image to be decoded; The extracting features of the image block data to be processed to obtain image block features comprises: The second processor core is used to extract features from the image block data to be processed to obtain image block features.

12. An image processing device, comprising: Storage unit; Multiple processor cores, configured as: In response to determining that the storage unit has stored partially decoded image data having a preset size, determining image block data to be processed according to the partially decoded image data, wherein the partially decoded image data is obtained by image decoding a portion of the image to be processed; and Feature extraction is performed on the image block data to be processed to obtain image block features.

13. The device according to claim 12, wherein: The partially decoded image data includes K decoded image block data, The plurality of processor cores are further configured to perform the following operations in response to determining that the storage unit has stored partially decoded image data having a preset size, and determining image block data to be processed according to the partially decoded image data: In response to determining that the storage unit has stored K decoded image block data, the image block data to be processed is determined according to the K decoded image block data, wherein the decoded image block data is obtained by performing image decoding on the image block data to be decoded of the image to be processed, and the number of the image block data to be decoded in the image to be processed is greater than or equal to K, and K is an integer greater than or equal to 1.

14. The device according to claim 13, wherein: The image to be processed includes N×M image block data to be decoded, and the plurality of image block data to be decoded corresponds one to one with the plurality of decoded image block data. The K decoded image block data are I×J decoded image block data, I is an integer greater than or equal to 1 and less than or equal to N, and J is an integer greater than or equal to 1 and less than or equal to M.

15. The device according to claim 13, wherein: The decoded image block data is obtained by performing image decoding on the image block data to be decoded of the image to be processed by the following operations: The plurality of image block data to be decoded are sequentially decoded according to a preset direction to obtain a plurality of decoded image block data, wherein the preset direction includes: one of a row direction and a column direction of the image to be processed.

16. The device according to claim 15, wherein: The image to be processed includes N×M image block data to be decoded, and the K decoded image block data are I×J decoded image block data. The plurality of processor cores are further configured to perform the following operations in response to determining that the storage unit has stored K decoded image block data, and determining the image block data to be processed according to the K decoded image block data: In response to determining that the number of the decoded image block data in the storage unit is greater than or equal to a preset number threshold, the image block data to be processed is determined according to I×J decoded image block data, wherein I is an integer greater than or equal to 1 and less than or equal to N, and J is an integer greater than or equal to 1 and less than or equal to M.

17. The device according to claim 16, wherein: When J is an integer greater than 1 and the preset direction is the row direction of the image to be processed, the preset number threshold is M+I; When I is an integer greater than 1 and the preset direction is the column direction of the image to be processed, the preset number threshold is N+J.

18. The device according to claim 12, wherein: The multiple processor cores are further configured to perform the following operations to determine the image block data to be processed according to the partially decoded image data: Perform at least one of a plurality of preset processes on the partially decoded image data to obtain intermediate image data, wherein the plurality of preset processes include image format conversion processing and image block size conversion processing; The intermediate image data is converted into the image block data to be processed.

19. The device according to claim 12, wherein: The plurality of continuous rows of the image block data to be processed are sequentially stored in a plurality of continuous address spaces.

20. The device according to claim 12, wherein: The multiple processor cores are further configured to perform the following operations to extract features from the image block data to be processed to obtain image block features: Linear projection is performed on the image block data to be processed to obtain the image block features.

21. The apparatus according to claim 12, wherein the plurality of processor cores are further configured to: At least one image block feature corresponding to the image to be processed is processed based on the attention mechanism to obtain an image processing result.

22. The device according to claim 18, wherein The plurality of processor cores include a first processor core and a second processor core, The first processor core is configured to: perform at least one of a plurality of preset processes on the partially decoded image data; The partially decoded image data is obtained by using the first processor core to perform image decoding on a part of the image to be decoded; The second processor core is configured to: perform feature extraction on the image block data to be processed to obtain image block features.

23. An image processing device comprising the apparatus as claimed in claims 12 to 22.

24. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

25. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 11.

26. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images