Image processing method, device, electronic device and storage medium

Abstract

The present application is applicable to the field of image processing technology, and provides an image processing method, device, electronic device and storage medium, including: applied to a processor, including: cutting a target image to be transformed based on a first size to obtain multiple image blocks, the first size is a size determined according to the available space of a tightly coupled memory and an affine transformation parameter, and the size of any image block is less than or equal to the first size; for each image block, storing the image block in the tightly coupled memory, performing affine transformation processing on the image block according to the affine transformation parameter, and obtaining the transformed image block from the tightly coupled memory; determining the transformed target image according to each transformed image block. The present application can ensure the normal implementation of the affine transformation of an image.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to an image processing method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Affine transformation is a linear transformation in mathematics and computer graphics, including translation, rotation, scaling, shearing and other operations. Since affine transformation can preserve the flatness and parallelism of two-dimensional images, affine transformation is currently widely used in computer vision.

[0003] Currently, images are usually processed by affine transformation through processors. However, with the development of computer vision, the resolution and size of images are getting larger and larger. When the amount of data of images that need to be affine transformed is large, the tightly coupled memory in the processor may not be able to store the images before and after the transformation at the same time, resulting in the affine transformation of the image cannot be performed normally. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, an apparatus, an electronic device, and a storage medium, which can ensure the normal implementation of the affine transformation of the image.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0006] Cutting the target image to be transformed based on a first size to obtain a plurality of image blocks, wherein the first size is a size determined according to available space of a tightly coupled memory and an affine transformation parameter, and a size of any of the image blocks is less than or equal to the first size;

[0007] For each of the image blocks, storing the image block in the tightly coupled memory, performing affine transformation processing on the image block according to the affine transformation parameters, and obtaining the transformed image block from the tightly coupled memory;

[0008] The transformed target image is determined according to each of the transformed image blocks.

[0009] In a possible implementation manner of the first aspect, before the step of cutting the target image to be transformed based on the first size to obtain a plurality of image blocks, the step further includes:

[0010] Determining whether the available space meets the storage requirement corresponding to the target image according to the size of the target image and the affine transformation parameters;

[0011] Correspondingly, the target image to be transformed is cut based on the first size to obtain a plurality of image blocks, including:

[0012] When it is determined that the available space does not meet the storage requirement, the target image is cut based on the first size to obtain a plurality of image blocks.

[0013] In a possible implementation manner of the first aspect, before the target image is segmented based on the first size to obtain a plurality of image blocks, the method further includes:

[0014] The first size is determined according to the available space, the affine transformation parameters, and a pixel depth of the target image.

[0015] In a possible implementation manner of the first aspect, determining the first size according to the available space, the affine transformation parameter, and the pixel depth of the target image includes:

[0016] Determine the actual available space of the tightly coupled memory according to the data processing mode of the tightly coupled memory and the available space;

[0017] The first size is determined according to the actual available space, the affine transformation parameter, and the pixel depth.

[0018] In a possible implementation manner of the first aspect, the tightly coupled memory is a tightly coupled memory of an NPU, and for each of the image blocks, storing the image block in the tightly coupled memory, performing affine transformation processing on the image block according to the affine transformation parameters, and obtaining the transformed image block from the tightly coupled memory includes:

[0019] For any of the image blocks, the image block and the affine transformation parameters are sent to the NPU, and the NPU is used to perform affine transformation processing on the image block stored in the tightly coupled memory according to the affine transformation parameters to obtain the transformed image block;

[0020] When it is determined that the NPU completes the affine transformation processing on the image block, acquiring the transformed image block from the NPU;

[0021] The steps of sending the image block and the affine transformation parameters to the NPU and obtaining the transformed image block from the NPU are repeatedly performed based on each of the image blocks until the transformed image blocks corresponding to all the image blocks are obtained.

[0022] In a possible implementation manner of the first aspect, before the step of cutting the target image to be transformed based on the first size to obtain a plurality of image blocks, the step further includes:

[0023] Determining a first starting position corresponding to each transformed image block according to a second size, where the second size is a size of the transformed image block determined according to the target image and the affine transformation parameter;

[0024] Determine a corresponding second starting position according to each of the first starting positions and the affine transformation parameters, wherein the second starting position is a first pixel position of the image block before transformation;

[0025] Correspondingly, the target image to be transformed is cut based on the first size to obtain a plurality of image blocks, including:

[0026] The target image is cut based on each of the second starting positions and the first size to obtain a plurality of the image blocks.

[0027] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0028] a cutting module, configured to cut a target image to be transformed based on a first size to obtain a plurality of image blocks, wherein the first size is a size determined according to an available space of a tightly coupled memory and an affine transformation parameter, and a size of any of the image blocks is less than or equal to the first size;

[0029] a processing module, configured to store each of the image blocks in the tightly coupled memory, perform affine transformation processing on the image block according to the affine transformation parameters, and obtain the transformed image block from the tightly coupled memory;

[0030] The target image determination module is used to determine the transformed target image according to each transformed image block.

[0031] In a possible implementation manner of the second aspect, the image processing apparatus further includes:

[0032] A judging module, used for judging whether the available space meets the storage requirement corresponding to the target image according to the size of the target image and the affine transformation parameters;

[0033] Correspondingly, the cutting module is specifically used for:

[0034] When it is determined that the available space does not meet the storage requirement, the target image is cut based on the first size to obtain a plurality of image blocks.

[0035] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the image processing method described in any one of the first aspects above when executing the computer program.

[0036] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image processing method described in any one of the first aspects above are implemented.

[0037] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the image processing method described in any one of the first aspects above.

[0038] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0039] In the embodiment of the present application, before the target image to be transformed is affine transformed, the target image is first cut into a plurality of image blocks according to the first size. Since the first size is determined according to the available space of the tightly coupled memory and the affine transformation parameters, that is, based on the storage requirement that the tightly coupled memory needs to store the images before and after the transformation at the same time, the maximum size of the image before the transformation that meets the storage requirement is determined according to the available space of the tightly coupled memory and the affine transformation parameters. Therefore, for each image block whose size is less than or equal to the first size, the tightly coupled memory can better store the image block and the transformed image block at the same time, so that the processor can better perform affine transformation processing on the image block in the tightly coupled memory, and then determine the transformed target image according to each transformed image block, thereby ensuring the normal implementation of the affine transformation of the target image, and at the same time can reduce the requirements for the tightly coupled memory space of the processor, without setting up an additional storage device, which is beneficial to practical application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0041] Figure 1 It is a flowchart of an image processing method provided by an embodiment of the present application;

[0042] Figure 2 is a flowchart of another image processing method provided in an embodiment of the present application;

[0043] Figure 3 is a flowchart of another image processing method provided in an embodiment of the present application;

[0044] Figure 4 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0045] Figure 5It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0047] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0048] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0049] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0050] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0051] Embodiment 1:

[0052] Figure 1 A flowchart of an image processing method provided by an embodiment of the present application is shown, which is applied to a processor and is described in detail as follows:

[0053] Step S101, cutting a target image to be transformed based on a first size to obtain multiple image blocks, wherein the first size is a size determined according to available space of a tightly coupled memory and affine transformation parameters, and the size of any of the image blocks is less than or equal to the first size.

[0054] Tightly Coupled Memory (TCM) is a high-speed memory directly connected to the processor through a dedicated interface. Since the processor is directly connected to the TCM, the access latency of the processor to the TCM can be greatly reduced. Therefore, the embodiment of the present application uses the TCM to store images before and after affine transformation, and accelerates the execution of affine transformation processing through the TCM with low latency and high bandwidth, effectively improving the efficiency of image affine transformation.

[0055] The above-mentioned affine transformation parameters are parameters used when performing affine transformation on the target image to be transformed, and usually include linear transformation parameters and translation parameters.

[0056] It can be understood that the method for obtaining the above-mentioned target image to be transformed and its corresponding affine transformation parameters can be determined according to the actual application scenario, and the embodiment of the present application does not impose any specific limitation on the method for obtaining the target image and its corresponding affine transformation parameters.

[0057] As an example, the above image processing method can be applied to a panorama generation scenario. In a panorama generation scenario, a panorama is usually obtained by stitching at least two images (assuming they are called panoramas to be stitched). Before stitching, at least one image to be stitched usually needs to be affine transformed to achieve image alignment or shape adjustment through affine transformation. At this time, the target image is the image to be stitched to be transformed. The target image can be an image taken by a user through an ordinary camera device (such as a mobile phone camera, etc.), an image input by a user, or an image automatically generated based on text input by the user.

[0058] As another example, the above-mentioned image processing method can be applied to the field of target tracking, and the target tracking task usually requires locating and tracking the target in continuous video frames captured by various camera devices (such as surveillance cameras). Affine transformation is used to estimate the movement of the target between continuous video frames to help the tracking algorithm maintain accurate tracking of the target in complex scenes. At this time, the target image to be transformed can be each video frame, or a video frame in which the size difference between the suspected target and the target in a given reference image is large, etc., so that the suspected target in the video frame and the target in the reference image can have the same size or angle through affine transformation, which is conducive to improving the accuracy of target recognition and thus improving the reliability of target tracking.

[0059] In the embodiment of the present application, since the first size is determined based on the available space of the tightly coupled memory and the affine transformation parameters, it can reflect the maximum size of the image block before the transformation allowed when the tightly coupled memory satisfies the requirement of simultaneously storing the image blocks before and after the transformation. Therefore, based on the first size, the target image is cut into multiple image blocks, so that the size of any of the above image blocks is less than or equal to the first size, thereby ensuring that the tightly coupled memory can simultaneously store the image blocks before and after the transformation, thereby ensuring the normal implementation of the affine transformation of the image blocks.

[0060] Step S102: for each of the above image blocks, store the above image blocks in the above tightly coupled memory, perform affine transformation processing on the above image blocks according to the above affine transformation parameters, and obtain the transformed above image blocks from the above tightly coupled memory.

[0061] It should be noted that when performing affine transformation processing on an image block, one image block is usually stored in a tightly coupled memory for affine transformation processing at a time. After completing the affine transformation processing of the current image block and obtaining the transformed image block, the image blocks before and after the transformation in the tightly coupled memory can be deleted, and then any image block among the image blocks that have not been affine transformed can be stored in the tightly coupled memory for affine transformation processing. Through the above processing, it is ensured that the tightly coupled memory with limited space can simultaneously store the image block before and after the transformation, that is, it is ensured that the affine transformation of the image block can be normally implemented.

[0062] In some embodiments, during the process of performing affine transformation processing on image blocks in a tightly coupled memory, if the available space of the tightly coupled memory changes and the newly added available space meets the storage conditions of the image blocks (i.e., storing the image blocks before and after the transformation at the same time), two or more image blocks can be stored in the tightly coupled memory at the same time for affine transformation processing.

[0063] Step S103, determining the transformed target image according to each of the transformed image blocks.

[0064] Since in the process of performing affine transformation on the target image, the target image is cut into multiple image blocks and affine transformation is performed separately, each transformed image block obtained is not a complete image. Therefore, after obtaining the transformed image block corresponding to each image block, it is necessary to combine the position of the transformed image block in the target image and splice the transformed image blocks into a complete image to obtain the transformed target image.

[0065] As an example, when the server performs affine transformation on image blocks, it may perform affine transformation on each image block in sequence. Then, when determining the transformed target image, it may directly splice the changed image blocks in sequence to obtain the transformed target image.

[0066] As another example, when the server performs cutting processing on the target image, it can mark the serial number of each image block according to the position of the first pixel of each image block. When performing affine transformation processing, it can perform affine transformation processing on each image block in any order. When splicing, the serial numbers corresponding to the changed image blocks are installed to obtain the changed target image.

[0067] In the embodiment of the present application, before the target image to be transformed is affine transformed, the target image is first cut into a plurality of image blocks according to the first size. Since the first size is determined according to the available space of the tightly coupled memory and the affine transformation parameters, that is, the maximum size of the image block before transformation can be determined based on the storage requirement that the tightly coupled memory needs to store the images before and after transformation at the same time, in combination with the available space of the tightly coupled memory and the affine transformation parameters, so for each image block whose size is less than or equal to the first size, the tightly coupled memory can better store the image block before transformation and the image block after transformation at the same time, so that the processor can better perform affine transformation processing on the image block in the tightly coupled memory, and then determine the transformed target image according to each transformed image block, so as to ensure the normal implementation of the affine transformation of the target image, and at the same time can reduce the requirements for the tightly coupled memory space of the processor, without setting up additional storage devices, which is beneficial to practical applications.

[0068] In some embodiments, before the above step S101, the method further includes:

[0069] It is determined whether the available space meets the storage requirement corresponding to the target image according to the size of the target image and the affine transformation parameters.

[0070] Correspondingly, the above step S101 includes:

[0071] When it is determined that the available space does not meet the storage requirement, the target image is cut based on the first size to obtain a plurality of image blocks.

[0072] To improve the processing efficiency of affine transformation, after acquiring the target image, the processor can first analyze and determine the storage space required when performing affine transformation on the target image, that is, determine the storage requirement corresponding to the target image. The storage requirement reflects the storage space required to store both the target image before transformation and the target image after transformation.

[0073] Among them, since the affine transformation parameters determine the specific transformation method required for the target image, that is, the affine transformation parameters can reflect the relationship between the target image before the transformation and the target image after the transformation, therefore, in the embodiment of the present application, the affine transformation parameters can be combined to analyze the storage requirements corresponding to the target image, and then, it can be better judged whether the current available space of the tightly coupled memory meets the storage requirements.

[0074] like Figure 2 As shown, if it is determined that the available space of the tightly coupled memory does not meet the storage requirements corresponding to the target image, the first size is determined according to the target image and the affine transformation parameters, and then the target image is cut into multiple image blocks according to the first size to ensure the normal implementation of the affine transformation of the target image.

[0075] If it is determined that the available space of the tightly coupled memory can meet the storage requirements corresponding to the target image, the complete target image can be directly affine transformed in the tightly coupled memory to complete the affine transformation of the target image as quickly as possible.

[0076] In some embodiments, when the target image is a multi-channel image, the target image may be split first to obtain a plurality of single-channel images, and the plurality of single-channel images may be spliced ​​into a single-channel image to obtain a single-channel target image. When determining whether the available space of the tightly coupled memory meets the storage requirements corresponding to the target image, the determination is made based on the single-channel target image, and when the cutting process is performed based on the first-size target image, the cutting process is performed on the single-channel target image.

[0077] As another example, when the target image is a multi-channel image, after splitting it into multiple single-channel images, the storage requirements of each single-channel image can be judged separately. When it is determined that the storage requirements corresponding to the single-channel image cannot be met, the first size is determined based on the single-channel image and the single-channel image is cut.

[0078] In an embodiment of the present application, it is first determined whether the currently available space of the tightly coupled memory can meet the storage requirements of the target image. If the storage requirements cannot be met, the target image is cut to ensure the normal implementation of the affine transformation. While ensuring the normal implementation of the affine transformation, the efficiency of the image affine transformation is improved as much as possible.

[0079] In some embodiments, before the above step S101, the method further includes:

[0080] The first size is determined according to the available space, the affine transformation parameters and the pixel depth of the target image.

[0081] In order to improve the efficiency of the image affine transformation as much as possible, the maximum size of the image block before the transformation can be analyzed based on the available space of the tightly coupled memory and the affine transformation parameters while satisfying the storage condition (i.e., being able to store the image blocks before and after the transformation at the same time), and the required first size is obtained, so that when the image blocks with a size less than or equal to the first size are subsequently affine transformed, the affine transformation of the image blocks can be normally implemented.

[0082] Among them, taking into account that the pixel depth of the image will affect the size of the storage space required to be occupied by each pixel of the image, therefore, when determining the first size, the size of the target image, the pixel depth, the available space of the tightly coupled memory and the affine transformation parameters are combined to calculate the first size to ensure the accuracy of the first size, and further ensure that image blocks with a size less than or equal to the first size can meet the storage condition, thereby ensuring the normal implementation of the affine transformation processing.

[0083] As an example, the coordinates of each vertex of the transformed image block can be calculated based on the forward affine transformation parameters and the vertex coordinates of the image block before the transformation, and then the second size of the transformed image block can be calculated based on the coordinates of the two diagonal vertices, and then the first size of the image block before the transformation can be calculated based on the second size.

[0084] As another example, considering that each vertex in the target image usually corresponds to a vertex in the transformed target image after affine transformation, and the two diagonal vertices in the image can usually reflect the maximum size of the image in the vertical and horizontal directions, therefore, in the process of determining the first size, the coordinates of each vertex of the image block before transformation can be reversely calculated based on the inverse affine transformation parameters and the vertex coordinates of the transformed image block, and then the required first size can be calculated based on the coordinates of the two diagonal vertices. Through the above reverse calculation method, it can be better ensured that the mapping of each pixel in the transformed image block is within the image block before transformation, which is conducive to improving the accuracy and reliability of the affine transformation.

[0085] For example, assuming that the transformed image block is a square, and assuming that the second size of the transformed image block is rd (at this time, rd is an unknown number), the coordinates of its four vertices p00-p11 are respectively: p00 (0, 0), p01 (rd-1, 0), p10 (0, rd-1), p11 (rd-1, rd-1).

[0086] Assume that the inverse affine transformation parameters calculated according to the affine transformation parameters corresponding to the target image are: Based on the inverse affine transformation parameters and the coordinates of the transformed image block, the four vertices p'00-p'11 of the image block before transformation can be calculated as follows: p'00(0, 0), p'01(a1*(rd-1), a3*(rd-1)), p'10(a2*(rd-1), a4*(rd-1)), p'11((a1+a2)*(rd-1), (a3+a4)*(rd-1)).

[0087] After determining the coordinates of the four vertices of the image block before transformation, the coordinates of the two diagonal vertices (x min ,y min ), (x max ,y max ), the maximum size of the image block can be determined according to the vertex coordinates of the two diagonal corners and used as the first size.

[0088] Optionally, the coordinates of the two vertices of the diagonal angle can be expressed as follows:

[0089] x min =min(a1,a2,a1+a2)*(rd-1)

[0090] x max =max(a1,a2,a1+a2)*(rd-1)

[0091] y min =min(a3,a4,a3+a4)*(rd-1)

[0092] y max =max(a3,a4,a3+a4)*(rd-1)

[0093] Then the size of the image block before transformation (ie, the first size) can be expressed as follows:

[0094] rs w =x max -x min

[0095] rs h =y max -y min

[0096] Among them, rs w Represents the length of the image block before transformation, rs h Indicates the width of the image block before transformation, x max Indicates the maximum value of the two vertex coordinates of the diagonal image block before transformation in the horizontal direction, x minIndicates the minimum value of the two vertex coordinates of the diagonal image block before transformation in the horizontal direction, y max Represents the maximum value of the two vertex coordinates of the diagonal image block before transformation in the horizontal direction, y min Indicates the minimum value of the horizontal coordinates of the two diagonal vertices of the image block before transformation.

[0097] Finally, the first size of the image block before transformation is represented by the second size of the transformed image block and the affine transformation parameter, and then an equation is established according to the first size, the second size, the pixel depth and the available space, and the equation is solved to obtain the actual values ​​of the first size and the second size.

[0098] As an example, an equation of the following form may be obtained according to the first size, the second size, the pixel depth, and the available space:

[0099] rs w *rs n +rd*rd<=dm_size / cn

[0100] Among them, dm_size represents the available space of the tightly coupled memory, and cn represents the pixel depth of the target image.

[0101] As an example, in the process of solving the equation, it can be assumed that the above maximum and minimum values ​​are expressed as follows:

[0102]

[0103] Then the diagonal coordinates of the two points of the image block before transformation are (coffe[0]*(rd-1), coffe[2]*(rd-1)) and coffe[1]*(rd-1), coffe[3]*(rd-1)), that is, the first size can be expressed as follows:

[0104] rs w =x max -x min =ceil(coffe(1)-coffe(0)))*(rd-1)

[0105] rs h =y max -y min =ceil(coffe(3)-coffe(2)))*(rd-1)

[0106] Among them, ceil means rounding up.

[0107] Assuming k = ceil(coffe(1) - coffe(0)) * ceil(coffe(3) - coffe(2)), the above equation can be expressed as follows:

[0108] rs w *rs h +rd*rd=k*(rd-1)*(rd-1)+rd*rd

[0109] =(k+1)*rd*rd<=dm_size / cn

[0110] Then, by solving the equation (k+1)*rd*rd<=dm_size / cn, the required first size and second size can be obtained.

[0111] In an embodiment of the present application, the maximum size of the image block that meets the storage conditions is analyzed in combination with the pixel depth of the target image, the affine transformation parameters used in the actual affine transformation processing, and the current available space of the tightly coupled memory, so that the processor can better perform affine transformation processing on image blocks with a size less than or equal to the first size in the tightly coupled memory.

[0112] In some embodiments, determining the first size according to the available space, the affine transformation parameters, and the pixel depth of the target image includes:

[0113] The actual available space of the tightly coupled memory is determined according to the data processing mode of the tightly coupled memory and the available space.

[0114] The first size is determined according to the actual available space, the affine transformation parameter and the pixel depth.

[0115] The data processing method of the tightly coupled memory is a method of processing the data stored in the tightly coupled memory, including but not limited to data processing methods such as ping-pong scheduling mode, local memory mode, dynamic allocation mode and static allocation mode.

[0116] Since different tightly coupled memories may use different data processing methods to process data, and different data processing methods have different effects on the storage space required in the image processing process, therefore, in the embodiment of the present application, the actual available space of the tightly coupled memory is first calculated based on the data processing method of the tightly coupled memory and the available space of the tightly coupled memory (that is, the storage space that can be used to store the image before and after the affine transformation).

[0117] After the relatively accurate actual available space is determined, the first size is calculated based on the actual available space, corresponding affine transformation parameters, and the pixel depth of the target image.

[0118] For example, it is assumed that the data processing mode of the tightly coupled memory of model A includes a ping-pong scheduling mode and a non-ping-pong scheduling mode. For the tightly coupled memory of model A, since the ping-pong scheduling mode can store data alternately through two buffers to achieve seamless connection between data processing and transmission, but it requires more storage space, therefore, when the data processing mode of the tightly coupled memory is the ping-pong scheduling mode, the actual available space of the tightly coupled memory is half of the available space. For example, it is assumed that the available space of the tightly coupled memory A1 is 100 megabytes, then the actual available space of the tightly coupled memory A1 in the ping-pong scheduling mode is 50 megabytes.

[0119] When the data processing mode of the tightly coupled memory is non-ping-pong scheduling mode, its actual available space is equal to the available space. For example, assuming that the available space of the tightly coupled memory A2 is 100 megabytes, the actual available space of the tightly coupled memory A1 in the non-ping-pong scheduling mode is 100 megabytes.

[0120] It can be understood that when it is necessary to determine whether the available space of the tightly coupled memory can meet the storage requirement of the target image, it can be determined whether the relatively accurate actual available space can meet the storage requirement of the target image.

[0121] In the embodiment of the present application, the influence of the data processing method of the tightly coupled memory on the storage space required for data processing is fully considered, the data processing method and the available space of the tightly coupled memory are combined to calculate the actually available space of the tightly coupled memory, and then the first size is calculated based on the actually available space and the affine transformation parameters, so as to effectively ensure the reliability and accuracy of the first size.

[0122] In some embodiments, the tightly coupled memory is a tightly coupled memory of an NPU, and the step S102 includes:

[0123] For any of the above image blocks, the above image block and the above affine transformation parameters are sent to the above NPU, and the above NPU is used to perform affine transformation processing on the above image block stored in the above tightly coupled memory according to the above affine transformation parameters to obtain the transformed image block.

[0124] When it is determined that the NPU completes the affine transformation processing on the image block, the transformed image block is obtained from the NPU.

[0125] The steps of sending the image blocks and the affine transformation parameters to the NPU and obtaining the transformed image blocks from the NPU are repeatedly performed based on each of the image blocks until the transformed image blocks corresponding to all the image blocks are obtained.

[0126] NPU (Neural Processing Unit) is an embedded neural network processor, which is a computing chip or processor designed specifically for accelerating neural networks and deep learning algorithms. Since NPU adopts a data-driven parallel computing architecture, it is particularly good at processing image data, which can significantly improve image processing efficiency and reduce power consumption. Therefore, in the embodiment of the present application, the target image can be processed by NPU to perform affine transformation to improve the speed and efficiency of affine transformation operations.

[0127] For multiple image blocks before transformation (assuming N, N is greater than 1), the processor sends any one of the N image blocks and affine transformation parameters to the NPU, so that the NPU can perform efficient affine transformation processing on the image block.

[0128] The NPU receives and stores the image block, and then performs affine transformation processing on the image block according to the affine transformation parameters to obtain a transformed image block. After determining that the NPU has completed the affine transformation processing on the image block, the processor can obtain the transformed image block from the NPU.

[0129] As an example, the NPU may send a completion notification to the processor, indicating that the affine transformation processing of the image block has been completed through the completion notification, so that the processor can obtain the transformed image block from the NPU in a timely manner.

[0130] As another example, a subscription service may be set up to take an image block processed by the NPU with affine transformation as a message. After the NPU completes the affine transformation processing on the image block, it may send the transformed image block as a message directly to a processor that subscribes to the message.

[0131] like Figure 3 As shown, since the tightly coupled memory of the NPU is limited, an image block before and after the transformation is usually stored at the same time. Therefore, the processor needs to perform affine transformation processing on the next image block after obtaining the transformed image block corresponding to the image block currently undergoing affine transformation, that is, send any image block among the image blocks that have not undergone affine transformation to the NPU for affine transformation processing until the transformed image blocks corresponding to all image blocks are obtained.

[0132] In some embodiments, before the above step S101, the method further includes:

[0133] A first starting position corresponding to each transformed image block is determined according to a second size, wherein the second size is a size of the transformed image block determined according to the target image and the affine transformation parameters.

[0134] A corresponding second starting position is determined according to each of the first starting positions and the affine transformation parameters, wherein the second starting position is the first pixel position of the image block before transformation.

[0135] Correspondingly, the target image to be transformed is cut based on the first size to obtain multiple image blocks, including:

[0136] The target image is cut based on each of the second starting positions and the first size to obtain a plurality of the image blocks.

[0137] Since the affine transformation process usually requires calculating the pixel value of each new pixel in the transformed image based on the pixel values ​​of one or more pixels in the image before the transformation, and the embodiment of the present application performs affine transformation processing on multiple image blocks obtained by cutting the target image respectively, if the image block does not have the pixel value of the required pixel, it may affect the calculation of the pixel value of the new pixel. Therefore, in order to ensure efficient and accurate affine transformation processing of the image block, before cutting the target image according to the first size, the starting position of each transformed image block in the transformed target image can be determined according to the second size (that is, the size of the transformed image block) to obtain the first starting position corresponding to each transformed image block, that is, determine the position of the first pixel of each transformed image block.

[0138] Since the first pixel of the transformed image block usually corresponds to the first pixel of the image block before the transformation, after determining the position of each first pixel after the transformation, for each transformed image block, the position of the first pixel of the image block before the transformation can be reversely calculated based on its corresponding first starting position and affine transformation parameters to obtain the second starting position.

[0139] After obtaining the first pixel position (i.e., the second starting position) of the image block before transformation corresponding to each transformed image block through the above calculation, the target image can be cut according to the first size and each second starting position to obtain each image block before transformation, and the size of any image block is less than or equal to the first size.

[0140] As an example, the coordinates of the first pixel of the transformed image block (i.e., the first starting position) can be expressed as (xd, yd), and the coordinate position (xs, ys) corresponding to the first starting position in the image block before the transformation can be expressed as follows:

[0141] xs=a1*xd+a2*yd+tx

[0142] ys=a3*xd+a4*yd+ty

[0143] Then the coordinates of the first pixel of the corresponding image block before transformation (xtile ,y tile ), that is, the second starting position can be expressed as follows:

[0144] x tile =xs+x min

[0145] y tile =ys+y min

[0146] Since the first pixel position of the image block before the transformation is calculated based on the position of the first pixel of the corresponding image block after the transformation, that is, the affine transformation parameters and the pixel value calculation method of the new pixel (such as bilinear interpolation or cubic interpolation, etc.) can be combined in the calculation process to analyze and calculate the pixels required for the pixel value of the new pixel after the transformation, and the starting position of the corresponding image block (that is, the second starting position) is obtained by reverse calculation. Therefore, the target image is cut based on the calculated second initial positions and the first size, which can ensure that each image block obtained contains all the pixels required for the pixel value calculation of the new pixel in the affine transformation process, effectively ensuring the efficient implementation of the affine transformation processing of the image block.

[0147] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0148] Embodiment 2:

[0149] Corresponding to the image processing method described in the above embodiment, Figure 4 A structural block diagram of an image processing device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0150] Reference Figure 4 The device comprises: a cutting module 41, a processing module 42 and a target image determination module 43.

[0151] The cutting module 41 is used to cut the target image to be transformed based on a first size to obtain multiple image blocks. The above-mentioned first size is a size determined according to the available space of the tightly coupled memory and the affine transformation parameters. The size of any of the above-mentioned image blocks is less than or equal to the above-mentioned first size.

[0152] The processing module 42 is used for storing each of the above-mentioned image blocks in the above-mentioned tightly coupled memory, performing affine transformation processing on the above-mentioned image blocks according to the above-mentioned affine transformation parameters, and obtaining the transformed image blocks from the above-mentioned tightly coupled memory.

[0153] The target image determination module 43 is used to determine the transformed target image according to each of the transformed image blocks.

[0154] In the embodiment of the present application, before the target image to be transformed is affine transformed, the target image is first cut into a plurality of image blocks according to the first size. Since the first size is determined according to the available space of the tightly coupled memory and the affine transformation parameters, that is, the maximum size of the image block before transformation can be determined based on the storage requirement that the tightly coupled memory needs to store the images before and after transformation at the same time, in combination with the available space of the tightly coupled memory and the affine transformation parameters, so for each image block whose size is less than or equal to the first size, the tightly coupled memory can better store the image block before transformation and the image block after transformation at the same time, so that the processor can better perform affine transformation processing on the image block in the tightly coupled memory, and then determine the transformed target image according to each transformed image block, so as to ensure the normal implementation of the affine transformation of the target image, and at the same time can reduce the requirements for the tightly coupled memory space of the processor, without setting up additional storage devices, which is beneficial to practical applications.

[0155] In some embodiments, the image processing apparatus further includes:

[0156] The judgment module is used to judge whether the available space meets the storage requirement corresponding to the target image according to the size of the target image and the affine transformation parameters.

[0157] Correspondingly, the cutting module 41 is used for:

[0158] When it is determined that the available space does not meet the storage requirement, the target image is cut based on the first size to obtain a plurality of image blocks.

[0159] In some embodiments, the image processing apparatus further includes:

[0160] The first size determination module is used to determine the first size according to the available space, the affine transformation parameters and the pixel depth of the target image.

[0161] In some embodiments, the image processing apparatus further includes:

[0162] The actual available space calculation module is used to determine the actual available space of the tightly coupled memory according to the data processing mode of the tightly coupled memory and the available space.

[0163] The first size calculation module is used to determine the first size according to the actual available space, the affine transformation parameters and the pixel depth.

[0164] In some embodiments, the tightly coupled memory is a tightly coupled memory of an NPU, and the processing module 42 includes:

[0165] The transformation unit is used to send the image block and the affine transformation parameters to the NPU for any of the above-mentioned image blocks, and the NPU is used to perform affine transformation processing on the image block stored in the above-mentioned tightly coupled memory according to the above-mentioned affine transformation parameters to obtain the transformed image block.

[0166] The acquisition unit is used to acquire the transformed image block from the NPU when it is determined that the NPU completes the affine transformation processing on the image block.

[0167] The loop unit is used to repeatedly execute the steps of sending the above-mentioned image blocks and the above-mentioned affine transformation parameters to the above-mentioned NPU to obtaining the transformed above-mentioned image blocks from the above-mentioned NPU based on each of the above-mentioned image blocks, until the transformed above-mentioned image blocks corresponding to all the above-mentioned image blocks are obtained.

[0168] In some embodiments, the image processing apparatus further includes:

[0169] The first starting position calculation module is used to determine the first starting position corresponding to each transformed image block according to a second size, where the second size is the size of the transformed image block determined according to the target image and the affine transformation parameters.

[0170] The second starting position calculation module is used to determine the corresponding second starting position according to each of the first starting positions and the affine transformation parameters, wherein the second starting position is the first pixel position of the image block before transformation.

[0171] Correspondingly, the cutting module 41 is specifically used for:

[0172] The target image is cut based on each of the second starting positions and the first size to obtain a plurality of the image blocks.

[0173] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0174] Embodiment three:

[0175] Figure 5 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 5 As shown, the electronic device 5 of this embodiment includes: at least one processor 50 ( Figure 5Only one processor is shown in the figure), a memory 51, and a computer program 52 stored in the memory 51 and executable on the at least one processor 50, wherein the processor 50 implements the steps in any of the above-mentioned method embodiments when executing the computer program 52.

[0176] The electronic device 5 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device may include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will appreciate that Figure 5 It is only an example of the electronic device 5 and does not constitute a limitation on the electronic device 5. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0177] The processor 50 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0178] In some embodiments, the memory 51 may be an internal storage unit of the electronic device 5, such as a hard disk or memory of the electronic device 5. In other embodiments, the memory 51 may also be an external storage device of the electronic device 5, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. Further, the memory 51 may also include both an internal storage unit and an external storage device of the electronic device 5. The memory 51 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 51 may also be used to temporarily store data that has been output or is to be output.

[0179] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0180] An embodiment of the present application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps in any of the above-mentioned method embodiments when executing the computer program.

[0181] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0182] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0183] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0184] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0185] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0186] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0187] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: Applicable to processors, including: Cutting the target image to be transformed based on a first size to obtain a plurality of image blocks, wherein the first size is a size determined according to available space of a tightly coupled memory and an affine transformation parameter, and a size of any of the image blocks is less than or equal to the first size; For each of the image blocks, storing the image block in the tightly coupled memory, performing affine transformation processing on the image block according to the affine transformation parameters, and obtaining the transformed image block from the tightly coupled memory; The transformed target image is determined according to each of the transformed image blocks.

2. The image processing method according to claim 1, characterized in that: Before the target image to be transformed is cut based on the first size to obtain a plurality of image blocks, the method further includes: Determining whether the available space meets the storage requirement corresponding to the target image according to the size of the target image and the affine transformation parameters; Correspondingly, the target image to be transformed is cut based on the first size to obtain a plurality of image blocks, including: When it is determined that the available space does not meet the storage requirement, the target image is cut based on the first size to obtain a plurality of image blocks.

3. The image processing method according to claim 2, characterized in that: Before the target image is cut based on the first size to obtain a plurality of image blocks, the method further includes: The first size is determined according to the available space, the affine transformation parameters, and a pixel depth of the target image.

4. The image processing method according to claim 3, characterized in that: The determining the first size according to the available space, the affine transformation parameter, and the pixel depth of the target image includes: Determine the actual available space of the tightly coupled memory according to the data processing mode of the tightly coupled memory and the available space; The first size is determined according to the actual available space, the affine transformation parameter, and the pixel depth.

5. The image processing method according to claim 1, wherein: The tightly coupled memory is a tightly coupled memory of the NPU, and for each of the image blocks, the image block is stored in the tightly coupled memory, the image block is affine transformed according to the affine transformation parameters, and the transformed image block is obtained from the tightly coupled memory, including: For any of the image blocks, the image block and the affine transformation parameters are sent to the NPU, and the NPU is used to perform affine transformation processing on the image block stored in the tightly coupled memory according to the affine transformation parameters to obtain the transformed image block; When it is determined that the NPU completes the affine transformation processing on the image block, acquiring the transformed image block from the NPU; The steps of sending the image block and the affine transformation parameters to the NPU and obtaining the transformed image block from the NPU are repeatedly performed based on each of the image blocks until the transformed image blocks corresponding to all the image blocks are obtained.

6. The image processing method according to any one of claims 1 to 5, characterized in that: Before the target image to be transformed is cut based on the first size to obtain a plurality of image blocks, the method further includes: Determining a first starting position corresponding to each transformed image block according to a second size, where the second size is a size of the transformed image block determined according to the target image and the affine transformation parameter; Determine a corresponding second starting position according to each of the first starting positions and the affine transformation parameters, wherein the second starting position is a first pixel position of the image block before transformation; Correspondingly, the target image to be transformed is cut based on the first size to obtain a plurality of image blocks, including: The target image is cut based on each of the second starting positions and the first size to obtain a plurality of the image blocks.

7. An image processing device, characterized in that: include: a cutting module, configured to cut a target image to be transformed based on a first size to obtain a plurality of image blocks, wherein the first size is a size determined according to an available space of a tightly coupled memory and an affine transformation parameter, and a size of any of the image blocks is less than or equal to the first size; a processing module, configured to store each of the image blocks in the tightly coupled memory, perform affine transformation processing on the image block according to the affine transformation parameters, and obtain the transformed image block from the tightly coupled memory; The target image determination module is used to determine the transformed target image according to each transformed image block.

8. The image processing device according to claim 7, wherein: The image processing device further comprises: A judging module, used for judging whether the available space meets the storage requirement corresponding to the target image according to the size of the target image and the affine transformation parameters; Correspondingly, the cutting module is specifically used for: When it is determined that the available space does not meet the storage requirement, the target image is cut based on the first size to obtain a plurality of image blocks.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images