Image processing apparatus, method, electronic device and storage medium

By introducing a first windowing unit and a first processing unit into the image processing device, the data calculation process of the image pyramid is simplified, the processing efficiency is improved and the power consumption is reduced, thus solving the problem of complex calculations in traditional image processing devices.

CN123118431APending Publication Date: 2026-11-17VIVO MOBILE COMM HANGZHOU CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510636104.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-16
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Traditional image processing devices involve complex calculations in constructing image pyramids, resulting in low efficiency.

Method used

An image processing device including a first windowing unit and a first processing unit is used to acquire the first input data of an N-layer image pyramid and process it sequentially according to the processing time sequence of the valid signals. The image blocks of each layer are processed to simplify the calculation process and construct the image pyramid.

Benefits of technology

It improves image processing efficiency, reduces computing resource consumption and power consumption, and is beneficial for chip hardware implementation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118431A_ABST
    Figure CN123118431A_ABST
Patent Text Reader

Abstract

This application discloses an image processing apparatus, method, electronic device, and storage medium, belonging to the field of image processing technology. The image processing apparatus includes a first processing module and a second processing module. The first processing module includes a first windowing unit and a first processing unit connected in sequence. The first windowing unit is used to acquire first input data for each level of an N-layer image pyramid based on the original image, and transmit the first input data of each level to the first processing unit. The first input data of each level includes at least one first image block. The first processing unit is used to process the first image block corresponding to each first valid signal sequentially according to the first processing time sequence of the first valid signal corresponding to each first image block to obtain second input data for each level, and transmit the second input data of each level to the second processing module. The second processing module is used to construct an image pyramid of the original image based on the second input data of each level.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and specifically relates to an image processing apparatus, method, electronic device and storage medium. Background Technology

[0002] Image pyramids are commonly used for multi-scale detection in image processing and computer vision. By analyzing the multi-resolution characteristics of image pyramids, the adaptability, accuracy, and robustness of electronic devices can be effectively improved. Furthermore, the multi-resolution characteristics of image pyramids enable them to perform exceptionally well when processing raw images with noise, scale variations, and complex backgrounds, and they hold great promise for applications in autonomous driving, medical diagnosis, remote sensing monitoring, and image fusion.

[0003] An image pyramid can be described as a collection of images with different spatial resolutions, each containing all the features of the original image. The original image is decomposed into sub-images with different spatial resolutions after image pyramid transformation. Larger, higher-resolution sub-images are located at the lower levels of the image pyramid, while smaller, lower-resolution sub-images are located at the upper levels.

[0004] Traditional image processing devices employ instantiation modules that involve complex data calculations during the construction of image pyramids, resulting in low image processing efficiency. Summary of the Invention

[0005] The purpose of this application is to provide an image processing apparatus, method, electronic device, and storage medium that can simplify data calculation and improve image processing efficiency.

[0006] In a first aspect, embodiments of this application provide an image processing apparatus, comprising: a first processing module and a second processing module. The first processing module includes a first windowing unit and a first processing unit connected in sequence. The first windowing unit is configured to acquire first input data for each level of an N-layer image pyramid based on an original image, and transmit the first input data for each level to the first processing unit. The first input data for each level includes at least one first image block, and N is an integer greater than 1. The first processing unit is configured to process the first image block corresponding to each first valid signal sequentially according to a first processing timing sequence of the first valid signal corresponding to each first image block to obtain second input data for each level, and transmit the second input data for each level to the second processing module. The second processing module is configured to construct an image pyramid corresponding to the original image based on the second input data for each level.

[0007] Secondly, embodiments of this application provide an image processing method, which includes: acquiring first input data for each level of an N-layer image pyramid based on an original image, wherein the first input data for each level includes at least one first image block, and N is an integer greater than 1; processing the first image block corresponding to each first valid signal sequentially according to the first processing timing of the first valid signal corresponding to each first image block to obtain second input data for each level; and constructing an image pyramid corresponding to the original image based on the second input data for each level.

[0008] Thirdly, embodiments of this application provide an image processing apparatus, which includes an acquisition module and a processing module; the acquisition module is used to acquire first input data for each level of an N-layer image pyramid based on an original image, wherein the first input data for each level includes at least one first image block, and N is an integer greater than 1; the processing module is used to process the first image block corresponding to each first valid signal sequentially according to a first processing time sequence of the first valid signal corresponding to each first image block to obtain second input data for each level; and to construct an image pyramid corresponding to the original image based on the second input data for each level.

[0009] Fourthly, embodiments of this application provide an electronic device that includes an image processing apparatus as described in the first aspect, or an image processing apparatus as described in the third aspect.

[0010] Fifthly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the method described in the second aspect.

[0011] In a sixth aspect, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the second aspect.

[0012] In a seventh aspect, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the second aspect.

[0013] Eighthly, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least two processors to implement the method described in the first aspect.

[0014] In this embodiment, the image processing device includes a first processing module and a second processing module. The first processing module includes a first windowing unit and a first processing unit connected in sequence. The first windowing unit acquires first input data for each level of an N-layer image pyramid based on the original image and transmits the first input data for each level to the first processing unit. The first input data for each level includes at least one first image block, where N is an integer greater than 1. After acquiring at least one first image block included in each first input data, the first processing unit processes the first image block corresponding to each first valid signal sequentially according to the first processing timing of the first valid signal corresponding to each first image block to obtain the second input data for each level. Thus, this solution only requires the first processing unit, which can quickly determine the second input data for each level without affecting the output timing, simplifying the data calculation process in the first processing unit, and then transmitting the second input data for each level to the second processing module. Furthermore, after acquiring the second input data for each level, the second processing module can construct the image pyramid corresponding to the original image based on the second input data for each level, thus improving the image processing efficiency of the entire process. Attached Figure Description

[0015] Figure 1 These are structural diagrams of an N-layer Gaussian pyramid and an N-layer Laplace pyramid provided by existing technology;

[0016] Figure 2a This is a schematic diagram of the image processing device corresponding to the existing 4-layer Gaussian pyramid;

[0017] Figure 2b This is a schematic diagram of the image processing device corresponding to the four-layer Laplacian pyramid provided by existing technology;

[0018] Figure 3 This is one of the structural schematic diagrams of the image processing apparatus provided in the embodiments of this application;

[0019] Figure 4 This is a schematic diagram of the structure of the first window opening unit provided in the embodiment of this application;

[0020] Figure 5 This is a schematic diagram of the windowing of the first input data at the first level provided in the embodiments of this application;

[0021] Figure 6 This is one of the structural schematic diagrams of the first processing unit provided in the embodiments of this application;

[0022] Figure 7 This is a comparative schematic diagram showing the time taken up by the valid signal provided in the embodiments of this application;

[0023] Figure 8This is a schematic diagram of the reading process of the target valid signal provided in the embodiments of this application;

[0024] Figure 9 This is a second schematic diagram of the structure of the first processing unit provided in the embodiments of this application;

[0025] Figure 10 This is one of the schematic diagrams of the first FIFO circuit corresponding to the second timing reconstruction unit provided in the embodiments of this application;

[0026] Figure 11 This is a second schematic diagram of the structure of the first first-in-first-out circuit corresponding to the second timing reconstruction unit provided in the embodiments of this application;

[0027] Figure 12 This is one of the structural schematic diagrams of the second processing module provided in the embodiments of this application;

[0028] Figure 13 This is a second schematic diagram of the structure of the second processing module provided in the embodiments of this application;

[0029] Figure 14 This is one of the schematic diagrams of the second first-in-first-out circuit corresponding to the fourth timing reconstruction unit provided in the embodiments of this application;

[0030] Figure 15 This is a second schematic diagram of the structure of the second first-in-first-out circuit corresponding to the fourth timing reconstruction unit provided in the embodiments of this application;

[0031] Figure 16 This is the third schematic diagram of the structure of the second processing module provided in the embodiments of this application;

[0032] Figure 17 This is a schematic diagram of the data upsampling process provided in an embodiment of this application;

[0033] Figure 18 This is a schematic diagram of the second processing unit provided in an embodiment of this application;

[0034] Figure 19 This is a second schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application;

[0035] Figure 20 This is the third schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application;

[0036] Figure 21 This is the fourth schematic diagram of the image processing apparatus provided in the embodiments of this application;

[0037] Figure 22 This is the fifth schematic diagram of the image processing apparatus provided in the embodiments of this application;

[0038] Figure 23 This is a schematic flowchart of the image processing method provided in the embodiments of this application;

[0039] Figure 24 This is the sixth schematic diagram of the image processing apparatus provided in the embodiments of this application;

[0040] Figure 25 This is one of the structural schematic diagrams of the electronic device provided in the embodiments of this application;

[0041] Figure 26 This is a second schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0042] Figure 27 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0044] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0045] The terms "at least one," "at least one of," etc., used in the specification and claims of this application refer to any one, any two, or a combination of two or more of the included items. For example, at least one of a, b, and c can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more items, and its meaning is similar to that of "at least one."

[0046] To better understand the embodiments of this application, the prior art will first be described in detail:

[0047] In related technologies, image pyramids typically include Gaussian pyramids and Laplace pyramids.

[0048] I. The construction process of the Gauss Pyramid is as follows:

[0049] First, the original image is used as the input data for the first level of the Gaussian pyramid, i.e., the first image G0. Then, the first image G0 is convolved with a pyramid filter kernel (such as a Gaussian filter kernel) to obtain the convolution result. This convolution result is then downsampled by a factor of h in both the row and column directions to obtain the first image G1, where h is an integer greater than 1. The height of the first image G1 is 1 / h of the height of the convolution result, and the width of the first image G1 is also 1 / h of the width of the convolution result. Next, the first image G1 is used as the input data for the second level of the Gaussian pyramid, and the above downsampling process is repeated based on the first image G1 to obtain the first image G2, and so on, until the first image G is obtained. N-1 N is an integer greater than 1. Finally, based on the input data of each level, i.e. based on the first image G0, the first image G1, the first image G2, ..., and the first image G... N-1 Construct the Gaussian pyramid corresponding to the original image above.

[0050] In this context, the pyramid filter kernel is a downsampling filter kernel, and the Gaussian filter kernel has a size of a × b, where a and b are both integers greater than or equal to 1. Optionally, a and b can be the same or different; no specific limitation is made here.

[0051] In other words, the Gaussian pyramid constructed above has N levels. Optionally, the lowest level in the Gaussian pyramid is the 1st level, and the highest level is the Nth level; or, the highest level in the Gaussian pyramid is the 1st level, and the lowest level is the Nth level, and the lowest level is the Nth level.

[0052] It should be noted that as the number of levels in the Gaussian pyramid increases (i.e., the higher the level), the high-frequency detail information of the first image at the corresponding level is gradually lost, resulting in a lower spatial resolution of the first image at that level. In other words, the spatial resolution of the first image G0 > the spatial resolution of the first image G1 > ... > the spatial resolution of the first image G N-1 Given the spatial resolution, the first image G0 has the clearest image quality. N-1 The image quality is the blurriest.

[0053] II. The construction process of the Pyramid of Laplace is as follows:

[0054] To compare the information differences contained in the first image of each level in the Gaussian pyramid, the output data of each level can be upsampled and magnified by h times to obtain the magnified result of each level. Then, the magnified result of each level is convolved with a Gaussian filter kernel to obtain the second image of each level. In this way, N second images can be obtained, namely second image E0, second image E1, ..., and the first image E... N-1 In other words, the second image at each level is obtained by downsampling and then upsampling the first image at the corresponding level. The second image at each level has the same size as the first image at the corresponding level.

[0055] Since the above downsampling process can be considered as a low-pass filter (such as a Gaussian filter), the high-frequency detail information of the second image corresponding to each level is less than that of the first image of the corresponding level. For the j-th level in the N levels of the Gaussian pyramid, j∈{1,2,……,N}, the second image E of the j-th level is... j-1 With the first image G of the j-th level j-1 By performing the difference operation, the second image E is obtained. j-1 With the first image G j-1 The information differences between them can be addressed using L. j-1 =G j-1 -E j-1 This indicates that, among them, information difference L j-1 This also represents the high-frequency detail information present in the Gaussian pyramid at level j. Finally, based on the information differences at each level, namely information difference L0, information difference L1, information difference L2, ..., and information difference L... N-1 Then construct the Laplacian pyramid corresponding to the original image above.

[0056] It should be noted that the number of levels in the constructed Laplace pyramid is also N, where the lowest level in the Gaussian pyramid is level 1 and the highest level is level N.

[0057] In summary, the construction processes of both the Gaussian pyramid and the Laplacian pyramid involve continuous downsampling of the image and multiple identical convolution operations. In particular, the construction process of the Laplacian pyramid also involves continuous upsampling of the image.

[0058] It should be noted that the maximum value of the number of levels N in both the Gaussian pyramid and the Laplacian pyramid is determined by the spatial resolution and sampling rate h of the original image. For example, Figure 1 A schematic diagram of an N-layer Gaussian pyramid and an N-layer Laplace pyramid is shown. Figure 1 In the diagram, the N-layer Gaussian pyramid is constructed based on h-fold downsampling, and the N-layer Laplace pyramid is constructed based on both h-fold downsampling and h-fold upsampling. Figure 1 As can be seen from this, assuming the spatial resolution of the original image is 1024*1024 and the sampling rate h is 2, then the formula 1024 = h can be used. N =2 N The calculation yields N=10, meaning that 10 first images can be obtained to construct a 10-layer Gaussian pyramid of images. In addition, 10 information differences can be obtained to construct a 10-layer Laplacian pyramid of images.

[0059] It should be noted that both the N-layer Gaussian pyramid and the N-layer Laplacian pyramid can be constructed using image processing devices. The following examples illustrate the image processing devices for a 4-layer Gaussian pyramid and a 4-layer Laplacian pyramid, respectively:

[0060] For example, for a Gaussian filter kernel of size a×b, containing a Gaussian pyramid of 4 levels, the following can be used: Figure 2a The image processing apparatus shown includes an instantiation module. Figure 2a In this design, the image processing device includes four levels of first instantiation modules. Each level's first instantiation module is obtained by encapsulating the corresponding level's first windowing module and first computational kernel module (i.e., downsampling module). Based on this, during the construction of the Gaussian pyramid, the encapsulated first instantiation modules can be called at the corresponding level to achieve data processing at that level, ultimately resulting in the construction of the Gaussian pyramid.

[0061] The first windowing module of each level is used to perform windowing operations on the input image of the corresponding level; the first calculation kernel module of each level is used to downsample the windowed input image of the corresponding level.

[0062] For example, for a Gaussian filter kernel of size a×b, with a Laplace pyramid containing 4 levels, the following can be used: Figure 2b The image processing apparatus shown includes an instantiation module. Figure 2b In the middle, the Pyramid of Laplace includes Figure 2a The diagram shows four levels of first-level instantiation modules and four levels of second-level instantiation modules. Each level's second-level instantiation module is obtained by encapsulating the corresponding level's second windowing module and second computational kernel module (i.e., upsampling module). Based on this, during the construction of the Laplace pyramid, the encapsulated first and second-level instantiation modules can be called at the corresponding level to achieve data processing at that level, ultimately resulting in the construction of the Laplace pyramid.

[0063] The second windowing module of each level is used to perform windowing operations on the input image of the corresponding level; the second calculation kernel module of each level is used to upsample the windowed input image of the corresponding level.

[0064] Combination Figure 2a and Figure 2b It is evident that since each of the four first instantiation modules contains a first computational core module with the same operation, and each of the four second instantiation modules contains a second computational core module with the same operation, the data calculation process during the construction of the corresponding image pyramid is quite complex, resulting in low image processing efficiency. Furthermore, a large number of computational core modules occupies space in the image processing device and wastes power, which is detrimental to chip hardware implementation.

[0065] To address the aforementioned technical problems, embodiments of this application provide an image processing apparatus, method, electronic device, and storage medium. This image processing apparatus can be applied to scenarios such as image processing, computer vision, and medical imaging.

[0066] In this embodiment, the image processing device includes a first processing module and a second processing module. The first processing module includes a first windowing unit and a first processing unit connected in sequence. The first windowing unit acquires first input data for each level of an N-layer image pyramid based on the original image and transmits the first input data for each level to the first processing unit. The first input data for each level includes at least one first image block, where N is an integer greater than 1. After acquiring at least one first image block included in each first input data, the first processing unit processes the first image block corresponding to each first valid signal sequentially according to the first processing timing of the first valid signal corresponding to each first image block to obtain the second input data for each level. Thus, multiple first computational core modules with identical operations are not required, nor are multiple second computational core modules with identical operations. Only the first processing unit is needed to process all the first input data without affecting the output timing, thereby quickly determining the second input data for each level. This simplifies the data calculation process in the first processing unit, improves data processing efficiency, and then transmits the second input data for each level to the second processing module. Furthermore, the second processing module constructs an image pyramid corresponding to the original image based on the second input data for each level, improving the overall image processing efficiency.

[0067] For example, suppose a user needs to identify a moving vehicle in an original image. After acquiring the original image containing the moving vehicle, the image processing device, through a first windowing unit, obtains the first input data for each level of an N-layer image pyramid based on the original image, and transmits the first input data for each level to the first processing unit. The first input data for each level includes at least one first image block. After obtaining the at least one first image block included in each first input data, the first processing unit processes the first image block corresponding to each first valid signal sequentially according to the first processing timing of the first valid signal corresponding to each first image block to obtain the second input data for each level. In this way, multiple first computing core modules with the same operation are not required, nor are multiple second computing core modules with the same operation. Only the first processing unit is needed to process all the first input data without affecting the output timing, so as to quickly determine the second input data for each level. This simplifies the data calculation process in the first processing unit, improves data processing efficiency, and then transmits the second input data for each level to the second processing module. Furthermore, the second processing module constructs an image pyramid corresponding to the original image based on the second input data of each level, and finally identifies the moving vehicles in the image pyramid. The whole process improves the efficiency of image processing.

[0068] Furthermore, the image processing device provided in this application embodiment does not require setting up multiple first computing core modules with the same operation, nor does it require setting up multiple second computing core modules with the same operation. Only the first processing unit is needed to quickly determine the second input data of each level. This reduces the area occupied by the image processing device and also saves the power consumption of the image processing device to a certain extent, which is beneficial to chip hardware implementation.

[0069] It should be noted that the larger the value of the number of layers N, the more obvious the benefit in terms of area occupied; the larger the value of the sampling rate h, the more obvious the effect of power saving.

[0070] The image processing apparatus provided in the embodiments of this application is described in detail below:

[0071] Figure 3 A schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application is shown. Figure 3 As shown, the image processing apparatus provided in this application embodiment may include: a first processing module 10 and a second processing module 20 connected to the first processing module 10. The first processing module 10 may include a first windowing unit 101 and a first processing unit 102 connected in sequence.

[0072] The first windowing unit 101 is used to acquire the first input data of each level in the N-layer image pyramid based on the original image, and transmit the first input data of each level to the first processing unit 102. The first input data of each level includes at least one first image block, and N is an integer greater than 1.

[0073] The first processing unit 102 is configured to process the first image block corresponding to each first valid signal sequentially according to the first processing timing of the first valid signal corresponding to each first image block, to obtain the second input data of each level, and to transmit the second input data of each level to the second processing module 20.

[0074] The second processing module 20 is used to construct an image pyramid corresponding to the original image based on the second input data of each level.

[0075] In the embodiments of this application, the original image refers to image data that has not been processed, modified, or enhanced in any way.

[0076] Optionally, the original image can be acquired by an image processing device, or it can be acquired by an image acquisition device (such as a camera, scanner, and sensor) and then transmitted to the image processing device; no specific limitation is made here. This effectively preserves all the original information of the image captured by the image processing device or the image acquisition device at a specific moment.

[0077] In this embodiment, the maximum value of the number of levels in the N-layer image pyramid, i.e., the maximum value of N, is determined by the spatial resolution and sampling rate h of the original image. It should be noted that... Figure 1 The maximum value of N has already been illustrated by example, and will not be elaborated on here.

[0078] In this embodiment, the size of the first input data for each level is different, and as the number of levels in the N-level image pyramid increases, the size of the first input data for the corresponding level gradually decreases. That is, the number of at least one first image patch included in the first input data for each level is different, and as the number of levels in the N-level image pyramid increases, the number of at least one first image patch included in the first input data for the corresponding level gradually decreases.

[0079] In this embodiment of the application, each image block refers to a rectangular image region obtained by downsampling the first input data of the current level and then cropping it.

[0080] In the embodiments of this application, the first valid signal corresponding to each first image block refers to the output result of the first input data of the corresponding level after downsampling.

[0081] In this embodiment of the application, the first processing sequence of each first valid signal refers to the order in which the first image blocks corresponding to each first valid signal are sorted and processed sequentially according to a certain priority.

[0082] In this embodiment, the size of the second input data for each level is different, and as the number of levels in the N-level image pyramid increases, the size of the second input data for the corresponding level gradually decreases. It should be noted that the size of the second input data for each level is the same as the size of the first input data for the corresponding level.

[0083] In this embodiment, the first windowing unit 101 acquires the first input data of each level in the N-layer image pyramid based on the original image, and transmits the first input data of each level to the first processing unit 102. The first input data of each level includes at least one first image block, and N is an integer greater than 1. After acquiring at least one first image block included in each first input data, the first processing unit 102 can process the first image block corresponding to each first valid signal in sequence according to the first processing timing of the first valid signal corresponding to each first image block to obtain the second input data of each level. In this way, there is no need for multiple first computing core modules with the same operation, nor is there a need for multiple second computing core modules with the same operation. Only the first processing unit 102 is needed to process all the first input data to quickly determine the second input data of each level, which simplifies the data calculation process in the first processing unit 102 and improves the data processing efficiency. Then, the second input data of each level is transmitted to the second processing module 20. The second processing module 20 constructs the image pyramid corresponding to the original image based on the second input data of each level. The whole process improves the image processing efficiency.

[0084] In some embodiments of this application, Figure 4 A schematic diagram of the structure of a first window opening unit provided in an embodiment of this application is shown. Figure 4 In the image pyramid, the first windowing unit 101 may include a first sub-windowing unit 1011 and N-1 second sub-windowing units 1012. The first sub-windowing unit 1011 corresponds to the first level in the N-layer image pyramid, and the N-1 second sub-windowing units correspond one-to-one with the other N-1 levels in the N-layer image pyramid, excluding the first level.

[0085] The first sub-window unit 1011 is used to take the original image as the first input data of the first level; downsample the first input data of the first level to obtain at least one first image block of the first level; and output the output data of the first level based on the at least one first image block of the first level.

[0086] The second sub-windowing unit 1012 is used to take the output data of the (i-1)th level as the first input data of the i-th level, i∈{2,3,……,N}; to downsample the first input data of the i-th level to obtain at least one first image block of the i-th level; and to output the output data of the i-th level based on the at least one first image block of the i-th level.

[0087] In this embodiment of the application, after acquiring the original image, the first sub-windowing unit 1011 can first use the original image as the first input data of the first level; then, the first sub-windowing unit 1011 windows the first input data of the first level and downsamples the first input data of the first level to reduce the spatial resolution or size of the first input data of the first level, while retaining the main features of the first input data of the first level, thereby obtaining at least one first image block containing the key information of the first input data of the first level, and outputting at least one first image block of the first level as the output data of the first level. For each second sub-windowing unit 1012, after acquiring the output data of the previous level, the second sub-windowing unit 1012 can first use the output data of the previous level as the first input data of the current level; then, the second sub-windowing unit 1012 windows the first input data of the current level and downsamples the first input data of the current level to reduce the spatial resolution or size of the first input data of the current level, while retaining the main features of the first input data of the current level, thereby obtaining at least one first image block containing the key information of the first input data of the current level, and outputting at least one first image block of the current level as the output data of the current level. Based on this, the first windowing unit 101 can obtain the first input data of the first level output by the first sub-windowing unit 1011, and the first input data of the other N-1 levels besides the first level output by the N-1 second sub-windowing units, that is, the first windowing unit 101 can finally obtain N first input data, so that the subsequent first processing unit 102 can process these N first input data accordingly.

[0088] It should be noted that the above windowing operation can achieve window sparseness, prevent unnecessary pixels from being sent to the first processing unit 102, and greatly reduce the power consumption of downsampling calculation.

[0089] In some embodiments of this application, the first sub-windowing unit 1011 is specifically used to perform a convolution operation between the first input data of the first level and a pyramid filter kernel of size a×b to obtain a×b first image blocks corresponding to the first input data of the first level; and to perform h-fold downsampling on the a×b first image blocks to obtain c×d first image blocks of the first level.

[0090] Where a, b, c, and d are all integers greater than or equal to 1, h is an integer greater than 1, c is calculated based on a and h, and d is calculated based on b and h.

[0091] Optionally, based on the aforementioned a×b first image blocks, one row of the first image blocks in row a is input to the first processing unit 102 in real time, and the remaining a-1 rows of the first image blocks can be read from memory (such as Static Random-Access Memory, SRAM), meaning that the memory stores a-1 rows of the first image blocks. Similarly, in the aforementioned c×d first image blocks, one row of the first image blocks in row c is input to the first processing unit 102 in real time, and the remaining c-1 rows of the first image blocks can be read from the aforementioned SRAM, meaning that the memory stores c-1 rows of the first image blocks, and so on. Each level stores (corresponding row number - 1) first image blocks in memory so that the first processing unit 102 can quickly extract and calculate them.

[0092] The calculation process for c and d is explained below:

[0093] Optionally, the image processing device can calculate c using the first formula c = a / h; the image processing device can calculate d using the second formula d = b / h. It should be noted that if the result of a / h is not an integer, the result of a / h can be rounded up or down to obtain c. Similarly, if the result of b / h is not an integer, the result of b / h can be rounded up or down to obtain d.

[0094] In this embodiment, after acquiring the first input data of the first level, the first sub-windowing unit 1011 can perform a convolution operation on the first input data of the first level and a pyramid filter kernel of size a×b to obtain the convolution result. The convolution result is the a×b first image blocks corresponding to the first input data of the first level, and the pyramid filter kernel is a downsampling filter kernel. Then, the first sub-windowing unit 1011 can use any of the following implementation methods to downsample the a×b first image blocks by a factor of h to obtain c×d first image blocks containing the key information of the first input data of the first level.

[0095] Implementation method 1: The first sub-window unit 1011 performs h-fold downsampling on a×b first image blocks by row to obtain c×b first image blocks; then performs h-fold downsampling on the c×b first image blocks by column to obtain c×d first image blocks of the first level.

[0096] Implementation Method 2: The first sub-window unit 1011 downsamples a×b first image blocks by column by h times to obtain a×d first image blocks; then it downsamples the a×d first image blocks by row by h times to obtain c×d first image blocks of the first level.

[0097] In other words, regardless of whether it is implementation method 1 or implementation method 2, the c×d first image blocks of the first level can be determined.

[0098] The following section uses implementation method 2 as an example to elaborate on the determination of the first level output data by the first sub-window unit 1011: For a pyramid filter kernel of size k×k, which is a downsampling filter kernel with a sampling rate h of 2, such as Figure 5 The diagram shown is a schematic representation of the windowing of the first input data at the first level, as provided in an embodiment of this application. From... Figure 5As can be seen, the first sub-windowing unit 1011 windows the first input data of the first level to obtain k×k first image blocks, and performs a 2x downsampling (dsx2) on these k×k first image blocks to obtain (k×k) / 4 first image blocks of the first level. Specifically, the first sub-windowing unit 1011 takes the first image block data00 in the k×k first image blocks as the starting center, slides from left to right on the first row of first image blocks line0, and performs a 2x downsampling on the first row of first image blocks line0 column by column, retaining the pixel centers of the first image blocks data00, data02, data04, etc., to obtain k / 2 first image blocks in the first row of first image blocks line0. Since the first sub-windowing unit 1011 performs a 2x downsampling on k×k first image blocks, the pixel centers of the second row of first image blocks line1, the fourth row of first image blocks line3, the sixth row of first image blocks line5, etc., will be lost, and these pixel centers do not need to be processed again. Then, the first sub-windowing unit 1011 takes the first image block data20 in the k×k first image blocks as the starting center, and slides from left to right in the third row of first image blocks line2. It performs a 2x downsampling on the columns of the third row of first image blocks line2, retaining the pixel centers of the columns of first image blocks data20, data22, data24, ..., to obtain k / 2 first image blocks in the third row of first image blocks line2. This process is repeated to determine the output data of the first level. The output data of the first level is a (k×k) / 4 window with data*0, data*2, data*4, ... as the starting centers in line0, line2, line4, ... respectively. The valid signals corresponding to this (k×k) / 4 window are the valid data signals at the corresponding bits of line0, line2, line4, ..., that is, the first valid signals corresponding to the first image blocks in the (k×k) / 4 window.

[0099] In other words, the first valid signal corresponding to at least one first image block in the current level is the result output after downsampling the first input data of the previous level.

[0100] It should be noted that the number of first image blocks included in the output data of the first level is equivalent to 1 / 4 of the number of first image blocks included in the original image, which greatly reduces the amount of data entering the subsequent first processing unit 102 and improves the data processing efficiency of the subsequent first processing unit 102.

[0101] In this embodiment of the application, the process of the second sub-windowing unit 1012 windowing the first input data of the current level is as follows: the second sub-windowing unit 1012 performs a convolution operation on the first input data of the current level and the pyramid filter kernel to obtain the convolution result, and downsamples the convolution result in the row and column directions respectively to obtain at least one first image block of the current level, and outputs the at least one first image block of the current level as the output data of the current level.

[0102] It should be noted that the process of the second sub-window unit 1012 opening a window for the first input data of the current level is similar to the process of the first sub-window unit 1011 opening a window for the first input data of the first level, and will not be described in detail here.

[0103] It should be noted that the aforementioned first processing unit 102 may include a first signal reconstruction unit 1021 and a first shared computing unit 1022 connected in sequence; or, the aforementioned first processing unit 102 may include N first timing reconstruction units 1023 and second shared computing units 1024, all of which can obtain the second input data of each level. The following is a detailed description of these two types of first processing units 102:

[0104] The first type of first processing unit 102 is described in detail below:

[0105] In some embodiments of this application, such as Figure 6 The diagram shown is a schematic representation of the structure of the first processing unit provided in an embodiment of this application. Figure 6 In the process, the first processing unit 102 may include a first signal reconstruction unit 1021 and a first shared computing unit 1022 connected in sequence.

[0106] The first signal reconstruction unit 1021 is used to generate a first target valid signal (valid_new) based on the hierarchical priority of each level, and transmit the first target valid signal to the first shared computing unit 1022. The first target valid signal includes a first valid signal corresponding to each first image block and a first processing timing corresponding to each first valid signal.

[0107] The first shared computing unit 1022 is used to process the first image block corresponding to each first valid signal in sequence according to the first processing timing to obtain the second input data of each level, and transmit the second input data of each level to the second processing module 20.

[0108] In this embodiment, the priority of each level is pre-defined, with lower-level levels having higher priority. Specifically, the priority of level 1 > level 2 > ... > level N. This allows for fixed-priority arbitration, allocating computational resources to the first valid signal at each level based on its priority, ensuring that higher levels do not require computational work from lower levels.

[0109] In this embodiment, after acquiring the first valid signal corresponding to each first image block, the first signal reconstruction unit 1021 can reconstruct the first valid signal corresponding to each first image block based on the hierarchical priority of each level to obtain the first target valid signal corresponding to the first valid signal of each first image block. At this time, the first target valid signal also includes the first processing timing corresponding to each first valid signal. Then, the first signal reconstruction unit 1021 transmits the first target valid signal to the first shared computing unit 1022. After acquiring the first target valid signal, the first shared computing unit 1022 needs to process the first image blocks of each level, which may lead to data contention. To avoid data contention, a fixed priority arbitration is introduced, i.e., a hierarchical priority is introduced. In this way, the first shared computing unit 1022 can process the first image blocks corresponding to each first valid signal of the first target valid signal sequentially based on the first processing timing included in the first target valid signal to obtain the second input data of each level.

[0110] It should be noted that, as the number of levels in the N-level image pyramid increases, the number of at least one first image blocks included in the first input data of the corresponding level gradually decreases. Therefore, the processing time of the first shared computing unit 1022 for at least one first image block included in the first input data of the corresponding level also gradually shortens. That is, the processing time of the first shared computing unit 1022 for at least one first image block included in the first input data of the current level is shorter than the processing time for at least one first image block included in the first input data of the previous level, thereby improving data processing efficiency.

[0111] Optionally, the first signal reconstruction unit 1021 is specifically used to generate a target valid signal for each level based on the level priority of each level; and to generate a first target valid signal based on the target valid signal for each level.

[0112] In this embodiment, the first input data of each level can satisfy the sparsity characteristic of the corresponding level. The sparsity characteristic, taking 2x downsampling as an example, means that if the first valid signal of at least one first image block included in the first row of the first input data of the first level (i.e., the output data level0 of the first level) is continuous, then the first valid signal of the pixels in the row of the output data level1 of the second level will be spaced by 2. Based on this, the first valid signal of the pixels in the row of the output data level3 of the third level will be spaced by 4, the first valid signal of the pixels in the row of the output data level4 of the fourth level will be spaced by 8, and so on, until the first valid signal of the pixels in the row of the output data levelN of the Nth level will be spaced by 2. N This ensures that the invalid time of the first valid signal in the first level can be allocated to other higher levels to complete the calculation work, and that the target valid signal in each level will not conflict with the original first valid signal in the corresponding level.

[0113] For example, taking a Gaussian pyramid with three levels and a 2x downsampling (dsx2) as an example, the process of the first signal reconstruction unit 1021 generating the target effective signal for each level based on the level priority of each level, and generating the first target effective signal based on the target effective signal for each level, is explained in detail below: Since the pyramid filter kernel windowing causes row delays between each level in the N-level image pyramid, extracting the interval where the output data level0 of the first level, the output data level1 of the second level, and the output data level2 of the third level are simultaneously effective can more intuitively show the comparison between the time occupied by the target effective signal at each level (referred to as: time taken) and the time occupied by the first effective signal in the original image, such as... Figure 7 As shown. The basic principle is as follows: Since the time occupied by the target valid signal of each level is 1 / h of the first valid signal of the first input data of the corresponding level, the total time occupied by at least one first image block included in the first input data of each level after h-fold downsampling is (1 / d + 1 / d) 2 +…+1 / d n Since the value is always less than 1, the time occupied by the target valid signal at each level is always less than the time occupied by the first valid signal of the first input data at the corresponding level. For the target valid signal generated at the same level, it will always be pulled up at the current level, that is, each level will be pulled up before the target valid signal of the next level arrives, and will not contribute to the overall calculation delay.

[0114] If multiple levels of target valid signals are simultaneously high, the lowest level target valid signal is used first, while the higher level target valid signals are maintained until the lower level target valid signals are low before being read. For example... Figure 8As shown, if level0 to (N-1) valid are simultaneously raised, valid_new is first assigned a value of 1, and level1 to (n-1) valid are maintained. When level0 valid is 0, level1 valid is read, and valid_new is assigned a value of 2, and so on, assigning values ​​to valid_new according to the one-hot encoding method. Combined with... Figure 7 and Figure 8 It can be seen that during the process of generating the image pyramid by 2x downsampling, when level0 to 2valid are simultaneously raised, the reconstructed valid_new will be assigned the values ​​1, 2, 1, 4, 1, 2, 1, 0, 1, 2, 1, 4, 1, 2, 1, 0 in sequence starting from the current clock cycle.

[0115] Optionally, the image processing device can construct the image pyramid from the lowest level, or it can start from the highest level. In this case, the original image can be upsampled by h times to construct the pyramid image. It should be noted that the layer priority involved in constructing the image pyramid from the highest level is the opposite of that involved in constructing the image pyramid from the lowest level. That is, the spatial resolution of the first image G0 < the spatial resolution of the first image G1 < ... < the spatial resolution of the first image G... N-1 The spatial resolution is designed to avoid interfering with the overall data processing timing.

[0116] The second type of first processing unit 102 is described in detail below:

[0117] In some embodiments of this application, such as Figure 9 The diagram shown is a schematic representation of the structure of the first processing unit provided in an embodiment of this application. Figure 9 In this process, the first processing unit 102 may include N first temporal reconstruction units 1023 and a second shared computing unit 1024. The N first temporal reconstruction units 1023 correspond one-to-one with the N levels of the aforementioned N-layer image pyramid. Specifically, the first first temporal reconstruction unit 1023 is connected to the first sub-windowing unit 1011, and the other N-1 first temporal reconstruction units 1023 are respectively connected to N-1 second sub-windowing units.

[0118] The second temporal reconstruction unit is configured to generate a first read signal when a first condition is met, and based on the first read signal, output a first valid signal corresponding to each first image block in the first input data of the corresponding level. The second temporal reconstruction unit is any one of the N first temporal reconstruction units 1023 except for the first temporal reconstruction unit corresponding to the first level; the first condition includes that the third temporal reconstruction unit has output the first valid signal of the corresponding level; the third temporal reconstruction unit includes the first temporal reconstruction unit among the N first temporal reconstruction units 1023 with a higher priority than the second temporal reconstruction unit.

[0119] The second shared computing unit 1024 is used to take the input order of each first valid signal as the first processing timing of each first valid signal, and process the first image block corresponding to each first valid signal in sequence according to the first processing timing to obtain the second input data of each level, and transmit the second input data of each level to the second processing module 20.

[0120] In this embodiment of the application, the first read signal is a pulse signal.

[0121] In this embodiment, each first timing reconstruction unit 1023 has its own priority. In this case, the N first timing reconstruction units 1023 have priority arbitration, meaning that these N first timing reconstruction units 1023 constitute a priority-based timing shaping circuit, maximizing the utilization efficiency of the second shared computing unit 1024. Specifically, when a first timing reconstruction unit with a higher priority than the second timing reconstruction unit among the N first timing reconstruction units 1023 has already output the first valid signal of its corresponding level, the second timing reconstruction unit can output the first valid signal corresponding to each first image block in the first input data of the corresponding level. Since the first shared computing unit 1022 needs to process the first image block of each level, there will be data competition. In order to avoid data competition, the priority of each first temporal reconstruction unit 1023 is introduced. In this way, the second shared computing unit 1024 can use the input order of each first valid signal as the first processing time order of each first valid signal based on the priority of each first temporal reconstruction unit 1023, and then process the first image block corresponding to each first valid signal in sequence according to the first processing time order to obtain the second input data of each level.

[0122] It should be noted that the higher the priority of the first timing reconstruction unit 1023, the earlier the input order of the first valid signal of the corresponding level.

[0123] In some embodiments of this application, the N first timing reconstruction units 1023 can be N first first-in-first-out (FIFO) circuits; each first FIFO circuit can include a first FIFO queue and a first judgment unit. Figure 10 The diagram shown is a schematic representation of the structure of the first FIFO circuit corresponding to the second timing reconstruction unit provided in an embodiment of this application. Figure 10 In the second timing reconstruction unit, the first first-in-first-out circuit may include a first first-in-first-out queue and a first judgment unit.

[0124] The second timing reconstruction unit includes a first judgment unit, which is used to determine whether the third timing reconstruction unit has output the first valid signal of the corresponding level, and transmit the first judgment result to the first first-in-first-out queue included in the second timing reconstruction unit.

[0125] The second time-series reconstruction unit includes a first first-in-first-out queue, which is used to generate a first read signal when the first judgment result indicates that the first condition is met, and output a first valid signal corresponding to each first image block in the first input data of the corresponding level based on the first read signal.

[0126] In this embodiment, for each second time-series reconstruction unit, which includes a first FIFO queue and a first judgment unit, the first judgment unit first determines whether the third time-series reconstruction unit has output the first valid signal of the corresponding level and generates a first judgment result. Then, it transmits the first judgment result to the first FIFO queue. After obtaining the first judgment result, the first FIFO queue can analyze it. If the first judgment result indicates that the first time-series reconstruction unit with a higher priority than the second time-series reconstruction unit among the N first time-series reconstruction units 1023 has output the first valid signal of the corresponding level, it indicates that the first FIFO queue corresponding to the first time-series reconstruction unit with a higher priority than the second time-series reconstruction unit is empty. At this time, a first read signal can be generated, and based on the first read signal, the first valid signal corresponding to each first image block in the first input data of the corresponding level is output. The whole process can effectively avoid data competition when the subsequent second shared computing unit 1024 processes each image block.

[0127] The first first-in-first-out circuit corresponding to the second timing reconstruction unit is described in detail below:

[0128] For example, such as Figure 11 The diagram shown is a schematic representation of the structure of the first FIFO circuit corresponding to the second timing reconstruction unit provided in an embodiment of this application. Figure 11In this circuit, the first FIFO circuit may include a first FIFO queue and a first judgment unit. The second sub-windowing unit 1012 is connected to the first FIFO circuit. The second sub-windowing unit 1012 transmits the output data of the corresponding level (i.e., the first valid signals of c×d first image blocks) to the first FIFO queue through the gas_wd interface. If the first FIFO queue is empty, the data is transmitted to the first judgment unit. The first judgment unit performs a read operation to determine whether the third timing reconstruction unit has finished outputting the first valid signals of the corresponding level, and transmits the first judgment result to the first FIFO queue through the gas_rd interface. When the first judgment result indicates that the first condition is met, the first input-output queue of the first time-series reconstruction unit with a priority higher than the second time-series reconstruction unit among the N first time-series reconstruction units 1023 is empty. At the same time, the first first-in-first-out queue included in the first first-in-first-out circuit is also empty. At this time, a first read signal is generated, and based on the first read signal, the first valid signal corresponding to each first image block in the output data of the corresponding level is output through the gas_rdata interface, so as to be subsequently input to the second shared computing unit 1024 for calculation.

[0129] It should be noted that the data writing logic of the first FIFO queue is the same as the data caching logic of the SRAM mentioned above, and will not be elaborated here.

[0130] In some embodiments of this application, the second processing module 20 is specifically used to: take the second input data of each level as the image data of the image pyramid corresponding to the original image.

[0131] It can be understood that the image pyramid corresponding to the original image includes N layers of data, and each layer of data is the second input data of the corresponding level in the N-layer image pyramid. That is, the N layers of data in the image pyramid corresponding to the original image correspond one-to-one with the second input data of the N levels in the N-layer image pyramid.

[0132] In this embodiment of the application, after the second processing module 20 obtains the second input data of each level, it can directly determine the second input data of each level as the image data of the image pyramid corresponding to the original image, and the image pyramid is a Gaussian pyramid.

[0133] It should be noted that if the image processing device constructs the Laplacian pyramid of the original image, the following problems will occur: Because the first shared computing unit 1022 experiences data contention when dealing with the first valid signal of the first input data at each level, although the fixed-priority arbitration can allocate computing resources for each level, the timing of the second input data at each level cannot be guaranteed to have the sparsity characteristic of the corresponding level (i.e., the interval between the first valid signals of the output rows will become smaller when data contention occurs). Similarly, because the second shared computing unit 1024 experiences data contention when dealing with the first valid signal of the first input data at each level, although the priority-based timing shaping circuit can allocate computing resources for each level, the timing of the second input data at each level still cannot be guaranteed to have the sparsity characteristic of the corresponding level. In other words, during the construction of the Laplacian pyramid, the downsampled output is the upsampled input, and the upsampled data flows from higher to lower levels. As a result, the data processing priority of the second processing module 20 is higher for lower levels, making it easy for data to be overwritten during the upsampling process.

[0134] Therefore, to address the issue of data overwriting, the second processing module 20 may include a second windowing unit 201, a second signal reconstruction unit 202, a third shared computing unit 203, and a first construction unit 204 connected in sequence; or, the second processing module 20 may include a third windowing unit 206, a second processing unit 207, and a second construction unit 208. By performing timing optimization (retiming) on ​​the second input data at each level, data overwriting during the upsampling process can be avoided. Furthermore, the area required for hardware implementation of upsampling can be minimized. These two types of second processing modules 20 are described in detail below:

[0135] The detailed description of the first type of second processing module 20 is as follows:

[0136] In some embodiments of this application, such as Figure 12 The diagram shown is a structural schematic of the second processing module provided in an embodiment of this application. Figure 12 In the second processing module 20, there may be a second windowing unit 201, a second signal reconstruction unit 202, a third shared computing unit 203 and a first construction unit 204 connected in sequence.

[0137] The second windowing unit 201 is used to upsample the second input data of each level sequentially based on the sampling timing of the second input data of each level to obtain the third input data of each level, and transmit the third input data of each level to the second signal reconstruction unit 202. The third input data of each level includes at least one second image block.

[0138] The second signal reconstruction unit 202 is used to generate a second target valid signal based on the sampling timing of the third input data of each level, and transmit the second target valid signal to the third shared computing unit 203. The second target valid signal includes the second valid signal corresponding to each second image block in the third input data of each level, and the second processing timing corresponding to each second valid signal; the sampling timing of the third input data of each level is the same as the sampling timing of the second input data of each level.

[0139] The third shared computing unit 203 is used to process the second image block corresponding to each second valid signal in sequence according to the second processing timing to obtain the fourth input data of each level, and transmit the fourth input data of each level to the first construction unit 204.

[0140] The first construction unit 204 is used to construct an image pyramid corresponding to the original image based on the second input data of each level and the fourth input data of each level.

[0141] In this embodiment of the application, the first construction unit 204 can fuse the second input data of each level and the fourth input data of each level to obtain the fused data corresponding to each level, and use the fused data corresponding to each level as the image data of the image pyramid corresponding to the original image.

[0142] In this embodiment, during the second processing module 20's processing of the second input data at each level, timing optimization is first performed on the second input data at each level to obtain the sampling timing of the second input data at each level. Based on the sampling timing of the second input data at each level, the second windowing unit 20 sequentially upsamples the second input data at each level to obtain the third input data at each level. In this way, the third input data at each level also carries the sampling timing of the corresponding level. The second windowing unit 20 then transmits the third input data at each level to the second signal reconstruction unit 202. After obtaining the third input data at each level, the second signal reconstruction unit 202 can directly generate a second target valid signal based on the sampling timing of the third input data at each level. The second target valid signal also includes the second processing timing corresponding to each second valid signal. The higher the sampling timing of the third input data at each level, the earlier the second processing timing of the corresponding second valid signal is. Then, the second signal reconstruction unit 202 transmits the second target valid signal to the third shared computing unit 203. After acquiring the second target valid signal, the third shared computing unit 203 processes the second image block corresponding to each second valid signal sequentially according to the second processing timing included in the second target valid signal, obtaining the fourth input data for each level, and transmitting the fourth input data for each level to the first construction unit 204. Finally, the first construction unit 204 performs differential processing on the fourth input data of each level and the second input data of the corresponding level to obtain the information difference of each level, and then constructs the image pyramid corresponding to the original image based on the information difference of each level. This image pyramid is a Laplacian pyramid. The entire process can solve the timing problem caused by downsampling and then upsampling the first input data of each level, effectively avoiding the data competition problem when processing the second image block corresponding to the second valid signal of each level, that is, effectively avoiding the situation where data is easily overwritten at higher levels.

[0143] Optionally, the second windowing unit 201 may include N third sub-windowing units, and the N third sub-windowing units correspond one-to-one with the N levels of the N-layer image pyramid.

[0144] In some embodiments of this application, such as Figure 13 The diagram shown is a structural schematic of the second processing module provided in an embodiment of this application. Figure 13 In the second processing module 20, there may also be N fourth temporal reconstruction units 205; the N fourth temporal reconstruction units 205 correspond one-to-one with the N levels of the N-layer image pyramid.

[0145] The fourth temporal reconstruction unit 205 is used to generate a second read signal when a second condition is met, and output a second valid signal corresponding to each second image block in the second input data of the corresponding level based on the second read signal; the second condition includes that the number of second valid signals output by the fourth temporal reconstruction unit 205 at the corresponding level reaches a preset number for the corresponding level of the fourth temporal reconstruction unit 205.

[0146] In this embodiment of the application, the preset number of each fourth temporal reconstruction unit 205 corresponding to the level is the total number of image blocks of the first input data of the corresponding level.

[0147] In this embodiment, each fourth timing reconstruction unit 205 is a priority-based timing shaping circuit to maximize the utilization efficiency of the third shared computing unit 203. Specifically, when the number of second valid signals output by the fourth timing reconstruction unit 205 at the corresponding level reaches a preset number for the corresponding level, the fourth timing reconstruction unit 205 can generate a second read signal and, based on the second read signal, output the second valid signal corresponding to each second image block in the second input data of the corresponding level. Since the third shared computing unit 203 needs to process the second image blocks of each level, there may be data overwriting. To avoid data overwriting, a priority is introduced for each fourth timing reconstruction unit 205. In this way, the third shared computing unit 203 can use the input order of each second valid signal as the second processing order of each second valid signal based on the priority of each fourth timing reconstruction unit 205, and then process the second image blocks corresponding to each second valid signal sequentially according to the second processing order to obtain the fourth input data of each level.

[0148] It should be noted that the higher the priority of the fourth timing reconstruction unit 205, the earlier the input order of the second valid signal of the corresponding level.

[0149] In some embodiments of this application, the N fourth timing reconstruction units 205 are N second first-in-first-out (FIFO) circuits; each second FIFO circuit includes a second FIFO queue, a counter, and a second judgment unit. For example... Figure 14 The diagram shown is a schematic diagram of the second first-in-first-out circuit corresponding to the fourth timing reconstruction unit provided in an embodiment of this application. Figure 14 In the second FIFO circuit corresponding to the fourth timing reconstruction unit, it may include a second FIFO queue, a counter, and a second judgment unit.

[0150] The second first-in-first-out queue is used to generate a second read signal when the second judgment result indicates that the second condition is met, and based on the second read signal, output the second valid signal corresponding to each second image block in the second input data of the corresponding level;

[0151] The counter is used to count the number of second valid signals that have been output in the second first-in-first-out queue and to transmit the count to the second judgment unit.

[0152] The second judgment unit is used to determine whether the number of second valid signals output by the corresponding level counted by the counter has reached the preset number of the corresponding level of the fourth timing reconstruction unit, and to transmit the second judgment result to the second first-in-first-out queue.

[0153] In this embodiment, for each fourth time-series reconstruction unit 205, which includes a second first-in-first-out (FIFO) queue and a counter, the counter first counts the number of second valid signals output in the second FIFO queue and transmits this count to the second judgment unit. The second judgment unit compares this count with a preset number for the corresponding level of the fourth time-series reconstruction unit 205, determines whether the count has reached the preset number, generates a second judgment result, and transmits the second judgment result to the second FIFO queue. When the second FIFO queue determines that the second judgment result indicates that the second condition is met, it generates a second read signal and, based on the second read signal, outputs the second valid signal corresponding to each second image block in the second input data of the corresponding level. This entire process effectively avoids data contention when the subsequent third shared computing unit 203 processes each second image block.

[0154] The second first-in-first-out circuit corresponding to the fourth timing reconstruction unit is described in detail below:

[0155] For example, such as Figure 15 The diagram shown is a schematic diagram of the second first-in-first-out circuit corresponding to the fourth timing reconstruction unit provided in an embodiment of this application. Figure 15In this circuit, the second FIFO circuit may include a second FIFO queue, a counter, and a second judgment unit. The third sub-windowing unit of the corresponding layer of the fourth timing reconstruction unit 205 is connected to this second FIFO circuit. This third sub-windowing unit transmits the output data of the corresponding layer (i.e., the second valid signal of the second image block) to the second FIFO queue via the gas_wd interface, and then continues to transmit it to the counter. The counter counts the number of second valid signals already output in the second FIFO queue and transmits this count to the second judgment unit. The second judgment unit compares this count with a preset number for the corresponding layer of the fourth timing reconstruction unit 205, determines whether the count has reached the preset number, generates a second judgment result, and transmits the second judgment result to the second FIFO queue via the gas_rd interface. When the second first-in-first-out queue determines that the second judgment result indicates that the second condition is met, it generates a second read signal and, based on the second read signal, outputs the second valid signal corresponding to each second image block in the second input data of the corresponding level through the gas_rdata interface, so that it can be subsequently input to the third shared computing unit 203 for calculation.

[0156] The second type of second processing module 20 is described in detail below:

[0157] In some embodiments of this application, such as Figure 16 The diagram shown is a structural schematic of the second processing module provided in an embodiment of this application. Figure 16 In this process, the second processing module 20 may include a third window opening unit 206, a second processing unit 207, and a second building unit 208.

[0158] The third windowing unit 206 is used to upsample the second input data of each level to obtain the fifth input data of each level, and transmit the fifth input data of each level to the second processing unit 207. The fifth input data of each level includes at least one third image block.

[0159] The second processing unit 207 is used to process the third image block corresponding to each third valid signal in sequence according to the third processing timing of the third valid signal corresponding to each third image block, to obtain the sixth input data of each level, and to transmit the sixth input data of each level to the second construction unit 208.

[0160] The second building unit 208 is also used to build an image pyramid corresponding to the original image based on the second input data of each level and the sixth input data of each level.

[0161] In this embodiment of the application, the second construction unit 208 can fuse the second input data of each level and the sixth input data of each level to obtain the fused data corresponding to each level, and use the fused data corresponding to each level as the image data of the image pyramid corresponding to the original image.

[0162] In this embodiment, after obtaining the second input data of each level, the third windowing unit 206 can upsample the second input data of each level to obtain the fifth input data of each level, and transmit the fifth input data of each level to the second processing unit 207. Since the second processing unit 207 needs to process the third image block of each level, there may be data overwriting. To avoid data overwriting, the second processing unit 207 can process the third image block corresponding to each third valid signal in sequence according to the third processing timing of the third valid signal corresponding to each third image block to obtain the sixth input data of each level. The size of the sixth input data of each level is the same as the size of the second input data of the corresponding level. Then, the second processing unit 207 transmits the sixth input data of each level to the second construction unit 208. The second construction unit 208 performs differential processing on the sixth input data of each level and the second input data of the corresponding level to obtain the information difference of each level, and then constructs the image pyramid corresponding to the original image based on the information difference of each level. The image pyramid is a Laplacian pyramid. The entire process can solve the timing problem caused by downsampling and then upsampling the first input data of each level, and effectively avoid the data competition problem when processing the third image block corresponding to the third valid signal of each level, that is, effectively avoid the situation where data is easily overwritten at higher levels.

[0163] Optionally, the third windowing unit 206 may include N fourth sub-windowing units, and the N fourth sub-windowing units correspond one-to-one with the N levels of the N-layer image pyramid.

[0164] It should be noted that the data upsampling process involves increasing the number of image blocks included in the second input data to obtain the image blocks corresponding to the fifth input data. For a k×k pyramid filter kernel, which is an upsampling filter kernel, generating even rows of the fifth input data corresponding to the second input data requires caching [(k+1) / 2]-1 image rows in memory, and generating odd rows of the fifth input data requires caching (k+1) / 2 image rows in memory. Therefore, each level stores (k+1) / 2 image rows of cached data in memory. Specifically, if the result of [(k+1) / 2]-1 is not an integer, the result is rounded up to obtain the number of cached image rows; if the result of (k+1) / 2 is not an integer, the result is rounded down to obtain the number of cached image rows.

[0165] The following section will elaborate on this by taking the example of upsampling the second input data of the corresponding level for each fourth sub-window unit to obtain the fifth input data of the corresponding level:

[0166] For example, consider a 5×5 pyramid filter kernel, which is an upsampling filter kernel, such as... Figure 17 The diagram shown is a schematic representation of the data upsampling process provided in an embodiment of this application. Figure 17 As can be seen, three rows of image data need to be cached. During downsampling, the output data of level N serves as the second input data for level N-1 during upsampling. Pixels from the second input data are filled according to adjacent rows and columns. A k×k filter window is generated in the filled image, using the pixels of each row and column as the filter center, to construct the fifth input data. The number of filter windows is the same as the number of pixels in the first input image of level N-1.

[0167] In some embodiments of this application, such as Figure 18 The diagram shown is a schematic representation of the second processing unit provided in an embodiment of this application. Figure 18 In the middle, the second processing unit 207 may also include N fifth temporal reconstruction units 2071 and a fourth shared computing unit 2072; the N fifth temporal reconstruction units 2071 correspond one-to-one with the N levels of the N-layer image pyramid;

[0168] The sixth temporal reconstruction unit is used to generate a third read signal when the third condition is met, and based on the third read signal, output the third valid signal corresponding to each third image block in the fifth input data of the corresponding level; the sixth temporal reconstruction unit is any one of the N fifth temporal reconstruction units 2071; the third condition includes that the seventh temporal reconstruction unit has output the third valid signal of the corresponding level; the seventh temporal reconstruction unit includes the fifth temporal reconstruction unit with higher priority than the sixth temporal reconstruction unit among the N fifth temporal reconstruction units 2071;

[0169] The fourth shared computing unit 2072 is used to take the input order of each third valid signal as the third processing timing of each third valid signal, and process the third image block corresponding to each third valid signal in sequence according to the third processing timing to obtain the sixth input data of each level, and transmit the second input data of each level to the second construction unit 208.

[0170] It should be noted that, Figure 18 The implementation process of the second processing unit 207 shown is the same as described above. Figure 9The implementation process of the first processing unit 102 shown is similar, except that the second processing unit 207 is applied to the process of data upsampling, while the first processing unit 102 is applied to the process of data downsampling.

[0171] In this embodiment, each fifth timing reconstruction unit 2071 has its own priority. In this case, the N fifth timing reconstruction units 2071 have priority arbitration, meaning that these N fifth timing reconstruction units 2071 constitute a priority-based timing shaping circuit, maximizing the utilization efficiency of the fourth shared computing unit 2072. Specifically, when a fifth timing reconstruction unit with a higher priority than the sixth timing reconstruction unit has already output the third valid signal for its corresponding level, the sixth timing reconstruction unit can output the third valid signal corresponding to each third image block in the fifth input data for its corresponding level. Since the fourth shared computing unit 2072 needs to process the third image block of each level, data contention may occur. To avoid this, a priority is introduced for each fifth temporal reconstruction unit 2071. Based on this priority, the fourth shared computing unit 2072 can use the input order of each third valid signal as the third processing order for that signal. Then, according to this third processing order, the third image block corresponding to each third valid signal is processed sequentially to obtain the sixth input data for each level. Finally, the second construction unit 208 performs a difference operation between the sixth input data of each level and the second input data of the corresponding level to obtain the information difference for each level. Based on this information difference, an image pyramid corresponding to the original image is constructed; this image pyramid is a Laplacian pyramid. This entire process solves the timing problem caused by downsampling and then upsampling the first input data of each level, effectively avoiding data contention when processing the third image block corresponding to the third valid signal of each level, thus effectively preventing data overwriting at higher levels.

[0172] Optionally, the aforementioned fourth shared computing unit 2072 may include a third reconstruction module and a sub-shared computing unit. The third reconstruction module generates a third target valid signal based on the sampling timing of the fifth input data at each level, and transmits this third target valid signal to the sub-shared computing unit. The third target valid signal includes the third valid signal corresponding to each third image block in the fifth input data at each level, and the third processing timing corresponding to each third valid signal; the sampling timing of the fifth input data at each level is the same as the sampling timing of the second input data at each level. The sub-shared computing unit is used to take the input order of each third valid signal as the third processing timing of each third valid signal, and processes the third image block corresponding to each third valid signal sequentially according to the third processing timing to obtain the sixth input data at each level, and transmits the second input data at each level to the second construction unit 208. The entire scheme introduces the reconstructed third target valid signal, which effectively avoids data overwriting.

[0173] Optionally, the aforementioned priority-based timing shaping circuit can also be used to optimize multiple sets of circuits with strict timing relationships, enabling the priority-based timing shaping circuit to reuse more computational processing units with the same operation. These computational processing units are not limited to the first and second computational core modules mentioned in the embodiments of this application.

[0174] The following example, using the construction of an N-layer Gaussian pyramid, illustrates the image processing device:

[0175] Example 1: such as Figure 19 The diagram shown is a schematic representation of an image processing apparatus provided in an embodiment of this application. Figure 19 In this process, the image processing device may include a first processing module 10 and a second processing module 20.

[0176] The first processing module 10 may include a first window opening unit 101 and a first processing unit 102.

[0177] The first windowing unit 101 may include a first sub-windowing unit 1011 and N-1 second sub-windowing units 1012; the first processing unit 102 may include a first signal reconstruction unit 1021 and a first shared computing unit 1022.

[0178] In Example 1, the first sub-window unit 1011 uses the original image as the first input data of the first level; it downsamples the first input data of the first level to obtain at least one first image block of the first level, and outputs the output data of the first level based on the at least one first image block of the first level.

[0179] Each of the N-1 second sub-windowing units 1012 is used to take the output data of the (i-1)th level as the first input data of the i-th level, i∈{2,3,……,N}; to downsample the first input data of the i-th level to obtain at least one first image block of the i-th level; and to output the output data of the i-th level based on the at least one first image block of the i-th level.

[0180] At this time, the first window unit 101 can obtain the first input data of each level, and the number is N.

[0181] The first signal reconstruction unit 1021 generates a first target valid signal (valid_new) based on the hierarchical priority of each level, and transmits the first target valid signal to the first shared computing unit 1022. The first target valid signal includes a first valid signal corresponding to each first image block and a first processing timing corresponding to each first valid signal.

[0182] The first shared computing unit 1022 is used to process the first image block corresponding to each first valid signal in sequence according to the first processing timing to obtain the second input data of each level, and transmit the second input data of each level to the second processing module 20.

[0183] The second processing module 20 uses the second input data of each level as the image data of the Gaussian pyramid corresponding to the original image.

[0184] Example 2: such as Figure 20 The diagram shown is a schematic representation of an image processing apparatus provided in an embodiment of this application. Figure 20 In this process, the image processing device may include a first processing module 10 and a second processing module 20.

[0185] The first processing module 10 may include a first window opening unit 101 and a first processing unit 102.

[0186] The first windowing unit 101 may include a first sub-windowing unit 1011 and N-1 second sub-windowing units 1012; the first processing unit 102 may include N first timing reconstruction units 1023 and second shared computing units 1024.

[0187] In Example 2, the first sub-window unit 1011 uses the original image as the first input data of the first level; it downsamples the first input data of the first level to obtain at least one first image block of the first level, and outputs the output data of the first level based on the at least one first image block of the first level.

[0188] Each of the N-1 second sub-windowing units 1012 is used to take the output data of the (i-1)th level as the first input data of the i-th level, i∈{2,3,……,N}; to downsample the first input data of the i-th level to obtain at least one first image block of the i-th level; and to output the output data of the i-th level based on the at least one first image block of the i-th level.

[0189] At this time, the first window unit 101 can obtain the first input data of each level, and the number is N.

[0190] For N first time-series reconstruction units 1023, a second time-series reconstruction unit is configured to generate a first read signal when a first condition is met, and based on the first read signal, output a first valid signal corresponding to each first image block in the first input data of the corresponding level. The second time-series reconstruction unit is any one of the N first time-series reconstruction units 1023 except for the first time-series reconstruction unit corresponding to the first level; the first condition includes that the third time-series reconstruction unit has output the first valid signal of the corresponding level; the third time-series reconstruction unit includes a first time-series reconstruction unit among the N first time-series reconstruction units 1023 with a higher priority than the second time-series reconstruction unit.

[0191] The second shared computing unit 1024 is used to take the input order of each first valid signal as the first processing timing of each first valid signal, and process the first image block corresponding to each first valid signal in sequence according to the first processing timing to obtain the second input data of each level, and transmit the second input data of each level to the second processing module 20.

[0192] The second processing module 20 uses the second input data of each level as the image data of the Gaussian pyramid corresponding to the original image.

[0193] The following example, using the construction of an N-layer Laplacian pyramid, illustrates the image processing device:

[0194] Example 1: Combination Figure 19 ,like Figure 21 The diagram shown is a schematic representation of an image processing apparatus provided in an embodiment of this application. Figure 21 In this process, the image processing device may include a first processing module 10 and a second processing module 20.

[0195] The first processing module 10 may include a first windowing unit 101 (i.e., a plurality of windowing modules (caching k-1 rows)) and a first processing unit 102. The first windowing unit 101 may include a first sub-windowing unit 1011 (i.e., the windowing module corresponding to the first level (caching k-1 rows)) and N-1 second sub-windowing units 1012; the first processing unit 102 may include a first signal reconstruction unit 1021 (i.e., priority clipping and valid reconstruction) and a first shared computing unit 1022 (i.e., downsampling shared computing unit).

[0196] The second processing module 20 may include N fourth timing reconstruction units 205 (i.e., retiming), a second windowing unit 201 (i.e., multiple windowing modules (caching (k+1) / 2 rows)), a second signal reconstruction unit 202 (i.e., priority clipping and valid reconstruction), a third shared computing unit 203 (i.e., upsampling shared computing unit), and a first construction unit 204.

[0197] In Example 1, the first sub-window unit 1011 uses the original image as the first input data of the first level; it downsamples the first input data of the first level to obtain at least one first image block of the first level, and outputs the output data of the first level based on the at least one first image block of the first level.

[0198] Each of the N-1 second sub-windowing units 1012 is used to take the output data of the (i-1)th level as the first input data of the i-th level, i∈{2,3,……,N}; to downsample the first input data of the i-th level to obtain at least one first image block of the i-th level; and to output the output data of the i-th level based on the at least one first image block of the i-th level.

[0199] At this time, the first window unit 101 can obtain the first input data of each level, and the number is N.

[0200] The first signal reconstruction unit 1021 generates a first target valid signal (valid_new) based on the hierarchical priority of each level, and transmits the first target valid signal to the first shared computing unit 1022. The first target valid signal includes a first valid signal corresponding to each first image block and a first processing timing corresponding to each first valid signal.

[0201] The first shared computing unit 1022 is used to process the first image block corresponding to each first valid signal in sequence according to the first processing timing to obtain the second input data of each level, and transmit the second input data of each level to the second windowing unit 201.

[0202] The second windowing unit 201 is used to upsample the second input data of each level sequentially based on the sampling timing of the second input data of each level to obtain the third input data of each level, and transmit the third input data of each level to the second signal reconstruction unit 202. The third input data of each level includes at least one second image block.

[0203] The second signal reconstruction unit 202 is used to generate a second target valid signal based on the sampling timing of the third input data of each level, and transmit the second target valid signal to the third shared computing unit 203. The second target valid signal includes the second valid signal corresponding to each second image block in the third input data of each level, and the second processing timing corresponding to each second valid signal; the sampling timing of the third input data of each level is the same as the sampling timing of the second input data of each level.

[0204] The third shared computing unit 203 is used to process the second image block corresponding to each second valid signal in sequence according to the second processing timing to obtain the fourth input data of each level, and transmit the fourth input data of each level to the first construction unit 204.

[0205] The first building unit 204 is used to construct the Laplacian pyramid corresponding to the original image based on the second input data of each level and the fourth input data of each level.

[0206] Example 2: Combining Figure 20 ,like Figure 22 The diagram shown is a schematic representation of an image processing apparatus provided in an embodiment of this application. Figure 22 In this process, the image processing device may include a first processing module 10 and a second processing module 20.

[0207] The first processing module 10 may include a first windowing unit 101 (i.e., multiple windowing modules (caching k-1 rows)) and a first processing unit 102. The first windowing unit 101 may include a first sub-windowing unit 1011 (i.e., the windowing module corresponding to the first level (caching k-1 rows)) and N-1 second sub-windowing units 1012; the first processing unit 102 may include N first timing reconstruction units 1023 (i.e., retiming) and second shared computing units 1024 (i.e., downsampling shared computing units).

[0208] The second processing module 20 may include a third windowing unit 206 (i.e., multiple windowing modules (caching (k+1) / 2 rows)), a second processing unit 207 (i.e., retiming), and a second construction unit 208. The second processing unit 207 may include N fifth timing reconstruction units 2071 and a fourth shared computing unit 2072. The fourth shared computing unit 2072 may include a third reconstruction module (i.e., priority clipping and valid reconstruction) and a sub-shared computing unit (i.e., upsampling shared computing unit).

[0209] In Example 2, the first sub-window unit 1011 uses the original image as the first input data of the first level; it downsamples the first input data of the first level to obtain at least one first image block of the first level, and outputs the output data of the first level based on the at least one first image block of the first level.

[0210] Each of the N-1 second sub-windowing units 1012 is used to take the output data of the (i-1)th level as the first input data of the i-th level, i∈{2,3,……,N}; to downsample the first input data of the i-th level to obtain at least one first image block of the i-th level; and to output the output data of the i-th level based on the at least one first image block of the i-th level.

[0211] At this time, the first window unit 101 can obtain the first input data of each level, and the number is N.

[0212] For N first time-series reconstruction units 1023, a second time-series reconstruction unit is configured to generate a first read signal when a first condition is met, and based on the first read signal, output a first valid signal corresponding to each first image block in the first input data of the corresponding level. The second time-series reconstruction unit is any one of the N first time-series reconstruction units 1023 except for the first time-series reconstruction unit corresponding to the first level; the first condition includes that the third time-series reconstruction unit has output the first valid signal of the corresponding level; the third time-series reconstruction unit includes a first time-series reconstruction unit among the N first time-series reconstruction units 1023 with a higher priority than the second time-series reconstruction unit.

[0213] The second shared computing unit 1024 is used to take the input order of each first valid signal as the first processing timing of each first valid signal, and process the first image block corresponding to each first valid signal in sequence according to the first processing timing to obtain the second input data of each level, and transmit the second input data of each level to the third windowing unit 206.

[0214] The third windowing unit 206 is used to upsample the second input data of each level to obtain the fifth input data of each level, and transmit the fifth input data of each level to N fifth temporal reconstruction units 2071. The fifth input data of each level includes at least one third image block.

[0215] The sixth temporal reconstruction unit is used to generate a third read signal when the third condition is met, and based on the third read signal, output the third valid signal corresponding to each third image block in the fifth input data of the corresponding level; the sixth temporal reconstruction unit is any one of the N fifth temporal reconstruction units 2071; the third condition includes that the seventh temporal reconstruction unit has output the third valid signal of the corresponding level; the seventh temporal reconstruction unit includes the fifth temporal reconstruction unit with higher priority than the sixth temporal reconstruction unit among the N fifth temporal reconstruction units 2071;

[0216] The third reconstruction module is used to generate a third target valid signal based on the sampling timing of the fifth input data of each level, and transmit the third target valid signal to the sub-shared computing unit. The third target valid signal includes the third valid signal corresponding to each third image block in the fifth input data of each level, and the third processing timing corresponding to each third valid signal. The sampling timing of the fifth input data of each level is the same as the sampling timing of the second input data of each level.

[0217] The sub-shared computing unit is used to take the input order of each third valid signal as the third processing timing of each third valid signal, and process the third image block corresponding to each third valid signal in sequence according to the third processing timing to obtain the sixth input data of each level, and transmit the second input data of each level to the second construction unit 208.

[0218] The second building unit 208 is also used to build the Laplacian pyramid corresponding to the original image based on the second input data of each level and the sixth input data of each level.

[0219] The image processing method provided in the embodiments of this application is described in detail below:

[0220] It should be noted that the image processing method provided in this application can be executed by an image processing device, an electronic device, or a functional module within an electronic device. Some embodiments of this application use an electronic device executing the image processing method as an example to illustrate the image processing method provided in this application.

[0221] Figure 23 A schematic flowchart of the image processing method provided in an embodiment of this application is shown. Figure 23As shown, the image processing method provided in this application embodiment may include the following steps 2301-2303.

[0222] Step 2301: The electronic device acquires the first input data of each level in the N-layer image pyramid based on the original image.

[0223] The first input data for each level includes at least one first image patch, where N is an integer greater than 1.

[0224] In some embodiments of this application, step 2301 can be specifically implemented by steps 2301a-2301c.

[0225] Step 2301a: The electronic device uses the original image as the first input data of the first level; it downsamples the first input data of the first level to obtain at least one first image block of the first level, and outputs the output data of the first level based on the at least one first image block of the first level.

[0226] In some embodiments of this application, the step 2301a above, "the electronic device downsamples the first input data of the first level to obtain at least one first image block of the first level", can be specifically implemented by the following steps 2301a1-2301a2.

[0227] Step 2301a1: The electronic device performs a convolution operation on the first input data of the first level and a pyramid filter kernel of size a×b to obtain a×b first image blocks corresponding to the first input data of the first level.

[0228] Step 2301a2: The electronic device performs h-fold downsampling on a×b first image blocks to obtain c×d first image blocks of the first level.

[0229] Where a, b, c, and d are all integers greater than or equal to 1, h is an integer greater than 1, c is calculated based on a and h, and d is calculated based on b and h.

[0230] Step 2301b: The electronic device uses the output data of the (i-1)th level as the first input data of the i-th level.

[0231] Where i∈{2,3,……,N}.

[0232] Step 2301c: The electronic device downsamples the first input data of the i-th level to obtain at least one first image block of the i-th level, and outputs the output data of the i-th level based on the at least one first image block of the i-th level.

[0233] Step 2302: The electronic device processes the first image block corresponding to each first valid signal in sequence according to the first processing timing of the first valid signal corresponding to each first image block included in the first input data of each level, so as to obtain the second input data of each level.

[0234] In some embodiments of this application, step 2302 can be specifically implemented by the following steps 2302a-2302b.

[0235] Step 2302a: The electronic device generates a first target valid signal based on the hierarchical priority of each level. The first target valid signal includes a first valid signal corresponding to each first image block and a first processing timing sequence corresponding to each first valid signal.

[0236] Step 2302b: The electronic device processes the first image block corresponding to each first valid signal in sequence according to the first processing timing to obtain the second input data of each level.

[0237] Step 2303: The electronic device constructs an image pyramid corresponding to the original image based on the second input data of each level.

[0238] In some embodiments of this application, step 2303 described above can be implemented in one of the following ways.

[0239] Implementation Method 1: The electronic device uses the second input data of each level as the image data of the image pyramid corresponding to the original image.

[0240] Implementation Method 2: The electronic device upsamples the second input data of each level sequentially based on the sampling timing of the second input data of each level to obtain the third input data of each level. The third input data of each level includes at least one second image block. Based on the sampling timing of the third input data of each level, the electronic device generates a second target valid signal. The second target valid signal includes the second valid signal corresponding to each second image block in the third input data of each level, and the second processing timing corresponding to each second valid signal. The sampling timing of the third input data of each level is the same as the sampling timing of the second input data of each level. The electronic device processes the second image block corresponding to each second valid signal sequentially according to the second processing timing to obtain the fourth input data of each level. Based on the second input data and the fourth input data of each level, the electronic device constructs an image pyramid corresponding to the original image.

[0241] Implementation Method 3: The electronic device upsamples the second input data of each level to obtain the fifth input data of each level. The fifth input data of each level includes at least one third image block. The electronic device processes the third image block corresponding to each third valid signal in sequence according to the third processing timing of the third valid signal corresponding to each third image block to obtain the sixth input data of each level. The electronic device constructs the image pyramid corresponding to the original image based on the second input data and the sixth input data of each level.

[0242] In this embodiment, the electronic device acquires the first input data of each level in an N-layer image pyramid based on the original image. The first input data of each level includes at least one first image block, where N is an integer greater than 1. The electronic device then processes the first image block corresponding to each first valid signal sequentially according to the first processing timing of the first valid signal corresponding to each first image block to obtain the second input data of each level. In this way, the second input data of each level can be quickly determined, simplifying the data calculation process in the electronic device. Based on the second input data of each level, the image pyramid corresponding to the original image is constructed, and the whole process improves the image processing efficiency.

[0243] It should be noted that each of the above method embodiments, or various possible implementations of each method embodiment, can be executed individually or in combination of any two or more. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.

[0244] The image processing method provided in this application can be executed by an image processing device. This application uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application.

[0245] like Figure 24 The diagram shown is a structural schematic of the image processing apparatus provided in an embodiment of this application. Figure 24 In the image processing device, there are an acquisition module 2401 and a processing module 2402.

[0246] The acquisition module 2401 is used to acquire the first input data of each level in an N-layer image pyramid based on the original image. The first input data of each level includes at least one first image patch, where N is an integer greater than 1.

[0247] The processing module 2402 is used to process the first image block corresponding to each first valid signal in sequence according to the first processing time sequence of the first valid signal corresponding to each first image block to obtain the second input data of each level; and to construct the image pyramid corresponding to the original image based on the second input data of each level.

[0248] In some embodiments, the acquisition module 2401 is specifically used to take the original image as the first input data of the first level; downsample the first input data of the first level to obtain at least one first image patch of the first level, and output the output data of the first level based on the at least one first image patch of the first level; take the output data of the (i-1)th level as the first input data of the i-th level, i∈{2,3,……,N}; downsample the first input data of the i-th level to obtain at least one first image patch of the i-th level, and output the output data of the i-th level based on the at least one first image patch of the i-th level.

[0249] In some embodiments, the acquisition module 2401 is specifically used to perform a convolution operation between the first input data of the first level and a pyramid filter kernel of size a×b to obtain a×b first image blocks corresponding to the first input data of the first level; and to perform h-fold downsampling on the a×b first image blocks to obtain c×d first image blocks of the first level; wherein a, b, c and d are all integers greater than or equal to 1, h is an integer greater than 1, c is calculated based on a and h, and d is calculated based on b and h.

[0250] In some embodiments, the processing module 2402 is specifically used to generate a first target valid signal based on the hierarchical priority of each level. The first target valid signal includes a first valid signal corresponding to each first image block and a first processing timing sequence corresponding to each first valid signal. According to the first processing timing sequence, the first image block corresponding to each first valid signal is processed sequentially to obtain the second input data of each level.

[0251] In some embodiments, the processing module 2402 is specifically used to take the second input data of each level as the image data of the image pyramid corresponding to the original image.

[0252] In some embodiments, the processing module 2402 is specifically configured to: upsample the second input data of each level sequentially based on the sampling timing of the second input data of each level to obtain third input data of each level, wherein the third input data of each level includes at least one second image block; generate a second target valid signal based on the sampling timing of the third input data of each level, wherein the second target valid signal includes a second valid signal corresponding to each second image block in the third input data of each level, and a second processing timing corresponding to each second valid signal; the sampling timing of the third input data of each level is the same as the sampling timing of the second input data of each level; process the second image block corresponding to each second valid signal sequentially according to the second processing timing to obtain fourth input data of each level; and construct an image pyramid corresponding to the original image based on the second input data and the fourth input data of each level.

[0253] In some embodiments, the processing module 2402 is specifically used to upsample the second input data of each level to obtain the fifth input data of each level, wherein the fifth input data of each level includes at least one third image block; process the third image block corresponding to each third valid signal in sequence according to the third processing timing of the third valid signal corresponding to each third image block to obtain the sixth input data of each level; and construct the image pyramid corresponding to the original image based on the second input data of each level and the sixth input data of each level.

[0254] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0255] The image processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0256] The image processing apparatus provided in this application embodiment can achieve... Figure 23 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0257] The electronic devices in the embodiments of this application are described below:

[0258] like Figure 25 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Figure 25 In this context, electronic devices include, for example, Figure 3 , Figures 19-22The image processing apparatus shown; or, including, as shown in the image processing apparatus ... Figure 24 The image processing device shown.

[0259] Optionally, such as Figure 26 As shown, this application embodiment also provides an electronic device 2600, including a processor 2601 and a memory 2602. The memory 2602 stores a program or instructions that can run on the processor 2601. When the program or instructions are executed by the processor 2601, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0260] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0261] Figure 27 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0262] The electronic device 2700 includes, but is not limited to, components such as: radio frequency unit 2701, network module 2702, audio output unit 2703, input unit 2704, sensor 2705, display unit 2706, user input unit 2707, interface unit 2708, memory 2709, and processor 2710.

[0263] Those skilled in the art will understand that the electronic device 2700 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 2710 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 27 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0264] The processor 2710 is used to acquire first input data for each level of an N-layer image pyramid based on the original image, wherein the first input data for each level includes at least one first image block and N is an integer greater than 1; process the first image block corresponding to each first valid signal sequentially according to the first processing timing of the first valid signal corresponding to each first image block to obtain second input data for each level; and construct the image pyramid corresponding to the original image based on the second input data for each level.

[0265] Optionally, the processor 2710 is specifically configured to: use the original image as the first input data of the first level; downsample the first input data of the first level to obtain at least one first image block of the first level, and output the output data of the first level based on the at least one first image block of the first level; use the output data of the (i-1)th level as the first input data of the i-th level, i∈{2,3,……,N}; downsample the first input data of the i-th level to obtain at least one first image block of the i-th level, and output the output data of the i-th level based on the at least one first image block of the i-th level.

[0266] Optionally, the processor 2710 is specifically configured to perform a convolution operation on the first input data of the first level and a pyramid filter kernel of size a×b to obtain a×b first image blocks corresponding to the first input data of the first level; and to perform h-fold downsampling on the a×b first image blocks to obtain c×d first image blocks of the first level; wherein a, b, c and d are all integers greater than or equal to 1, h is an integer greater than 1, c is calculated based on a and h, and d is calculated based on b and h.

[0267] Optionally, the processor 2710 is specifically configured to generate a first target valid signal based on the hierarchical priority of each level. The first target valid signal includes a first valid signal corresponding to each first image block and a first processing timing sequence corresponding to each first valid signal. According to the first processing timing sequence, the first image block corresponding to each first valid signal is processed sequentially to obtain the second input data of each level.

[0268] Optionally, the processor 2710 is specifically configured to use the second input data of each level as image data of the image pyramid corresponding to the original image.

[0269] Optionally, the processor 2710 is specifically configured to: upsample the second input data of each level sequentially based on the sampling timing of the second input data of each level to obtain the third input data of each level, wherein the third input data of each level includes at least one second image block; generate a second target valid signal based on the sampling timing of the third input data of each level, wherein the second target valid signal includes the second valid signal corresponding to each second image block in the third input data of each level, and the second processing timing corresponding to each second valid signal; the sampling timing of the third input data of each level is the same as the sampling timing of the second input data of each level; process the second image block corresponding to each second valid signal sequentially according to the second processing timing to obtain the fourth input data of each level; and construct an image pyramid corresponding to the original image based on the second input data and the fourth input data of each level.

[0270] Optionally, the processor 2710 is specifically configured to upsample the second input data of each level to obtain the fifth input data of each level, wherein the fifth input data of each level includes at least one third image block; process the third image block corresponding to each third valid signal sequentially according to the third processing timing of the third valid signal corresponding to each third image block to obtain the sixth input data of each level; and construct the image pyramid corresponding to the original image based on the second input data of each level and the sixth input data of each level.

[0271] In this embodiment, the processor 2710 obtains the first input data of each level in the N-layer image pyramid based on the original image. The first input data of each level includes at least one first image block, where N is an integer greater than 1. The processor 2710 then processes the first image block corresponding to each first valid signal in sequence according to the first processing timing of the first valid signal corresponding to each first image block to obtain the second input data of each level. In this way, this solution can quickly determine the second input data of each level, simplifying the data calculation process in the processor 2710. Based on the second input data of each level, the image pyramid corresponding to the original image is constructed, and the whole process improves the image processing efficiency.

[0272] It should be understood that, in this embodiment, the input unit 2704 may include a graphics processing unit (GPU) 27041 and a microphone 27042. The GPU 27041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 2706 may include a display panel 27061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 2707 includes at least one of a touch panel 27071 and other input devices 27072. The touch panel 27071 is also called a touch screen. The touch panel 27071 may include a touch detection device and a touch controller. Other input devices 27072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0273] The memory 2709 can be used to store software programs and various data. The memory 2709 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 2709 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 2709 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0274] Processor 2710 may include one or more processing units; optionally, processor 2710 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 2710.

[0275] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0276] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0277] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0278] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0279] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described image processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0280] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0281] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0282] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing apparatus, characterized in that, It includes a first processing module and a second processing module, wherein the first processing module includes a first window opening unit and a first processing unit connected in sequence; The first windowing unit is used to acquire the first input data of each level in the N-layer image pyramid based on the original image, and transmit the first input data of each level to the first processing unit. The first input data of each level includes at least one first image block, and N is an integer greater than 1. The first processing unit is configured to process the first image block corresponding to each first valid signal sequentially according to the first processing timing of the first valid signal corresponding to each first image block, to obtain the second input data of each level, and to transmit the second input data of each level to the second processing module; The second processing module is used to construct an image pyramid corresponding to the original image based on the second input data of each level.

2. The image processing apparatus according to claim 1, characterized in that, The first windowing unit includes a first sub-windowing unit and N-1 second sub-windowing units, wherein the N-1 second sub-windowing units correspond one-to-one with the other N-1 levels in the N-layer image pyramid, excluding the first level. The first sub-window unit is used to take the original image as the first input data of the first level; downsample the first input data of the first level to obtain at least one first image block of the first level; and output the output data of the first level based on the at least one first image block of the first level. The second sub-windowing unit is used to take the output data of the (i-1)th level as the first input data of the i-th level, i∈{2,3,……,N}; to downsample the first input data of the i-th level to obtain at least one first image block of the i-th level; and to output the output data of the i-th level based on the at least one first image block of the i-th level.

3. The image processing apparatus according to claim 2, characterized in that, The first sub-window unit is specifically used for: By performing a convolution operation between the first input data of the first level and a pyramid filter kernel of size a×b, a×b first image blocks corresponding to the first input data of the first level are obtained; The a×b first image blocks are downsampled by h times to obtain the c×d first image blocks of the first level; Where a, b, c, and d are all integers greater than or equal to 1, h is an integer greater than 1, c is calculated based on a and h, and d is calculated based on b and h.

4. The image processing apparatus according to claim 1, characterized in that, The first processing unit includes a first signal reconstruction unit and a first shared computing unit connected in sequence; The first signal reconstruction unit is used to generate a first target valid signal based on the hierarchical priority of each level, and transmit the first target valid signal to the first shared computing unit. The first target valid signal includes a first valid signal corresponding to each first image block and a first processing timing sequence corresponding to each first valid signal. The first shared computing unit is configured to process the first image block corresponding to each first valid signal sequentially according to the first processing timing to obtain the second input data of each level, and transmit the second input data of each level to the second processing module.

5. The image processing apparatus according to claim 1, characterized in that, The first processing unit includes N first temporal reconstruction units and a second shared computing unit, wherein the N first temporal reconstruction units correspond one-to-one with the N levels of the N-layer image pyramid; The second temporal reconstruction unit is configured to generate a first read signal when a first condition is met, and based on the first read signal, output a first valid signal corresponding to each first image block in the first input data of the corresponding level; the second temporal reconstruction unit is any one of the N first temporal reconstruction units except the first temporal reconstruction unit corresponding to the first level; the first condition includes that the third temporal reconstruction unit has output the first valid signal of the corresponding level; the third temporal reconstruction unit includes the first temporal reconstruction unit among the N first temporal reconstruction units with a higher priority than the second temporal reconstruction unit; The second shared computing unit is used to take the input order of each first valid signal as the first processing timing of each first valid signal, and process the first image block corresponding to each first valid signal in sequence according to the first processing timing to obtain the second input data of each level, and transmit the second input data of each level to the second processing module.

6. The image processing apparatus according to claim 5, characterized in that, The N first timing reconstruction units are N first first-in-first-out circuits; each first first-in-first-out circuit includes a first first-in-first-out queue and a first judgment unit. The second timing reconstruction unit includes a first judgment unit, which is used to determine whether the third timing reconstruction unit has output the first valid signal of the corresponding level, and transmit the first judgment result to the first first-in-first-out queue included in the second timing reconstruction unit; The second time-series reconstruction unit includes a first first-in-first-out queue, which is used to generate the first read signal when the first judgment result indicates that the first condition is met, and output the first valid signal corresponding to each first image block in the first input data of the corresponding level based on the first read signal.

7. The apparatus according to claim 1, characterized in that, The second processing module is specifically used to: take the second input data of each level as the image data of the image pyramid corresponding to the original image.

8. The apparatus according to claim 1, characterized in that, The second processing module includes a second windowing unit, a second signal reconstruction unit, a third shared computing unit, and a first construction unit connected in sequence; The second windowing unit is used to upsample the second input data of each level sequentially based on the sampling timing of the second input data of each level to obtain the third input data of each level, and transmit the third input data of each level to the second signal reconstruction unit. The third input data of each level includes at least one second image block. The second signal reconstruction unit is used to generate a second target valid signal based on the sampling timing of the third input data of each level, and transmit the second target valid signal to the third shared computing unit. The second target valid signal includes a second valid signal corresponding to each second image block in the third input data of each level, and a second processing timing corresponding to each second valid signal. The sampling timing of the third input data at each level is the same as the sampling timing of the second input data at each level; The third shared computing unit is used to process the second image block corresponding to each second valid signal sequentially according to the second processing timing to obtain the fourth input data of each level, and transmit the fourth input data of each level to the first construction unit. The first construction unit is used to construct an image pyramid corresponding to the original image based on the second input data of each level and the fourth input data of each level.

9. The apparatus according to claim 8, characterized in that, The second processing module further includes N fourth temporal reconstruction units; the N fourth temporal reconstruction units correspond one-to-one with the N levels of the N-layer image pyramid; The fourth temporal reconstruction unit is used to generate a second read signal when a second condition is met, and based on the second read signal, output a second valid signal corresponding to each second image block in the second input data of the corresponding level; the second condition includes that the number of second valid signals output by the fourth temporal reconstruction unit at the corresponding level reaches a preset number for the corresponding level of the fourth temporal reconstruction unit.

10. The apparatus according to claim 9, characterized in that, The N fourth timing reconstruction units are N second first-in-first-out circuits; each second first-in-first-out circuit includes a second first-in-first-out queue, a counter, and a second judgment unit. The second first-in-first-out queue is used to generate a second read signal when the second judgment result indicates that the second condition is met, and to output a second valid signal corresponding to each second image block in the second input data of the corresponding level based on the second read signal; The counter is used to count the number of second valid signals that have been output in the second first-in-first-out queue and to transmit the count to the second judgment unit. The second judgment unit is used to determine whether the number of second valid signals output by the corresponding level counted by the counter reaches the preset number of the corresponding level of the fourth timing reconstruction unit, and transmits the second judgment result to the second first-in-first-out queue.

11. The apparatus according to claim 1, characterized in that, The second processing module includes a third window opening unit, a second processing unit, and a second construction unit; The third windowing unit is used to upsample the second input data of each level to obtain the fifth input data of each level, and transmit the fifth input data of each level to the second processing unit. The fifth input data of each level includes at least one third image block. The second processing unit is configured to process the third image block corresponding to each third valid signal sequentially according to the third processing timing of the third valid signal corresponding to each third image block, to obtain the sixth input data of each level, and to transmit the sixth input data of each level to the second construction unit. The second construction unit is further configured to construct an image pyramid corresponding to the original image based on the second input data of each level and the sixth input data of each level.

12. The apparatus according to claim 11, characterized in that, The second processing unit further includes N fifth temporal reconstruction units and a fourth shared computing unit; the N fifth temporal reconstruction units correspond one-to-one with the N levels of the N-layer image pyramid; The sixth temporal reconstruction unit is used to generate a third read signal when the third condition is met, and based on the third read signal, output a third valid signal corresponding to each third image block in the fifth input data of the corresponding level; the sixth temporal reconstruction unit is any one of the N fifth temporal reconstruction units; the third condition includes that the seventh temporal reconstruction unit has output the third valid signal of the corresponding level; the seventh temporal reconstruction unit includes a fifth temporal reconstruction unit with a higher priority than the sixth temporal reconstruction unit among the N fifth temporal reconstruction units; The fourth shared computing unit is used to take the input order of each third valid signal as the third processing sequence of each third valid signal, and process the third image block corresponding to each third valid signal in sequence according to the third processing sequence to obtain the sixth input data of each level, and transmit the second input data of each level to the second construction unit.

13. An image processing method, characterized in that, include: Based on the original image, the first input data of each level in the N-layer image pyramid is obtained, wherein the first input data of each level includes at least one first image patch, and N is an integer greater than 1; According to the first processing timing of the first valid signal corresponding to each first image block, the first image block corresponding to each first valid signal is processed sequentially to obtain the second input data of each level; Based on the second input data of each level, an image pyramid corresponding to the original image is constructed.

14. The method according to claim 13, characterized in that, The process of obtaining the first input data for each level of the N-layer image pyramid based on the original image includes: The original image is used as the first input data of the first level; the first input data of the first level is downsampled to obtain at least one first image patch of the first level, and the output data of the first level is output based on the at least one first image patch of the first level; Use the output data of level (i-1) as the first input data of level i, where i∈{2,3,……,N}; The first input data of the i-th level is downsampled to obtain at least one first image patch of the i-th level, and the output data of the i-th level is output based on the at least one first image patch of the i-th level.

15. The method according to claim 14, characterized in that, The step of downsampling the first input data of the first level to obtain at least one first image patch of the first level includes: By performing a convolution operation between the first input data of the first level and a pyramid filter kernel of size a×b, a×b first image blocks corresponding to the first input data of the first level are obtained; The a×b first image blocks are downsampled by h times to obtain the c×d first image blocks of the first level; Where a, b, c, and d are all integers greater than or equal to 1, h is an integer greater than 1, c is calculated based on a and h, and d is calculated based on b and h.

16. The method according to claim 13, characterized in that, The first image block corresponding to each first valid signal is processed sequentially according to the first processing time sequence of the first valid signal corresponding to each first image block to obtain the second input data of each level, including: Based on the hierarchical priority of each level, a first target valid signal is generated. The first target valid signal includes a first valid signal corresponding to each first image block and a first processing sequence corresponding to each first valid signal. According to the first processing sequence, the first image block corresponding to each first valid signal is processed sequentially to obtain the second input data of each level.

17. The method according to claim 13, characterized in that, The process of constructing the image pyramid corresponding to the original image based on the second input data of each level includes: The second input data of each level is used as the image data of the image pyramid corresponding to the original image.

18. The method according to claim 13, characterized in that, The process of constructing the image pyramid corresponding to the original image based on the second input data of each level includes: Based on the sampling timing of the second input data of each level, the second input data of each level is upsampled sequentially to obtain the third input data of each level, wherein the third input data of each level includes at least one second image patch; Based on the sampling timing of the third input data of each level, a second target valid signal is generated. The second target valid signal includes a second valid signal corresponding to each second image block in the third input data of each level, and a second processing timing corresponding to each second valid signal. The sampling timing of the third input data of each level is the same as the sampling timing of the second input data of each level. According to the second processing timing, the second image block corresponding to each second valid signal is processed sequentially to obtain the fourth input data of each level; Based on the second input data and the fourth input data of each level, an image pyramid corresponding to the original image is constructed.

19. The method according to claim 13, characterized in that, The process of constructing the image pyramid corresponding to the original image based on the second input data of each level includes: The second input data of each level is upsampled to obtain the fifth input data of each level, wherein the fifth input data of each level includes at least one third image patch; According to the third processing timing of the third valid signal corresponding to each third image block, the third image block corresponding to each third valid signal is processed sequentially to obtain the sixth input data of each level; Based on the second input data of each level and the sixth input data of each level, an image pyramid corresponding to the original image is constructed.

20. An image processing apparatus, characterized in that, include: Acquisition module and processing module; The acquisition module is used to acquire the first input data of each level in the N-layer image pyramid based on the original image. The first input data of each level includes at least one first image patch, where N is an integer greater than 1. The processing module is used to process the first image block corresponding to each first valid signal sequentially according to the first processing time sequence of the first valid signal corresponding to each first image block, so as to obtain the second input data of each level. Based on the second input data of each level, an image pyramid corresponding to the original image is constructed.

21. An electronic device, characterized in that, Includes the image processing apparatus according to any one of claims 1-12 or the image processing apparatus according to claim 20.

22. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the image processing method as described in any one of claims 13-19.

23. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the image processing method as described in any one of claims 13-19.

24. A chip, characterized in that, The chip includes a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the image processing method as described in any one of claims 13-19.