Image processing method, device, module and chip

Abstract

This application provides an image processing method, device, module, and chip. The method includes obtaining an original image and setting the number of image pyramid layers N. Using the original image as the initial input image of a sampling unit, starting from the original image, time-division multiplexing the sampling unit to upsample the input image N-1 times, generating an output image corresponding to each upsampling, wherein the subsequent input image for each upsampling from the second to the N-1th upsampling is the output image generated by the previous upsampling, where N ≥ 2 and N is an integer. Using the original image as the bottom layer of the image pyramid, and using the output image generated by each upsampling as the upper layer of the corresponding upsampled input image, thereby generating an image pyramid corresponding to the original image. This image processing method can generate an image pyramid while taking into account system resources, bandwidth, time consumption, and power consumption.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to an image processing method, device, module, and chip. Background Art

[0002] The inverted image pyramid is a structure that simply and effectively describes an image by resolution. It is obtained by sequentially upsampling an original input image. The bottom layer of the pyramid is the original input image with the lowest resolution, and the top layer is the output image with the highest resolution. The resolution of each intermediate layer is X times that of the immediately adjacent bottom layer and 1 / X that of the immediately adjacent top layer, where X is determined by the upsampling step size. For example, if a conventional 0.5 step size is used for upsampling, the resolution of each intermediate layer of the inverted image pyramid is 4 times that of the immediately adjacent bottom layer and 1 / 4 that of the immediately adjacent top layer.

[0003] The inverted image pyramid generation method repeatedly calls a horizontal and vertical upsampling processing unit to obtain a series of inverted pyramid image blocks, which are then combined to generate the inverted image pyramid. This method requires repeated reading and writing of data from a storage device (such as DDR), thus consuming a large amount of system bandwidth, and the processing speed is also limited by the system latency.

[0004] Another traditional method is to obtain a series of inverted pyramid images by cascading multiple horizontal and vertical upsampling processing units. Although this method can effectively reduce system bandwidth and latency, it consumes more power and occupies a larger chip area because it requires cascading multiple (>2) processing units. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide an image processing method, device, module and chip to solve the above problems.

[0006] In a first aspect, the present invention provides an image processing method for generating an image pyramid. The method includes: obtaining an original image and a set number of image pyramid layers N; using the original image as the initial input image of a sampling unit, and starting from the original image, time-division multiplexing the sampling unit to upsample the input image N-1 times to generate an output image corresponding to each upsampling, wherein the subsequent input image of each upsampling from the second to the N-1th times is the output image generated by the previous upsampling, N≥2 and N is an integer; using the original image as the bottom layer of the image pyramid, and using the output image generated by each upsampling as the upper layer of the corresponding upsampled input image, thereby generating an image pyramid corresponding to the original image.

[0007] In the image processing method described above, after obtaining an original image and a set number of image pyramid levels, N, the original image is used as the initial input image for the sampling unit. The sampling unit is then time-division multiplexed to perform N-1 upsampling operations on the image input to the sampling unit, thereby obtaining an output image corresponding to each upsampling operation. The original image is then used as the bottom layer of the image pyramid, and the output image generated by each upsampling operation is used as the top layer of the corresponding upsampled input image, thereby generating an image pyramid corresponding to the original image. Because, except for the initial reading of the original image, all subsequent images input to the sampling unit are the output images obtained from the previous upsampling operation, this method eliminates the need to repeatedly obtain the original image, i.e., repeatedly read the original image data from the front-end module or storage module, thereby saving system bandwidth and the time required to read the original image data. Furthermore, because the same sampling unit is time-division multiplexed, this method only requires a single sampling unit, eliminating the need for multiple cascaded upsampling units. This results in low power consumption and a small footprint, effectively utilizing system resources.

[0008] In an optional implementation of the first aspect, starting from the original image, time-division multiplexing the sampling unit to upsample the input image N-1 times to generate an output image corresponding to each upsampling, including: upsampling the original image by the sampling unit to generate an output image; after the output image of the previous upsampling is generated, inputting the output image of the previous upsampling into the sampling unit as the input image of this upsampling, time-division multiplexing the sampling unit to upsample the input image of this upsampling to generate the output image of this upsampling, and after generating the output image of this upsampling, continuing with the next upsampling operation; repeating this process until the number of upsampling times reaches N-1 times, and obtaining the output image corresponding to each upsampling.

[0009] In an optional implementation of the first aspect, the sampling unit includes a horizontal upsampling unit and a vertical upsampling unit, and the original image is upsampled by the sampling unit to generate an output image, including: upsampling each row of data in the original image by the horizontal upsampling unit to generate an initial image; and upsampling each column of data in the initial image by the vertical upsampling unit to generate the output image.

[0010] In an optional implementation of the first aspect, after the output image of the previous upsampling is generated, the output image of the previous upsampling is input into the horizontal upsampling unit as the input image of this upsampling, and the horizontal upsampling unit is time-division multiplexed to upsample the input image of this upsampling to generate an initial image of this upsampling; after the initial image of this upsampling is generated, the initial image of this upsampling is input into the vertical upsampling unit, and the vertical upsampling unit is time-division multiplexed to upsample the initial image of this upsampling to generate an output image of this upsampling, and after the output image of this upsampling is generated, the next upsampling operation is continued; this process is repeated until the number of upsampling times reaches N-1 times, and the output image corresponding to each upsampling is obtained.

[0011] In an optional implementation of the first aspect, after generating the upsampled initial image, the method further includes: caching data of preset rows in the initial image; inputting the data of the preset rows in the initial image after each caching into the vertical upsampling unit, and upsampling the data of the preset rows in the initial image input each time through the vertical upsampling unit to generate the upsampled output image.

[0012] In an optional implementation of the first aspect, before inputting the last up-sampled output image into the sampling unit as the input image for this up-sampling, the method further includes: caching data of a preset row of the last up-sampled output image.

[0013] In an optional implementation of the first aspect, the output image generated by each upsampling and the image input by each upsampling both have identifiers, and the output image generated by each upsampling has a mapping relationship with the identifiers of the image input by the next upsampling.

[0014] In a second aspect, the present invention provides an image processing device for generating an image pyramid, the device comprising: an acquisition module for acquiring an original image and a set number N of image pyramid layers; an upsampling module for using the original image as the initial input image of a sampling unit, starting from the original image, time-division multiplexing the sampling unit to upsample the input image N-1 times, and generating an output image corresponding to each upsampling, wherein the subsequent input image of each upsampling from the 2nd to the N-1th times is the output image generated by the previous upsampling, N≥2 and N is an integer; a generation module for using the original image as the bottom layer of the image pyramid, and using the output image generated by each upsampling as the upper layer of the corresponding upsampled input image, thereby generating an image pyramid corresponding to the original image.

[0015] In the image processing device described above, after obtaining an original image and a set number of image pyramid levels, N, this scheme uses the original image as the initial input image for the sampling unit. The sampling unit then uses time-division multiplexing to upsample the image input to the sampling unit N-1 times, thereby obtaining an output image corresponding to each upsampling step. The original image is then used as the bottom layer of the image pyramid, and the output image generated by each upsampling step serves as the upper layer of the corresponding upsampled input image, thereby generating an image pyramid corresponding to the original image. Because this scheme, except for the initial reading of the original image, all subsequent images input to the sampling unit are the output images obtained from the previous upsampling step, eliminates the need to repeatedly acquire the original image, i.e., repeatedly read the original image data from the front-end module or storage module, thereby saving system bandwidth and the time required to read the original image data. Furthermore, this scheme time-division multiplexes the same sampling unit, requiring only a single sampling unit, eliminating the need for multiple cascaded upsampling units. This results in low power consumption and a small footprint, effectively utilizing system resources.

[0016] In an optional implementation of the second aspect, the upsampling module is specifically configured to upsample the original image through the sampling unit to generate an output image; after the output image of the previous upsampling is generated, input the output image of the previous upsampling into the sampling unit as the input image of this upsampling, time-division multiplex the sampling unit to upsample the input image of this upsampling to generate the output image of this upsampling, and after the output image of this upsampling is generated, continue with the next upsampling operation; repeat this process until the number of upsampling times reaches N-1 times, and obtain the output image corresponding to each upsampling.

[0017] In an optional implementation of the second aspect, the sampling unit includes a horizontal upsampling unit and a vertical upsampling unit, and the upsampling module is further specifically used to upsample each row of data in the original image through the horizontal upsampling unit to generate an initial image; and upsample each column of data in the initial image through the vertical upsampling unit to generate the output image.

[0018] In an optional implementation manner of the second aspect, the upsampling module is further specifically used to, after the output image of the previous upsampling is generated, input the output image of the previous upsampling as the input image of this upsampling into the horizontal upsampling unit, time-division multiplexing the horizontal upsampling unit to upsample the input image of this upsampling to generate the initial image of this upsampling; after the initial image of this upsampling is generated, input the initial image of this upsampling into the vertical upsampling unit, time-division multiplexing the vertical upsampling unit to upsample the initial image of this upsampling to generate the output image of this upsampling, and after the output image of this upsampling is generated, continue with the next upsampling operation; repeat this process until the number of upsampling times reaches N-1 times, and obtain the output image corresponding to each upsampling.

[0019] In an optional implementation of the second aspect, the device also includes a cache module for caching data of preset rows in the initial image; the upsampling module is further specifically used to input the data of the preset rows in the initial image after each caching into the vertical upsampling unit, and upsample the data of the preset rows in the initial image input each time through the vertical upsampling unit to generate an output image of this upsampling.

[0020] In an optional implementation of the second aspect, the cache module is further configured to cache data of a preset row of the last upsampled output image.

[0021] In a third aspect, the present invention provides an image processing module, comprising an input image reading unit, an input control unit, an upsampling unit, and an output control unit; the input image reading unit is configured to read an original image and transmit the read original image to the input control unit; the input control unit is configured to transmit each received image as an input image to the upsampling unit; the upsampling unit is configured to generate an output image corresponding to each upsampling using a time-division multiplexing method for each received input image, and transmit the output image corresponding to each upsampling to the output control unit; the output control unit is configured to transmit the generated output image corresponding to each upsampling to the input control unit after each upsampling, and use the output image generated for each upsampling as an upper layer of the corresponding upsampling input image, thereby generating an image pyramid corresponding to the original image after N-1 upsampling times, wherein N is a set number of image pyramid layers, N ≥ 2 and N is an integer, and the bottom layer of the image pyramid is the original image.

[0022] In the image processing module designed above, this solution uses an output control unit to transmit the output image generated by each upsampling back to the input control unit, so that the input control unit transmits each received image as the input image to the upsampling unit, thereby time-division multiplexing the upsampling unit to upsample the input image, and then, after upsampling reaches N-1 times, generates an image pyramid corresponding to the original image. Except for the first time reading the original image, the image processing module designed in this solution all subsequent images input to the upsampling unit are the output images obtained by the previous upsampling and transmitted back by the output control unit. This eliminates the need to repeatedly use the input image reading unit to read the original image, that is, there is no need to repeatedly read the original image data from the front-end module or storage module, thereby saving system bandwidth and time saving the reading of the original image data. At the same time, this solution time-division multiplexes the same upsampling unit, so only one upsampling unit is required, without configuring multiple cascaded upsampling units, which makes the image processing module designed in this solution low in power consumption and occupies a small chip area.

[0023] In an optional embodiment of the third aspect, the output end of the input image reading unit is connected to the input end of the input control unit, the output end of the input control unit is connected to the input end of the upsampling unit, the output end of the upsampling unit is connected to the input end of the output control unit, and the output end of the output control unit is respectively connected to an external storage device and the input control unit.

[0024] In an optional embodiment of the third aspect, the upsampling unit includes a horizontal upsampling unit and a vertical upsampling unit, and the input end of the horizontal upsampling unit is connected to the output end of the input control unit, so as to use a time division multiplexing method to horizontally upsample the input image transmitted by the input control unit each time to generate an initial image corresponding to each horizontal upsampling, and transmit the initial image corresponding to each horizontal upsampling to the vertical upsampling unit; the input end of the vertical upsampling unit is connected to the output end of the horizontal upsampling unit, so as to use a time division multiplexing method to vertically upsample the initial image corresponding to each horizontal upsampling transmitted by the horizontal upsampling unit each time to generate an output image corresponding to each upsampling.

[0025] In an optional implementation of the third aspect, the image processing module further includes a line cache unit, which is arranged between the input image reading unit and the upsampling unit to cache image data of a preset line and then transmit it to the upsampling unit.

[0026] In an optional embodiment of the third aspect, the image processing module further includes a line cache unit, which is arranged between the horizontal upsampling unit and the vertical upsampling unit to preset line caches for the initial image corresponding to each horizontal upsampling and then transmit it to the vertical upsampling unit.

[0027] In a fourth aspect, the present invention provides a chip, comprising the image processing module described in any optional embodiment of the third aspect.

[0028] In a fifth aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method in the first aspect or any optional implementation of the first aspect is executed.

[0029] In a sixth aspect, the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method in the first aspect or any optional implementation of the first aspect is executed.

[0030] In a seventh aspect, the present application provides a computer program product, which, when running on a computer, enables the computer to execute the method in the first aspect or any optional implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 A schematic diagram of the image processing method provided in an embodiment of the present application;

[0033] Figure 2 A first exemplary diagram of the image processing method provided in an embodiment of the present application;

[0034] Figure 3 A second exemplary diagram of the image processing method provided in an embodiment of the present application;

[0035] Figure 4 A third exemplary diagram of the image processing method provided in an embodiment of the present application;

[0036] Figure 5 A fourth example diagram of the image processing method provided in an embodiment of the present application;

[0037] Figure 6 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0038] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0039] Figure 8 A first structural diagram of the image processing module provided in an embodiment of the present application;

[0040] Figure 9 A second structural diagram of the image processing module provided in an embodiment of the present application;

[0041] Figure 10 A third structural diagram of the image processing module provided in an embodiment of the present application;

[0042] Figure 11 A fourth structural diagram of the image processing module provided in an embodiment of the present application;

[0043] Figure 12 This is a fifth structural diagram of the image processing module provided in an embodiment of the present application.

[0044] Icon: 10-input image reading unit; 20-input control unit; 30-upsampling unit; 301-horizontal upsampling unit; 302-vertical upsampling unit; 40-output control unit; 50-output image unit; 60-row cache unit; 600-acquisition module; 610-upsampling module; 620-generation module; 630-cache module; 7-electronic device; 701-processor; 702-memory; 703-communication bus. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0046] First embodiment

[0047] The embodiment of the present application provides an image processing method, which is used to generate an image pyramid. Specifically, the image processing method is used to generate an inverted image pyramid. The image processing method can be applied to a chip, such as Figure 1 As shown, the method may include the following steps:

[0048] Step S100: obtaining an original image and setting the number of image pyramid layers N.

[0049] Step S110: Using the original image as the initial input image of the sampling unit, starting from the original image, the time-division multiplexing sampling unit upsamples the input image N-1 times to generate an output image corresponding to each upsampling, wherein the subsequent input image of each upsampling from the 2nd to the N-1th times is the output image generated by the previous upsampling, N≥2 and N is an integer.

[0050] Step S120: using the original image as the bottom layer of an image pyramid, and using the output image generated by each upsampling as the upper layer of the corresponding upsampled input image, thereby generating an image pyramid corresponding to the original image.

[0051] In step S100, the original image represents the image to be converted into the image pyramid. The original image can be obtained through real-time input from the front-end module or read from a storage device (such as a double data rate synchronous dynamic random access memory DDR). The image data format of the original image can be any image format such as YUV, RGB, etc.

[0052] In addition, in step S100, this solution also needs to set the number of layers N of the image pyramid, that is, the number of layers that the generated target image pyramid needs to have. For example, when N is 4, it means that the total number of layers from the bottom to the top of the image pyramid to be generated is 4.

[0053] In step S110, this solution uses the original image obtained in step S100 as the initial input image of the sampling unit, and then starts from the original image, time-division multiplexing the sampling unit to upsample the image input to the sampling unit N-1 times, generating an output image corresponding to each upsampling, wherein, starting from the second time, the image input to the sampling unit each time is the output image completed by the previous upsampling, N≥2 and N is an integer.

[0054] As a possible implementation, the execution process of step S110 may be as follows: first, the original image is used as an initial input image of a sampling unit, the original image is input into the sampling unit, and the sampling unit upsamples the original image to generate a first upsampled output image; then, the first upsampled output image is input into the sampling unit, and the sampling unit upsamples the first upsampled output image again through time division multiplexing to generate a second upsampled output image; then, the second upsampled output image is input into the sampling unit again, and the sampling unit upsamples the second upsampled output image again through time division multiplexing to generate a third upsampled output image;

[0055] The above process is repeated, that is, after the output image of the previous upsampling is generated, the output image of the previous upsampling is input into the sampling unit as the input image of this upsampling, the time-division multiplexing sampling unit upsamples the input image of this upsampling to generate the output image of this upsampling, and after the output image of this upsampling is generated, the output image of this upsampling is used as the input image of the next upsampling, until the number of upsampling times reaches N-1 times, and the output image corresponding to each upsampling is obtained.

[0056] The above step S110 is explained by taking the following example: Assume that the original image A is as follows: Figure 2 The original image resolution of the 2-row 2-column image shown in FIG. 1 is 4, that is, the number of pixels contained in the image is 4, and the number of image pyramid layers is set to 4. Then, according to the aforementioned step S110, the original image of 2 rows and 2 columns is first input into the sampling unit. The sampling unit assumes that the original image A of 2 rows and 2 columns is upsampled with a step size of 0.5, and then the following is obtained: Figure 2 The output image B of the first upsampling with 4 rows and 4 columns and a resolution of 16 is shown in .

[0057] The output image B of the first upsampling is input to the sampling unit again, and the sampling unit is time-division multiplexed to upsample the output image B with a step size of 0.5, and the following is obtained: Figure 2 The second upsampled output image C with 8 rows and 8 columns and a resolution of 64 is shown in .

[0058] The output image C of the second upsampling is input to the same sampling unit again, and the sampling unit is time-division multiplexed to upsample the output image C with a step size of 0.5, and the following is obtained: Figure 2 The third up-sampled output image D with 16 rows and 16 columns and a resolution of 258 is shown in , and then the output images B, output image C and output image D that have been up-sampled N-1=4-1=3 times are obtained.

[0059] After obtaining the output image of each upsampling in the aforementioned manner, the present solution may execute step S120 to use the original image as the bottom layer of the image pyramid and the output image generated by each upsampling as the upper layer of the corresponding upsampled input image, thereby generating an inverted image pyramid corresponding to the original image.

[0060] This solution can generate an inverted image pyramid corresponding to the original image through the following methods. As a possible implementation method, this solution can first use the original image as the bottom layer of the image pyramid. Then, after each upsampling is completed to generate an output image, the generated output image is superimposed on the top layer of the image pyramid.

[0061] For example, according to the above example, Figure 3 As shown, this solution can first use the original image A as the bottom layer of the image pyramid. After upsampling the original image A through the sampling unit to generate the output image B, the output image B can be superimposed on the original image A, thus forming a two-layer image pyramid; then, after generating the output image C, the output image C is superimposed on the top of the two-layer image pyramid, that is, the output image C is superimposed on the output image B, thus forming a three-layer image pyramid; finally, after generating the output image D, the output image D is superimposed on the top of the three-layer image pyramid, thereby generating the final required four-layer inverted image pyramid.

[0062] Through the above-mentioned image pyramid generation method, this solution does not need to identify the image pyramid layer number corresponding to each output image. Instead, each output image can be directly superimposed on the original image pyramid in sequence, and then the output image generated by the last sampling can be superimposed to obtain the final image pyramid.

[0063] As another possible implementation, this solution can store each generated output image. After the sampling number reaches N-1 times to generate all the output images, all the output images are superimposed to form the final inverted image pyramid. In this way, this solution can identify the number of layers corresponding to each output image or the stacking order corresponding to all the generated output images, and then stack the output images in sequence according to the number of layers or the corresponding stacking order to obtain the final inverted image pyramid.

[0064] For example, according to the above example, Figure 4 As shown, after generating all the output images B, output images C, and output images D, this scheme assigns the 0th layer identifier to the original image A, the 1st layer identifier to the output image B, the 2nd layer identifier to the output image C, and the 3rd layer identifier to the output image D. Then, when generating the image pyramid, the original image A has the 0th layer identifier and is placed at the bottom of the image pyramid, the output image B has the 1st layer identifier superimposed on the original image A, the output image C has the 2nd layer identifier superimposed on the output image B, and the output image D has the 3rd layer identifier superimposed on the output image C, thereby generating the final required four-layer inverted image pyramid.

[0065] In the above-described image processing method, after obtaining an original image and a set number of image pyramid levels, N, the original image is used as the initial input image for the sampling unit. The sampling unit is then time-division multiplexed to perform N-1 upsampling operations on the image input to the sampling unit, thereby obtaining an output image corresponding to each upsampling operation. The original image is then used as the bottom layer of the image pyramid, and the output image generated by each upsampling operation is used as the upper layer of the corresponding upsampled input image, thereby generating an image pyramid corresponding to the original image. Because, except for the initial reading of the original image, all subsequent images input to the sampling unit are the output images obtained from the previous upsampling operation, this method eliminates the need to repeatedly obtain the original image, i.e., repeatedly read the original image data from the front-end module or storage module, thereby saving system bandwidth and the time required to read the original image data. Furthermore, because the same sampling unit is time-division multiplexed, this method requires only a single sampling unit, eliminating the need for multiple cascaded upsampling units. This results in low power consumption and a small footprint.

[0066] In an optional implementation of this embodiment, the aforementioned sampling unit may be a combination of a horizontal sampling unit and a vertical sampling unit, i.e., a single sampling unit includes both horizontal upsampling and vertical upsampling functions. In addition to using the aforementioned sampling unit, the horizontal upsampling and vertical upsampling functions of the sampling unit may be separated, thereby making the sampling unit include a horizontal upsampling unit and a vertical upsampling unit.

[0067] On the basis of the above-mentioned functional separation, the sampling unit in the aforementioned step S110 upsamples the input image in the following manner: the output image of the previous upsampling is input into the horizontal upsampling unit as the input image of this upsampling, and the horizontal upsampling unit is time-division multiplexed to upsample the input image of this upsampling to generate the initial image of this upsampling; the initial image of this upsampling is then input into the vertical upsampling unit, and the vertical upsampling unit is time-division multiplexed to upsample the initial image of this upsampling to generate the output image of this upsampling, and after the output image of this upsampling is generated, the next upsampling operation is continued; this process is repeated until the number of upsampling times reaches N-1 times, and the output image corresponding to each upsampling is obtained.

[0068] According to the above example, the output image B is generated by the original image A and the output image C is generated by the output image B. Figure 5 As shown, the original image A is first input into the horizontal upsampling unit, which performs horizontal upsampling on the original image A with a step size of 0.5, so that the resolution of each row of the original image A is doubled, thereby forming an initial image A'; the initial image A' is input into the vertical upsampling unit, which performs vertical upsampling on the initial image A' with a step size of 0.5, so that the resolution of each column of the initial image A' is doubled, thereby obtaining the output image B.

[0069] The output image B is passed back to the horizontal upsampling unit, which horizontally upsamples the output image B with a step size of 0.5, so that the resolution of each row of the output image B is doubled, thereby forming an initial image B'; the initial image B' is input to the vertical upsampling unit, which vertically upsamples the initial image B' with a step size of 0.5, so that the resolution of each column of the initial image B' is doubled, thereby obtaining the output image C.

[0070] In an optional implementation manner of this embodiment, image data is generally transmitted based on each row of image data during the transmission process. Since the vertical upsampling unit generally upsamples multiple rows of data, that is, upsamples the image from a numerical direction, after the initial image of this upsampling is generated as mentioned above, this scheme can cache the data of a preset row in the initial image, and then after caching the data of the preset row, input the cached data of the preset row into the vertical upsampling unit, and upsample the data of the preset row in the initial image input each time through the vertical upsampling unit to generate the output image of this upsampling.

[0071] In an optional implementation of this embodiment, this solution can also cache a preset number of rows of input image data before inputting it to the horizontal upsampling unit. The cached data is the row data before the resolution is expanded, thereby saving cache space.

[0072] Figure 6 The schematic structural block diagram of the image processing device provided by this application is shown. It should be understood that the device is Figures 1 to 5 The method embodiment executed in the embodiment corresponds to the method, and can execute the steps involved in the aforementioned method. The specific functions of the device can be found in the description above. To avoid repetition, detailed description is omitted here. The device includes at least one software functional module that can be stored in a memory or embedded in the device's operating system (OS) in the form of software or firmware. Specifically, the device includes: an acquisition module 600 for acquiring an original image and a set number of image pyramid layers N; an upsampling module 610 for using the original image as the initial input image of the sampling unit, and starting from the original image, time-division multiplexing the sampling unit to upsample the input image N-1 times to generate an output image corresponding to each upsampling, wherein the subsequent input image for each upsampling from the second to the N-1th upsampling is the output image generated by the previous upsampling, where N ≥ 2 and N is an integer; and a generation module 620 for using the original image as the bottom layer of the image pyramid and using the output image generated by each upsampling as the upper layer of the corresponding upsampled input image, thereby generating an image pyramid corresponding to the original image.

[0073] In the image processing device designed above, after obtaining an original image and a set number of image pyramid levels, N, this scheme uses the original image as the initial input image for the sampling unit. The sampling unit then uses time-division multiplexing to upsample the image input to the sampling unit N-1 times, thereby obtaining an output image corresponding to each upsampling step. The original image is then used as the bottom layer of the image pyramid, and the output image generated by each upsampling step serves as the upper layer of the corresponding upsampled input image, thereby generating an image pyramid corresponding to the original image. Because this scheme, except for the initial reading of the original image, all subsequent images input to the sampling unit are the output images obtained by the previous upsampling step, eliminates the need to repeatedly acquire the original image, i.e., repeatedly read the original image data from the front-end module or storage module, thereby saving system bandwidth and the time required to read the original image data. Furthermore, this scheme time-division multiplexes the same sampling unit, requiring only a single sampling unit, eliminating the need for multiple cascaded upsampling units. This results in low power consumption and a small footprint.

[0074] In an optional implementation manner of this embodiment, the upsampling module 610 is specifically configured to upsample the original image through a sampling unit to generate an output image; after the output image of the previous upsampling is generated, the output image of the previous upsampling is input into the sampling unit as the input image of this upsampling, the time-division multiplexing sampling unit upsamples the input image of this upsampling to generate the output image of this upsampling, and after the output image of this upsampling is generated, continue with the next upsampling operation; repeat this process until the number of upsampling times reaches N-1 times, and obtain the output image corresponding to each upsampling.

[0075] In an optional implementation of this embodiment, the sampling unit includes a horizontal upsampling unit and a vertical upsampling unit, and the upsampling module 610 is further specifically used to upsample each row of data in the original image through the horizontal upsampling unit to generate an initial image; and upsample each column of data in the initial image through the vertical upsampling unit to generate an output image.

[0076] In an optional implementation manner of this embodiment, the upsampling module 610 is further specifically configured to, after the output image of the previous upsampling is generated, input the output image of the previous upsampling as the input image of this upsampling into the horizontal upsampling unit, and use the time-division multiplexing horizontal upsampling unit to upsample the input image of this upsampling to generate the initial image of this upsampling; after the initial image of this upsampling is generated, input the initial image of this upsampling into the vertical upsampling unit, and use the time-division multiplexing vertical upsampling unit to upsample the initial image of this upsampling to generate the output image of this upsampling; and after the output image of this upsampling is generated, continue with the next upsampling operation; repeat this process until the number of upsampling times reaches N-1 times, and obtain the output image corresponding to each upsampling.

[0077] In an optional implementation of this embodiment, the device also includes a cache module 630 for caching data of preset rows in the initial image; the upsampling module 610 is further specifically used to input the data of preset rows in the initial image that has been cached each time into the vertical upsampling unit, and upsample the data of preset rows in the initial image input each time through the vertical upsampling unit to generate an output image of this upsampling.

[0078] In an optional implementation manner of this embodiment, the cache module 630 is further configured to cache data of a preset row of the last up-sampled output image.

[0079] like Figure 7 As shown, the present application provides an electronic device 7, including: a processor 701 and a memory 702. The processor 701 and the memory 702 are interconnected and communicate with each other via a communication bus 703 and / or other forms of connection mechanisms (not shown). The memory 702 stores a computer program executable by the processor 701. When the computing device is running, the processor 701 executes the computer program to perform a method in any optional implementation manner, such as steps S100 to S130: obtaining an original image and a set number of image pyramid layers N; using the original image as the initial input image of the sampling unit, starting from the original image, time-division multiplexing the sampling unit to upsample the input image N-1 times to generate an output image corresponding to each upsampling, wherein the subsequent input image of each upsampling from the second to the N-1th times is the output image generated by the previous upsampling, N ≥ 2 and N is an integer; using the original image as the bottom layer of the image pyramid, and using the output image generated by each upsampling as the upper layer of the corresponding upsampled input image, thereby generating an image pyramid corresponding to the original image.

[0080] The present application provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method in any of the aforementioned optional implementations is executed.

[0081] Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0082] The present application provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the method in any optional implementation manner.

[0083] Second embodiment

[0084] The embodiment of the present application provides an image processing module, which uses hardware implementation to implement the aforementioned image processing method to generate an inverted pyramid, such as Figure 8 As shown, the image processing module includes an input image reading unit 10 , an input control unit 20 , an upsampling unit 30 , an output control unit 40 and an output image unit 50 .

[0085] The output end of the input image reading unit 10 is connected to the input end of the input control unit 20, the output end of the input control unit 20 is connected to the input end of the upsampling unit 30, the output end of the upsampling unit 30 is connected to the input end of the output control unit 40, and the output end of the output control unit 40 is connected to the input control unit 20 and the output image unit 50 respectively.

[0086] When the image processing module designed above is used, the input image reading unit 10 reads the original image input in real time by the previous module or reads the original image from the storage device through the data bus, and transmits the read original image to the input control unit 20.

[0087] The input control unit 20 transmits the original image to the upsampling unit 30 , and the upsampling unit 30 upsamples the original image with a preset step size to generate a first upsampled output image and transmits it to the output control unit 40 .

[0088] The output control unit 40 transmits the first upsampled output image back to the input control unit 20 and transmits the first upsampled output image to the output image unit 50, so that the output image unit 50 stores the first upsampled output image or overlays it on the original image originally stored in the output image unit 50.

[0089] After the first sampled output image is transmitted back to the input control unit 20, the input control unit 20 continues to transmit the first sampled output image to the upsampling unit 30, thereby time-division multiplexing the upsampling unit 30 to upsample the first upsampled output image to obtain a second upsampled output image, and transmits the second upsampled output image to the output control unit 40.

[0090] On the one hand, the output control unit 40 returns the second up-sampled output image to the input control unit 20, and on the other hand, transmits the second up-sampled output image to the output image unit 50, so that the output image unit 50 stores the second up-sampled output image or superimposes it on the upper layer of the first up-sampled output image.

[0091] The input control unit 20 transmits the second up-sampled output image to the up-sampling unit 30, and continues to time-division multiplex the up-sampling unit 30 for up-sampling, and repeats this process until the number of up-sampling times reaches N-1 times.

[0092] In the image processing module designed above, this solution uses an output control unit to transmit the output image generated by each upsampling back to the input control unit, so that the input control unit transmits each received image as the input image to the upsampling unit, thereby time-division multiplexing the upsampling unit to upsample the input image, and then, after upsampling reaches N-1 times, generates an image pyramid corresponding to the original image. Except for the first time reading the original image, the image processing module designed in this solution all subsequent images input to the upsampling unit are the output images obtained by the previous upsampling and transmitted back by the output control unit. This eliminates the need to repeatedly use the input image reading unit to read the original image, that is, there is no need to repeatedly read the original image data from the front-end module or storage module, thereby saving system bandwidth and time saving the reading of the original image data. At the same time, this solution time-division multiplexes the same upsampling unit, so only one upsampling unit is required, without configuring multiple cascaded upsampling units, which makes the image processing module designed in this solution low in power consumption and occupies a small chip area.

[0093] In an optional implementation of this embodiment, if Figure 9As shown, the upsampling unit 30 may include a horizontal upsampling unit 301 and a vertical upsampling unit 302, the input end of the horizontal upsampling unit 301 is connected to the output end of the input control unit 20, the input end of the vertical upsampling unit 302 is connected to the output end of the horizontal upsampling unit 301, and the output end of the vertical upsampling unit 302 is connected to the output control unit 40.

[0094] Through the above-designed structure, when performing each of the aforementioned upsampling, the time-division multiplexing horizontal upsampling unit 301 of this scheme first upsamples the input image in the row direction, so that the resolution in the row direction is expanded; then the time-division multiplexing vertical upsampling unit 302 upsamples the image after the horizontal upsampling in the column direction, so that the resolution in the column direction is expanded, and then generates an output image after the upsampling. The horizontal upsampling and vertical upsampling methods have been described in the first embodiment and will not be repeated here.

[0095] It should be noted here that the positions of the above-mentioned horizontal upsampling unit 301 and the vertical upsampling unit 302 are interchangeable, that is, the vertical upsampling unit 302 can be time-division multiplexed to expand the resolution of the input image in the column direction, and then the horizontal upsampling unit 301 can be time-division multiplexed to upsample the image in the row direction after the vertical upsampling is completed, so that the resolution in the row direction is expanded.

[0096] In an optional implementation of this embodiment, since the vertical upsampling unit 302 generally needs to upsample multiple rows of data in the column direction, and actual hardware processing is generally input row by row, this solution can design a row cache unit 60 to cache multiple rows of data.

[0097] As a possible implementation, Figure 10 As shown, the row buffer unit 60 may be provided between the horizontal upsampling unit 301 and the vertical upsampling unit 302; as another possible implementation, as shown in FIG. Figure 11 As shown, the line buffer unit 60 may be provided between the input control unit 20 and the horizontal upsampling unit 301 .

[0098] In an optional implementation of this embodiment, the output control unit 40 may receive a set number of image pyramid layers, and then after the upsampling unit 30 performs N-1 upsampling times, the generated output image is not transmitted back to the input control unit 20, thereby terminating the time division multiplexing process.

[0099] In an optional implementation of this embodiment, in a hardware environment, in order to identify the data accordingly, this solution can perform corresponding identification on various data. For example, if the original image data is used as the input of the 0th layer of the image pyramid, the input control unit 20 can set the identification of ltag_in[0]=1 for the original image data, set the identification of ltag_in[1]=1 for the input image data of the 1st layer of the image pyramid, and set the identification of ltag_in[N-1]=1 for the input image data of the Nth layer of the image pyramid.

[0100] Correspondingly, the output control unit 40 may set the ltag_out[0]=1 flag for the output data of the 0th layer of the image pyramid, and set the ltag_out[1:N-1]=1 flag for the output data of the 1st to Nth layers of the image pyramid. In this way, the hardware device can identify the data of the corresponding layer of the image pyramid based on the flag.

[0101] In an optional implementation of this embodiment, under a hardware environment, it is generally necessary to time-division multiplex the hardware logic of the upsampling unit in units of input lines. Assuming that the total number of lines of the original image is H, the total number of lines received by the input control unit 20 is (2 N -1) H line, on this basis, the input control unit 20 can have the following functions:

[0102] Since the input control unit 20 needs to receive (2 N -1)H lines, therefore, the input control unit 20 may assign a certain line number to each received image, for example, the line numbers of the input lines may be 0, 1, 2, 3, 4...(2 N -2), (2 N -1).

[0103] Then the input control unit can assign the row numbers of H rows in all row numbers to the input image of layer 0, assign the row numbers of 2H rows in all row numbers to the input image of layer 1, and so on. N-1 The row numbers of the rows are assigned to the input image of the N-2th layer. By setting this and the identifier of each layer, different parameters can be selected according to the identifier to control the upsampling process of each layer in the upsampling unit 30.

[0104] For example, the corresponding upsampling weighting coefficient of each layer can be set to ltag[0]={0.5.0.5}, ltag[1]={0.4.0.6}, ltag[2]={0.7,0.3}, ltag[3]={0.6,0,4}. Through the function of the above-mentioned input control unit 20, it can be determined according to which layer the current output row belongs to which group of the above-mentioned four groups of upsampling weighting coefficients to use. If ltag_in[0]=1, ltag[0] is selected, and if ltag_in[1]=1, ltag[1] is selected, so that different upsampling weighting systems can be used for upsampling of different layers.

[0105] In an optional implementation of this embodiment, if Figure 12 As shown, the output image can be returned not only through the output control unit 40, but also directly returned to the input control unit 20 after the vertical upsampling unit 302 vertically upsamples the input initial image data to generate the corresponding output image.

[0106] In an optional implementation of this embodiment, this solution designs a chip, which includes the image processing module described in any implementation of the second embodiment.

[0107] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0108] In addition, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0109] Furthermore, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0110] It should be noted that if the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0111] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0112] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: The method is used to generate an image pyramid, and the method includes: Get the original image and the set number of image pyramid layers N; Using the original image as the initial input image of the sampling unit, starting from the original image, time-division multiplexing the sampling unit to upsample the input image N-1 times, generating an output image corresponding to each upsampling, wherein the subsequent input image of each upsampling from the 2nd to the N-1th times is the output image generated by the previous upsampling, N ≥ 2 and N is an integer; and The original image is used as the bottom layer of an image pyramid, and the output image generated by each upsampling is used as the upper layer of the corresponding upsampled input image, thereby generating an image pyramid corresponding to the original image; Starting from the original image, time-division multiplexing the sampling unit to perform N-1 upsampling on the input image to generate an output image corresponding to each upsampling, including: Upsampling the original image by the sampling unit to generate an output image; After the output image of the previous upsampling is generated, the output image of the previous upsampling is input into the sampling unit as the input image of the current upsampling. The sampling unit is time-division multiplexed to upsample the input image of the current upsampling to generate the output image of the current upsampling. After the output image of the current upsampling is generated, the next upsampling operation is continued. This process is repeated until the number of upsampling times reaches N-1 times, and the output image corresponding to each upsampling is obtained.

2. The method according to claim 1, characterized in that The sampling unit includes a horizontal upsampling unit and a vertical upsampling unit, and upsampling the original image by the sampling unit to generate an output image includes: Upsampling each row of data in the original image by the horizontal upsampling unit to generate an initial image; Each column of data in the initial image is up-sampled by the vertical up-sampling unit to generate the output image.

3. The method according to claim 2, characterized in that After the output image of the previous upsampling is generated, the output image of the previous upsampling is input into the horizontal upsampling unit as the input image of the current upsampling, and the horizontal upsampling unit is used to upsample the input image of the current upsampling by time division multiplexing to generate an initial image of the current upsampling; After generating the initial image of the current upsampling, inputting the initial image of the current upsampling into the vertical upsampling unit, time-division multiplexing the vertical upsampling unit to upsample the initial image of the current upsampling to generate the output image of the current upsampling, and continuing to perform the next upsampling operation after generating the output image of the current upsampling; Repeat this process until the number of upsampling reaches N-1 times, and obtain the output image corresponding to each upsampling.

4. The method according to claim 2, characterized in that After generating the upsampled initial image, the method further includes: Buffering data of a preset row in the initial image; The data of the preset row in the initial image that is cached each time is input into the vertical upsampling unit, and the data of the preset row in the initial image that is input each time is upsampled by the vertical upsampling unit to generate an output image of this upsampling.

5. The method according to claim 1, wherein Before inputting the last up-sampled output image into the sampling unit as the input image for this up-sampling, the method further includes: buffering data of a preset row of the last up-sampled output image.

6. The method according to claim 1, characterized in that The output image generated by each upsampling and the image input by each upsampling both have identifiers, and the output image generated by each upsampling has a mapping relationship with the identifier of the image input by the next upsampling.

7. An image processing device, characterized in that The device is used to generate an image pyramid, and the device includes: An acquisition module is used to obtain the original image and the set number of image pyramid layers N; an upsampling module, configured to use the original image as an initial input image of a sampling unit, and starting from the original image, time-division multiplexing the sampling unit to upsample the input image N-1 times, generating an output image corresponding to each upsampling, wherein the subsequent input image of each upsampling from the second to the N-1th times is the output image generated by the previous upsampling, where N ≥ 2 and N is an integer; a generating module, configured to use the original image as the bottom layer of an image pyramid and use the output image generated by each upsampling as the upper layer of the corresponding upsampled input image, thereby generating an image pyramid corresponding to the original image; The upsampling module is specifically configured to upsample the original image through the sampling unit to generate an output image; after the output image of the previous upsampling is generated, input the output image of the previous upsampling into the sampling unit as the input image of the current upsampling, time-division multiplex the sampling unit to upsample the input image of the current upsampling to generate the output image of the current upsampling, and after the output image of the current upsampling is generated, continue with the next upsampling operation; repeat this process until the number of upsampling times reaches N-1 times, and obtain the output image corresponding to each upsampling.

8. An image processing module, characterized in that: The image processing module includes an input image reading unit, an input control unit, an upsampling unit and an output control unit; The input image reading unit is used to read the original image and transmit the read original image to the input control unit; The input control unit is configured to transmit the image received each time as an input image to the upsampling unit; The upsampling unit is configured to generate an output image corresponding to each upsampling by using a time division multiplexing method for each received input image, and transmit the output image corresponding to each upsampling to the output control unit; The output control unit is configured to transmit the generated output image corresponding to the upsampling to the input control unit after each upsampling, and use the output image generated each time as an upper layer of the upsampling input image of the corresponding time, thereby generating an image pyramid corresponding to the original image after N-1 upsampling times, where N is a set number of image pyramid layers, N ≥ 2 and N is an integer, and the bottom layer of the image pyramid is the original image; The output control unit is specifically configured to, after the output image of the previous upsampling is generated, input the output image of the previous upsampling into the sampling unit as the input image of the current upsampling, perform upsampling on the input image of the current upsampling by time-division multiplexing of the sampling unit to generate the output image of the current upsampling, and continue with the next upsampling operation after the output image of the current upsampling is generated; repeat this process until the number of upsampling times reaches N-1 times, thereby obtaining the output image corresponding to each upsampling.

9. A chip, characterized in that: The chip includes the image processing module described in claim 8.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

11. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images