Image processing method, chip, electronic device and storage medium

Abstract

The embodiment of the present application discloses an image processing method, chip, electronic device and storage medium, the method comprising: loading search window data from a storage cell array and caching the search window data to a register array; performing a group search on the register array according to the motion trajectory of multiple pixel points in the current frame, and outputting the pixel data of the reference points corresponding to the multiple pixel points in parallel; and determining the pixel data of the multiple pixel points according to the pixel data of the reference points. In this way, by loading the search window data from the storage cell array to the register array, the reference points of the multiple pixel points can be read in parallel from the register array in a few clock cycles or even one clock cycle, thereby solving the access conflict problem existing in directly reading the storage cell and improving the reading efficiency of the reference points. By implementing a group search on the register array through different types of multiplexers, the layout and wiring of the multiplexer can be simplified.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to an image processing method, chip, electronic device and storage medium. Background Art

[0002] Motion compensation technology predicts the motion trajectory of pixels based on the relationship between frames, and reads the pixel data of the reference point from the previous frame image based on the predicted motion trajectory (including the offset in the X and Y directions) to obtain the pixel data of the pixel point in the current frame.

[0003] The difficulty in hardware design for motion compensation technology lies in simultaneously acquiring pixel data from multiple reference points in the previous image frame within a single clock cycle. However, due to the randomness and real-time nature of motion trajectories, the motion of reference points follows an irregular pattern. When reading pixel data from reference points, pixel data from multiple reference points may be stored in the same memory cell, leading to memory cell access conflicts. Summary of the Invention

[0004] To solve the above technical problems, the embodiments of the present application hope to provide an image processing method, chip, electronic device and storage medium.

[0005] The technical solution of this application is achieved as follows:

[0006] In a first aspect, an image processing method is provided, the method comprising:

[0007] Loading search window data from a storage unit array; wherein the storage unit array stores pixel data of a reference frame;

[0008] Buffering the search window data in a register array;

[0009] According to the motion trajectories of multiple pixel points in the current frame, the register array is searched in groups, and pixel data of reference points corresponding to the multiple pixel points are output in parallel;

[0010] The pixel data of the plurality of pixel points are determined according to the pixel data of the reference point.

[0011] In a second aspect, an image processing chip is provided, comprising: a storage unit array, a register array, and a processor.

[0012] The storage unit array is used to store pixel data of the reference frame;

[0013] The register array is used to cache search window data;

[0014] The processor is used to load search window data from the storage unit array and cache the search window data to the register array; read pixel data of reference points corresponding to the multiple pixel points in parallel from the register array according to the motion trajectories of the multiple pixel points in the current frame; and determine the pixel data of the pixel point based on the pixel data of the reference point.

[0015] In a third aspect, an electronic device is provided, comprising: an image acquisition device and any image processing chip according to the second aspect.

[0016] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0017] In an embodiment of the present application, an image processing method, chip, electronic device and storage medium are provided, the method comprising: loading search window data from a storage cell array; wherein the storage cell array stores pixel data of a reference frame; caching the search window data to a register array; performing a group search on the register array according to the motion trajectory of multiple pixel points in the current frame, and outputting the pixel data of the reference points corresponding to the multiple pixel points in parallel; and determining the pixel data of the multiple pixel points according to the pixel data of the reference points. In this way, by loading the search window data from the storage cell array to the register array, the reference points of the multiple pixel points can be read in parallel from the register array in a few clock cycles or even one clock cycle, thereby solving the access conflict problem existing in directly reading the storage cell and improving the reading efficiency of the reference points. By implementing a group search on the register array through different types of multiplexers, the layout and wiring of the multiplexer can be simplified. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a basic framework for image processing networks;

[0019] Figure 2 This is a schematic diagram of the first flow chart of the image processing method according to an embodiment of the present application;

[0020] Figure 3 This is a storage structure for pixel data of a reference frame in an embodiment of the present application;

[0021] Figure 4 Schematic diagram of the composition structure of the memory cell array in the embodiment of the present application;

[0022] Figure 5 This is a first schematic diagram of the search window movement trajectory in an embodiment of the present application;

[0023] Figure 6 This is a schematic diagram of the first process of the data loading mode in an embodiment of the present application;

[0024] Figure 7 This is a second flow chart of the data loading mode in an embodiment of the present application;

[0025] Figure 8 Schematic diagram of reference point distribution of the search window in an embodiment of the present application;

[0026] Figure 9 Schematic diagram of the first group selection module for register group search in an embodiment of the present application;

[0027] Figure 10 Schematic diagram of the second group selection module for register group search in an embodiment of the present application;

[0028] Figure 11 Schematic diagram of the selection modules for the third and fourth groups of register group search in an embodiment of the present application;

[0029] Figure 12 This is a second schematic diagram of the movement trajectory of the search window in an embodiment of the present application;

[0030] Figure 13 This is a schematic diagram of the storage format of the motion trajectory in the embodiment of the present application;

[0031] Figure 14 Schematic diagram of the positions of 16 pixels in the current frame in an embodiment of the present application;

[0032] Figure 15 Schematic diagram of the composition structure of the image processing chip in the embodiment of the present application;

[0033] Figure 16 Schematic diagram of the structure of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0035] Current mobile real-time video applications face a problem: due to bandwidth limitations and real-time requirements, they transmit low-resolution video data. However, to provide a better visual experience for users, terminal displays generally have higher resolutions, resulting in a mismatch between the transmitted video data and the terminal display resolution. To address this resolution mismatch, the video data resolution needs to be increased for display.

[0036] Video super-resolution technology differs from single-frame super-resolution tasks in that it not only needs to generate images with rich spatial details, but also needs to maintain the coherence between images.

[0037] Current video super-resolution technology treats the video super-resolution problem as multiple single-frame super-resolution tasks. Each super-resolution task generates a high-resolution image from a low-resolution input image. However, due to the lack of connection between the resulting high-resolution images, temporal continuity is poor, resulting in artifacts. This approach also results in high computational complexity.

[0038] In order to solve the problem of temporal continuity, the latest algorithm research results are to consider the image of the previous frame when generating the next frame of image, and input it into the image processing network to improve temporal continuity. This can produce temporal continuity results, making the generated high-resolution images smoother and more natural.

[0039] Figure 1 It is a basic framework of an image processing network, which includes three components: a cycle generator 11, an optical flow estimation network 12, and an interpolation module 13. The image generation network implementation formula is as follows:

[0040] Vt=BicubicResize(F(X(t-1),X(t)))

[0041] G(t)=G(X(t),W(G(t-1),V(t)))+BicubicResize(X(t))

[0042] The optical flow estimation network 12 includes four parts: FNET, BiCubicResize, Warp and Concatenate. The input of the optical flow estimation network 12 is G(t-1), X(t-1) and X(t). G(t-1) is the previous frame image generated by the image generation network, which is used as the reference frame of the current frame. X(t) and X(t-1) are the input low-resolution images. The two frames are input into the FNET of the optical flow estimation network 12, and F(X(t-1), X(t)) is output. The coordinate offset of the current frame is obtained by the BicubicResize operator Vt=BicubicResize(F(X(t-1)) ),X(t))); For each pixel point that needs to be generated, the Warp operator needs to find the reference point W(G(t-1), Vt) in the G(t-1) frame according to the predicted motion trajectory Vt. The loop generator 11 (Generator) will generate G(X(t), W(G(t-1), V(t))) based on W(G(t-1), Vt) and the low-resolution image X(t). The interpolation module 13 (BicubicResize) is used to perform a cubic interpolation operation on X(t) to generate BicubicResize(X(t)). The results of the loop generator 11 and the interpolation module 13 are added to obtain the current frame G(t).

[0043] The difficulty of the latest video super-resolution algorithms lies in the hardware implementation of reference point reading. In view of this, the embodiment of the present application provides an image processing method to achieve parallel reading of reference points.

[0044] Figure 2 This is a first flow chart of the image processing method according to an embodiment of the present application. Figure 2 As shown, the method may specifically include:

[0045] Step 201: loading search window data from a storage unit array and caching the search window data into a register array; wherein the storage unit array stores pixel data of a reference frame;

[0046] The search window can be understood as the search range for reference points for the pixels to be generated. For example, the latest video super-resolution algorithms stipulate that each generated pixel must be found within a range of plus or minus 16 pixels from the same position in the previous frame G(t-1) (i.e., the reference frame) to output the reference point. If 16 pixels are output simultaneously, the selection must be made from (16+16+16)*(16+16+1)=1584 pixels. In practical applications, the window size can be adjusted to 48*32=1536.

[0047] Since the memory cell array stores pixel data of the entire reference frame, it is necessary to first load the search window data from the memory cell array and cache it in the register array, and then read the register array in parallel.

[0048] In some embodiments, the storage unit array includes: M horizontally distributed storage units, each storage unit including N vertically distributed storage spaces; M and N are both integers greater than 1. Storing pixel data of the reference frame in groups in multiple storage units facilitates loading search window data.

[0049] For example, Figure 3 The storage structure of the pixel data of the reference frame in the embodiment of the present application is as follows: Figure 3 As shown, each row of pixels in the reference frame occupies a storage space corresponding to a storage unit address, and the reference frames are grouped and stored in the storage unit array.

[0050] Figure 4 FIG. 1 is a schematic diagram of the composition structure of the memory cell array in the embodiment of the present application. Figure 3 As shown, a chip select signal is used to select a storage unit. Each storage unit occupies multiple addresses. By inputting the address, the storage space to be read is determined, and the pixel data of a row of reference points stored therein is read and cached in the register array.

[0051] For example, the search window size is 48*32, and each storage unit stores data with a width of 16 bytes. Each storage unit shares an address interface, and the storage unit is selected by the chip select signal. For example, assuming the image size is 128x64 bytes, 128 / 16=8 storage units are required for storage, and each storage unit actually occupies 64 addresses. When data needs to be read from a storage unit, the chip select signal selects the storage unit to be read and sends the address to the storage unit. The data read from the storage unit is stored in the register array. The size of the register array is fine-tuned according to the window size, for example, from 48x33 to 48x32=1536.

[0052] Specifically, the loading of the search window data from the storage cell array and caching the search window data to the register array includes: obtaining loading information of the search window; wherein the loading information includes a chip select signal and address information of the storage cell; selecting at least two adjacent storage cells from the M storage cells based on the chip select signal; and loading a row of valid data of the search window from the storage space of the at least two storage cells based on the address information, and caching the data to the register array until all valid data of the search window are loaded.

[0053] Exemplarily, in some embodiments, the method further includes: taking the upper left corner of the reference frame as the starting position of the search window, controlling the movement of the search window in a moving order from top to bottom and from left to right, and updating the loading information of the search window.

[0054] The reference points of several pixels in the current frame can be searched from a search window. After the pixel is determined, the search window needs to be moved and the search window data cached in the register array needs to be updated to search for reference points of other pixels in the current frame.

[0055] The reference frame includes valid data and border filling data. Figure 5 Schematic diagram of the search window moving trajectory in the embodiment of the present application, as shown in FIG. Figure 5 As shown, the white box in the middle is the valid data of the reference frame, and the gray parts around it represent the boundary filling data (Padding). The boundary filling data can also be understood as invalid data. The search window starts at the upper left corner in the initial stage. At this time, the first loading of data only requires reading 32*16 pixel data from the storage unit. The other data is Padding data and does not need to be read from the storage unit. The chip select signal selects two storage units, corresponding to Figure 5The first two storage units read one row of valid data per clock cycle, and a total of 16 clock cycles are used to fill the window with valid data. The padding data value is configured through the register and does not need to be read from the storage unit. Figure 5 As can be seen in the figure, each time the window starts to slide from the top, it takes 16 clock cycles to fill the register with valid data, and the other data is padding data. When the sliding window moves from the top to the bottom, each storage unit only needs to read one row of data per clock cycle, remove the register data in the top row, move the data up one row, and put the row of data read from the storage unit into the bottom row. Reusing other pixel data can generate new sliding window data. Figure 4 The pixel data storage format in ,only needs to read the data of 2 or 3 storage units each time to complete the generation of new window data, making full use of parallelism.

[0056] Figure 6 This is a first flow chart of the data loading mode in the embodiment of the present application. Figure 6 As shown in the figure, when the search window moves from top to bottom, that is, from time T0 to time T1, the search window also needs to move downward from time T1 to the lowest end, indicating that the vertical movement of the column is complete. When the search window completes the vertical movement, it needs to move horizontally back to the top of the image to load the next column of data, that is, the position corresponding to time T2, and repeat the data loading process.

[0057] like Figure 6 As shown, the search window needs to move downward from time T1 to the lowest end. The data within the window is gradually replaced by padding data. After 16 clock cycles, the search window moves to the bottom and then moves to the top of the image to continue loading data. At time T2, data loading also requires 16 clock cycles. Since the valid data required from time T1 to the bottom remains unchanged and is already within the search window, updating the search window data during this period does not affect the pixel data acquisition of the reference point.

[0058] In view of this, an embodiment of the present application also provides another optional data loading process. Specifically, the method also includes: after the search window moves from top to bottom to the first boundary position, each clock cycle loads a row of valid data at the top starting position of the next column of the search window, and caches it to the register array until the search window moves to the second boundary position; wherein, the reference frame includes valid data and boundary filling data, the first boundary position is the lower boundary of the valid data, that is, the search window corresponding to time T1, and the second boundary position is the lower boundary of the boundary filling data.

[0059] Figure 7FIG. 1 is a second flow chart of the data loading mode in the embodiment of the present application. Figure 7 As shown, from time T1 to time T2, each time the window moves downward, a row of data within the search window is removed. Without the need to read new valid data from the storage unit into the register array, the data required for the search window corresponding to time T2 can be loaded simultaneously using the storage unit's read interface. This allows the data required for the search window at time T2 to be ready after 16 clock cycles, eliminating the time required to reload the required valid data each time the window moves to the top. This speeds up data loading and increases hardware parallelism.

[0060] Step 202: performing group search on the register array according to the motion trajectories of multiple pixel points in the current frame, and outputting pixel data of reference points corresponding to the multiple pixel points in parallel;

[0061] How to find the corresponding reference point in a fixed search window is particularly important. Figure 8 As shown, the latest video super-resolution algorithm stipulates that each pixel generated must find a reference point within the range of plus or minus 16 pixels at the same position in the reference frame G(t-1) to output. There are two cases. The first case is that the motion trajectory is an integer, then the corresponding pixel can be directly found in the input reference frame image. If it is a decimal, it is necessary to refer to four surrounding points for interpolation to obtain the new pixel. Therefore, if the minimum hardware processing unit generates 16 points at a time, then at least 16 points need to be read in each time, and at most 16*4=64 points need to be read in each time to ensure that the minimum hardware processing unit generates 16 pixels each time. Figure 8 In the search window, the black box at the center represents the 16 pixels to be output. The reference points for these pixels can be integer reference points (i.e., reference points with integer coordinate offsets) or decimal reference points (i.e., reference points with decimal coordinate offsets). Integer reference points can be used to find the corresponding pixel in the G(t-1) image. In decimal reference points, the new pixel is interpolated using the four surrounding decimal reference points. Since each output pixel needs to be referenced within a range of plus or minus 16, if all 16 pixels are generated simultaneously, a selection must be made from (16+16+16)*(16+16+1)=1584 pixels. In practical applications, the window size can be adjusted to 48*32=1536.

[0062] Specifically, after obtaining the search window data, the register array is searched in groups based on the motion trajectory. Group search refers to dividing the register array into register sub-arrays of different sizes and searching the register sub-arrays using different types of multiplexers. Exemplary multiplexer types include 4-to-1, 8-to-1, 16-to-1, 32-to-1, 48-to-1, and 64-to-1. In practical applications, at least two multiplexers can be selected based on register array size, division method, and other factors to implement group selection of the register array.

[0063] For example, if the search window size is 48*32, the register array includes 48*32 registers, a total of 1536 registers. Due to the huge amount of window data, a pixel point is selected from approximately 1600 pixels. If a multiplexer is used (for example, it is not realistic to use 16 1600-to-1 Muxes) to directly search the register array, the multiplexer used has too many inputs, which will cause great difficulties in layout and routing. The embodiment of the present application provides a register array group search structure that can effectively reduce the input of the multiplexer, thereby simplifying the layout and routing of the multiplexer and effectively reducing the difficulty of hardware design. The group search structure is illustrated below.

[0064] In some embodiments, the size of the register array may be larger than the size of the search window, and the register array may cache search window data of the same size or a smaller size.

[0065] Exemplarily, in some embodiments, according to the motion trajectories of multiple pixels in the current frame, at least two groups of selection modules are controlled to perform group selection on the register array and output pixel data of multiple reference points in parallel; wherein different groups of selection modules are provided with different types of multiplexers;

[0066] The i-th group selection module includes multiple i-th type multiplexers, and the i-th type multiplexers are used to select an output from multiple inputs, and i is an integer greater than 1; the register array is divided into multiple register sub-arrays, and when i is 1, the multiple inputs of the i-th type multiplexers are pixel data of reference points of multiple registers in the register sub-array; when i is greater than 1, the multiple inputs of the i-th type multiplexers are pixel data of output reference points of multiple i-1 type multiplexers.

[0067] At least two groups of selection modules are cascaded together in grouped order. The outputs of multiple multiplexers in the first group of adjacent selection modules serve as multiple inputs to the multiplexers in the second group. The inputs of the first group of multiplexers are the pixel data of the reference points of the register, and the last group of multiplexers outputs the pixel data of the multiple reference points found in parallel. This grouped search structure can effectively reduce the input of the multiplexers, simplify the layout and wiring of the multiplexers, and effectively reduce the difficulty of hardware design.

[0068] In some embodiments, the method further includes: determining the number of multiplexers in each group of selection modules according to the size and parallelism of the register array.

[0069] Exemplarily, the motion trajectory includes a first offset in the X direction and a second offset in the Y direction; determining the coordinates of the reference point based on the first offset, the second offset and the coordinates of the pixel point; determining the selection paths of at least two selection modules based on the coordinates of the reference point; and obtaining pixel data of the reference point based on the selection path.

[0070] Exemplarily, the at least two groups of selection modules are 4 groups of selection modules, wherein the first group of selection modules includes a 16-to-1 multiplexer, the second group of selection modules includes a 64-to-1 multiplexer, the third group of selection modules includes a 48-to-1 multiplexer, and the fourth group of selection modules includes a 32-to-1 multiplexer.

[0071] For example, the register array includes 48*32 registers, and the parallelism of the register array is 16. The fourth group of selection modules includes 16 32-to-1 muxes, implementing 32-to-16 selection; the third group of selection modules includes 16 48-to-1 muxes, implementing 48-to-16 selection; the second group of selection modules includes 16 64-to-1 muxes, implementing 64-to-16 selection; and the first group of selection modules includes 4 16-to-1 muxes, implementing 16-to-4 selection.

[0072] Figures 9 to 11 This is a schematic diagram of the register group search structure in the embodiment of the present application. The 48*32 register array is divided into 96 4*4 register sub-arrays, and the register array is grouped and searched using 4 groups of selection modules as an example. Figure 9 As shown in the figure, in the innermost layer, 4 pixels are selected as output for every 16 pixels, that is, the 16-select-4 operation of the 4*4 register subarray is realized through the first group of selection modules, and the specification of the multiplexer (mux) is 16to1. Figure 10As shown in the figure, in the first middle layer, each first group selection module outputs 4 pixels, and the second group selection module selects 16 pixels from the 64 pixels output by the 16 first group selection modules. That is, the second group selection module includes 16 64to1muxes to implement a 64-to-16 operation. The second group selection module can be understood as selecting 16 pixels from the 16*16 register subarray for output. Figure 11 As shown, in the second middle layer, every three second-group selection modules form a group to output a total of 48 pixels. The third-group selection module selects 16 of these 48 pixels for output, so 16 48-to-1 muxes are used to implement the 48-to-16 operation. The third-group selection module can be understood as selecting 16 output points from the 48*16 register subarray. In the outermost layer, every two third-group selection modules form a group to output 32 pixels. Ultimately, 16 pixels are selected from these 32 pixels as output, so 16 32-to-1 muxes are used. The fourth-group selection module can be understood as selecting 16 output points from the entire register array. Therefore, the solution splits the 1600-to-1 multiplexer into four types: 64-to-1, 48-to-1, 32-to-1, and 16-to-1 multiplexers. This facilitates hardware implementation and avoids layout and routing issues.

[0073] Step 203: Determine pixel data of the plurality of pixel points according to the pixel data of the reference point.

[0074] Exemplarily, the motion trajectory includes a first offset in the X direction and a second offset in the Y direction; and determining the pixel data of the multiple pixels based on the pixel data of the reference point includes: when the first offset and the second offset are integers, outputting the pixel data of the reference point as the pixel data of the pixel point of the current frame; when the first offset or the second offset is a decimal, obtaining the pixel data of the multiple reference points, performing an interpolation operation on the pixel data of the multiple reference points, and obtaining the pixel data of the pixel point of the current frame. For example, when the offset is a decimal, obtaining the pixel data of four surrounding reference points, performing an interpolation operation on the pixel data of the four reference points, and obtaining the pixel data of the pixel point of the current frame.

[0075] That is to say, each pixel point corresponds to one or more reference points. When it corresponds to one reference point, the pixel data of the reference point is directly used as the pixel data of the pixel point of the current frame; when it corresponds to multiple reference points, the pixel data of multiple reference points are interpolated to obtain the pixel data of the pixel point of the current frame.

[0076] Figures 12 to 14 This is a second flow chart of the image processing method in the embodiment of the present application, according to Figure 12 The moving trajectory of the search window shown in the figure is used to load the search window data. Figure 13It is the motion trajectory of each pixel in the current frame generated by the optical flow estimation network, including two-channel data, namely the offset of the pixel in the X and Y directions. Figure 14 As shown, in order to obtain 16 pixel points in the current frame at the same time, it is necessary to find a reference point in the search window according to the motion trajectory of each pixel point and read the pixel data of the reference point, so as to obtain the pixel data of the 16 pixel points based on the pixel data of the reference point.

[0077] Using this technical solution, search window data is loaded from the memory cell array into the register array. Multiple pixel reference points can be read from the register array in parallel within a few clock cycles, or even a single clock cycle. This resolves the access conflict issue that exists when directly reading from the memory cells and improves reference point reading efficiency. Grouped searches of the register array using different types of multiplexers can simplify the layout and routing of the multiplexers.

[0078] In order to implement the method of the embodiment of the present application, the embodiment of the present application also provides an image processing chip, such as Figure 15 As shown, the image processing chip 150 includes: a storage unit array 1501, a register array 1502 and a processor 1503.

[0079] The storage unit array 1501 is used to store pixel data of the reference frame;

[0080] The register array 1502 is used to cache search window data;

[0081] The processor 1503 is used to load the search window data from the storage unit array 1501 and cache the search window data to the register array 1502; read the pixel data of the reference points corresponding to the multiple pixel points in parallel from the register array 1502 according to the motion trajectories of the multiple pixel points in the current frame; and determine the pixel data of the pixel point based on the pixel data of the reference point.

[0082] In some embodiments, as Figure 4 As shown, the storage unit array 1501 includes: M storage units distributed horizontally, each storage unit includes N storage spaces distributed vertically; M and N are both integers greater than 1;

[0083] The processor is configured to obtain loading information of a search window, wherein the loading information includes a chip select signal and address information of the storage unit; select at least two adjacent storage units from the M storage units based on the chip select signal; and load a row of valid data of the search window from the storage space of the at least two storage units based on the address information, and cache the data in the register array until all valid data of the search window is loaded.

[0084] In some embodiments, the processor 1503 uses the upper left corner of the reference frame as the starting position of the search window, controls the movement of the search window from top to bottom and from left to right, and updates the loading information of the search window.

[0085] In some embodiments, the processor 1503 is configured to, after the search window moves from top to bottom to a first boundary position, load a row of valid data from the next column of the search window at a top starting position in each clock cycle and cache the data in the register array until the search window moves to a second boundary position;

[0086] The reference frame includes valid data and boundary filling data, the first boundary position is the lower boundary of the valid data, and the second boundary position is the lower boundary of the boundary filling data.

[0087] In some embodiments, the chip further comprises at least two groups of selection modules.

[0088] The i-th group of selection modules includes a plurality of i-th type multiplexers, wherein the i-th type multiplexers are used to select an output from a plurality of inputs, where i is an integer greater than 1;

[0089] The register array is divided into a plurality of register sub-arrays, and when i is 1, the plurality of inputs of the i-th type of multiplexer are pixel data of reference points of a plurality of registers in the register sub-array;

[0090] When i is greater than 1, the multiple inputs of the i-th type multiplexer are the pixel data of the output reference points of multiple i-1-th type multiplexers;

[0091] The processor is used to control at least two groups of selection modules to group and select the register array according to the motion trajectories of multiple pixel points in the current frame, and output the pixel data of multiple reference points in parallel.

[0092] That is, at least two groups of selection modules are cascaded together in grouped order. The outputs of the multiple multiplexers in the first group of adjacent selection modules serve as the multiple inputs of the multiplexers in the second group. The input of the first group of multiplexers is the pixel data of the reference point of the register, and the last group of multiplexers outputs the pixel data of the multiple reference points found in parallel. This grouped search structure of the present application can effectively reduce the input of the multiplexers, simplify the layout and wiring of the multiplexers, and effectively reduce the difficulty of hardware design.

[0093] In some embodiments, the at least two groups of selection modules are four groups of selection modules, wherein:

[0094] The first group of selection modules includes a 16-to-1 multiplexer,

[0095] The second group of selection modules includes a 64-to-1 multiplexer,

[0096] The third group of selection modules includes a 48-to-1 multiplexer,

[0097] The fourth group selection module includes a 32-to-1 multiplexer.

[0098] Optionally, the image processing chip 150 may further include an input interface 1504. The input interface 1504 is used to communicate with other devices or chips, and specifically, to obtain information or data sent by other devices or chips.

[0099] Optionally, the image processing chip 150 may further include an output interface 1505. The output interface 1505 is used to communicate with other devices or chips, and specifically, information or data may be output to other devices or chips.

[0100] Optionally, the image processing chip 150 may be applied to the electronic device in the embodiment of the present application.

[0101] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0102] In practical applications, the various components in the image processing chip 150 are coupled together via a bus system. It is understood that the bus system is used to enable communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus.

[0103] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understood that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0104] The above-mentioned storage unit can be a volatile memory (volatile memory), such as a static random-access memory (SRAM), or a non-volatile memory (non-volatile memory), such as a random-access memory (RAM) or a flash memory.

[0105] Using this image processing chip, search window data is loaded from the memory cell array into the register array. Reference points for multiple pixels can be read from the register array in parallel within a few clock cycles or even a single clock cycle. This resolves the access conflict issue that exists when directly reading from the memory cells and improves reference point reading efficiency. Grouped searches of the register array are implemented using different types of multiplexers, simplifying the layout and routing of the multiplexers.

[0106] The embodiments of the present application also provide electronic devices, which may include mobile phones, tablet computers, laptop computers, handheld computers, personal digital assistants (PDAs), portable media players (PMPs), wearable devices, cameras, smart cars, etc.

[0107] like Figure 16 As shown, the electronic device 160 includes:

[0108] Image acquisition device 1601, used to acquire pixel data of an image;

[0109] The image processing chip 1602 includes any one of the image processing chips in the embodiments of the present application, and is used to execute the steps of the image processing method in the embodiments of the present application.

[0110] Of course, in actual application, as shown in Figure 16, the various components in the electronic device 160 are coupled together through the bus system 1603. It can be understood that the bus system 1603 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1603 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 16 Various buses are labeled as bus system 1603.

[0111] In an exemplary embodiment, the present application also provides a computer-readable storage medium, such as a memory including a computer program, which can be executed by a processor of an image processing chip to complete the steps of the aforementioned method.

[0112] An embodiment of the present application also provides a computer program product, including computer program instructions.

[0113] Optionally, the computer program product can be applied to the image processing chip in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the image processing chip in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0114] The embodiment of the present application also provides a computer program.

[0115] Optionally, the computer program can be applied to the image processing chip in the embodiment of the present application. When the computer program runs on a computer, the computer executes the corresponding processes implemented by the image processing chip in the various methods of the embodiment of the present application. For the sake of brevity, they are not described here.

[0116] It should be understood that the terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. The expressions "having", "can have", "including" and "comprising", or "can include" and "can include" in this application can be used to indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations or components), but do not exclude the presence of additional features.

[0117] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various types of information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another and are not necessarily used to describe a specific order or precedence. For example, first information could be referred to as second information, and similarly, second information could be referred to as first information without departing from the scope of the present invention.

[0118] The technical solutions described in the embodiments of this application can be combined arbitrarily unless there is any conflict.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed methods, devices and equipment can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0120] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0121] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0122] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Loading search window data from a storage unit array and caching the search window data in a register array; wherein the storage unit array stores pixel data of a reference frame; According to the motion trajectories of multiple pixel points in the current frame, the register array is searched in groups, and pixel data of reference points corresponding to the multiple pixel points are output in parallel; The pixel data of the plurality of pixel points are determined according to the pixel data of the reference point.

2. The method according to claim 1, characterized in that The storage unit array includes: M storage units distributed laterally, each storage unit including N storage spaces distributed longitudinally; M and N are both integers greater than 1; The step of loading the search window data from the storage unit array and caching the search window data in the register array includes: Acquire loading information of the search window; wherein the loading information includes a chip select signal and address information of the storage unit; Selecting at least two adjacent storage cells from the M laterally distributed storage cells based on the chip select signal; Based on the address information, a row of valid data of the search window is loaded from the storage space of the at least two storage units and cached in the register array until all valid data of the search window are loaded.

3. The method according to claim 2, characterized in that The method further comprises: The upper left corner of the reference frame is used as the starting position of the search window, and the movement of the search window is controlled in a moving order from top to bottom and from left to right, and the loading information of the search window is updated.

4. The method according to claim 3, characterized in that The method further comprises: After the search window moves from top to bottom to a first boundary position, a row of valid data of the next column of the search window at a top starting position is loaded in each clock cycle and cached in the register array until the search window moves to a second boundary position; The reference frame includes valid data and boundary filling data, the first boundary position is the lower boundary of the valid data, and the second boundary position is the lower boundary of the boundary filling data.

5. The method according to claim 1, wherein The step of performing group search on the register array according to the motion trajectories of the plurality of pixel points in the current frame and outputting pixel data of reference points corresponding to the plurality of pixel points in parallel includes: According to the motion trajectories of the multiple pixel points in the current frame, control at least two groups of selection modules to perform group selection on the register array and output pixel data of the multiple reference points in parallel; wherein different groups of selection modules are provided with different types of multiplexers; The i-th group of selection modules includes a plurality of i-th type multiplexers, wherein the i-th type multiplexers are used to select an output from a plurality of inputs, where i is an integer greater than or equal to 1; The register array is divided into a plurality of register sub-arrays, and when i is 1, the plurality of inputs of the i-th type of multiplexer are pixel data of reference points of a plurality of registers in the register sub-array; When i is greater than 1, the multiple inputs of the i-th type multiplexer are pixel data of the output reference points of multiple i-1-th type multiplexers.

6. The method according to claim 5, characterized in that The at least two groups of selection modules are 4 groups of selection modules, wherein: The first group of selection modules includes a 16-to-1 multiplexer, The second group of selection modules includes a 64-to-1 multiplexer, The third group of selection modules includes a 48-to-1 multiplexer, The fourth group selection module includes a 32-to-1 multiplexer.

7. An image processing chip, characterized in that: include: Memory cell array, register array and processor, The storage unit array is used to store pixel data of the reference frame; The register array is used to cache search window data; The processor is used to load search window data from the storage unit array and cache the search window data to the register array; read pixel data of reference points corresponding to the multiple pixel points in parallel from the register array according to the motion trajectories of the multiple pixel points in the current frame; and determine the pixel data of the pixel point based on the pixel data of the reference point.

8. The chip according to claim 7, characterized in that The storage unit array includes: M storage units distributed laterally, each storage unit including N storage spaces distributed longitudinally; M and N are both integers greater than 1; The processor is configured to obtain loading information of a search window, wherein the loading information includes a chip select signal and address information of the storage unit; select at least two adjacent storage units from the M laterally distributed storage units based on the chip select signal; and load a row of valid data of the search window from the storage space of the at least two storage units based on the address information, and cache the data in the register array until all valid data of the search window is loaded.

9. The chip according to claim 7, characterized in that The chip also includes at least two groups of selection modules, The i-th group of selection modules includes a plurality of i-th type multiplexers, wherein the i-th type multiplexers are used to select an output from a plurality of inputs, where i is an integer greater than or equal to 1; The register array is divided into a plurality of register sub-arrays, and when i is 1, the plurality of inputs of the i-th type of multiplexer are pixel data of reference points of a plurality of registers in the register sub-array; When i is greater than 1, the multiple inputs of the i-th type multiplexer are the pixel data of the output reference points of multiple i-1-th type multiplexers; The processor is used to control at least two groups of selection modules to perform group selection on the register array according to the motion trajectories of multiple pixel points in the current frame, and output pixel data of multiple reference points in parallel.

10. An electronic device, characterized in that: The electronic device comprises: an image acquisition device and the image processing chip according to any one of claims 7 to 9.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images