Pixel value mapping and encoding / decoding method, electronic device, and storage medium
Patent Information
- Application Number
- HK42023081008
- Authority / Receiving Office
- HK · HK
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2023-10-18
- Publication Date
- 2026-09-04
- Estimated Expiration
- 2043-03-19
AI Technical Summary
The current pixel value mapping process is cumbersome and inefficient, and there is a need to improve the mapping efficiency.
Parallel lookup operations are employed, using NEON technology to perform parallel lookups on multiple pixel values. The maximum and minimum value registers are used to determine a sub-table of pixel value ranges, and parallel lookup operations are performed on the sub-table, reducing instruction complexity and lookup time.
This greatly reduces the instruction complexity of the table lookup process, improves the efficiency of table lookup, and reduces the computational complexity and time of the mapping process.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing, and more particularly to a pixel value mapping and encoding / decoding method, an electronic device, and a storage medium. Background Technology
[0002] In the field of video coding technology, in order to improve encoding and decoding efficiency, an adaptive loop shaper (lumamapping with chroma scaling, LMCS) has been added. It mainly consists of two parts: luma mapping (LM) based on an adaptive piecewise linear model and chroma scaling (CS) based on luma.
[0003] The basic idea of brightness mapping based on an adaptive piecewise linear model is to better utilize the allowed range of brightness values at a specified bit depth. In the specific implementation process, brightness mapping first generates a forward mapping table FwdLUT and a reverse mapping table InvLUT based on the forward mapping function FwdMap and the reverse mapping function InvMap. Forward and reverse mappings are then completed by querying these tables.
[0004] However, the mapping process requires a table lookup operation for each pixel, which is cumbersome and inefficient.
[0005] In view of this, the urgent technical problem that existing technologies need to solve is how to improve mapping efficiency. Summary of the Invention
[0006] In view of this, embodiments of this application provide a pixel value mapping scheme to at least partially solve the above-mentioned problems.
[0007] According to a first aspect of the embodiments of this application, a pixel value mapping method is provided, comprising: determining an image to be processed and a mapping table corresponding to the image to be processed; performing a parallel lookup operation on the pixel values of multiple pixels in the image to be processed in the mapping table to obtain a mapping value corresponding to the pixel; and determining a mapped image corresponding to the image to be processed based on the queried mapping value.
[0008] Optionally, in any embodiment of this application, the method further includes: comparing the pixel values of the plurality of pixels to determine the maximum pixel value and the minimum pixel value; the step of performing a parallel lookup operation on the pixel values of the plurality of pixels in the image to be processed in the mapping table to obtain the mapping value corresponding to the pixel includes: determining a first sub-table from the mapping table corresponding to the range of pixel values of the maximum pixel value and the minimum pixel value; performing a parallel lookup operation on the pixel values of the plurality of pixels in the first sub-table to obtain the mapping value corresponding to the pixel.
[0009] Optionally, in any embodiment of this application, the number of the plurality of pixels is m times the parallelism of the parallel lookup operation, where m is a positive integer. The step of performing a parallel lookup operation based on NEON on the pixel values of the plurality of pixels in the first sub-table to obtain the mapping value corresponding to the pixel includes: dividing the pixel values of the plurality of pixels into m groups of pixel values to be queried according to the parallelism; and performing a parallel lookup operation based on NEON on the pixel values to be queried in the first sub-table in units of groups to obtain the mapping value corresponding to the pixel.
[0010] Optionally, in any embodiment of this application, the step of performing a parallel NEON-based lookup operation on the pixel value to be queried in the first sub-table in groups to obtain the mapping value corresponding to the pixel includes: dividing the first sub-table into multiple second sub-tables; performing a parallel NEON-based lookup operation on the pixel value to be queried in the multiple second sub-tables in groups to obtain multiple sets of query results corresponding to each group of pixel values to be queried; and merging the multiple sets of query results to obtain the mapping value corresponding to each group of pixel values to be queried.
[0011] Optionally, in any embodiment of this application, the operation of comparing the pixel values of the plurality of pixels to determine the maximum and minimum pixel values includes: initializing a maximum value register and a minimum value register, wherein the number of pixel values stored in the maximum value register or the minimum value register is i, and the number of the plurality of pixels is n times the number of pixel values i, where n is a positive integer; dividing the pixel values of the plurality of pixels into n groups of pixel values to be compared, each group of pixel values to be compared including i pixel values; performing a bitwise comparison between each group of pixel values to be compared and the pixel values stored in the maximum value register, and updating the i pixel values stored in the maximum value register according to the maximum value in the comparison result; and performing a bitwise comparison between each group of pixel values to be compared and the pixel values stored in the minimum value register, and updating the i pixel values stored in the minimum value register according to the minimum value in the comparison result; determining the maximum pixel value from the i pixel values stored in the maximum value register, and determining the minimum pixel value from the i pixel values stored in the minimum value register.
[0012] Optionally, in any embodiment of this application, determining the first sub-table corresponding to the range of pixel values between the maximum pixel value and the minimum pixel value from the mapping table includes: determining the number of pixel values in the first sub-table based on the difference between the maximum pixel value and the minimum pixel value; and determining the first sub-table from the mapping table starting from the minimum pixel value and based on the number of pixel values.
[0013] Optionally, in any embodiment of this application, performing a parallel lookup operation on the pixel values of the plurality of pixels of the image to be processed in the mapping table to obtain the mapping value corresponding to the pixel includes: segmenting the mapping table to obtain a plurality of third sub-tables; performing a parallel lookup operation on the pixel values of the plurality of pixels in the plurality of third sub-tables respectively to obtain a plurality of query results; and merging the plurality of query results to obtain the mapping value corresponding to the plurality of pixels respectively.
[0014] According to a second aspect of the embodiments of this application, an encoding method is provided, comprising: determining an image block to be processed; converting the luminance component of the image block to be processed from the original domain to the mapped domain through a forward mapping operation, and converting the luminance component of the image block to be processed from the mapped domain to the original domain through a reverse mapping operation, wherein at least one of the forward mapping operation and the reverse mapping operation is implemented by the method described above; and encoding the mapped luminance component to obtain a bitstream.
[0015] According to a third aspect of the embodiments of this application, a decoding method is provided, comprising: decoding a bitstream to obtain an image block to be processed; converting the luminance component of the image block to be processed from the original domain to the mapped domain through a forward mapping operation, and converting the luminance component of the image block to be processed from the mapped domain to the original domain through a reverse mapping operation, wherein at least one of the forward mapping operation and the reverse mapping operation is implemented by the method described above; and decoding based on the mapped luminance component to obtain a video frame.
[0016] According to a fourth aspect of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described above.
[0017] According to a fifth aspect of the embodiments of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method described above.
[0018] According to the solution provided in the embodiments of this application, the image to be processed and the mapping table corresponding to the image to be processed are determined; and a parallel lookup operation is performed on the pixel values of multiple pixels in the image to be processed in the mapping table to obtain the mapping value corresponding to the pixel; finally, the mapped image corresponding to the image to be processed is determined based on the queried mapping value. Since the parallel lookup operation supports executing the parallel lookup operation in response to a single instruction, the mapping value corresponding to the pixel values of multiple pixels can be retrieved in the mapping table at one time through the parallel lookup operation, which greatly reduces the instruction complexity of the lookup process and improves the lookup efficiency. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.
[0020] Figure 1A This is a flowchart illustrating the steps of a pixel value mapping method according to an embodiment of this application;
[0021] Figure 1B for Figure 1A The illustration shows a usage scenario diagram of the embodiment shown;
[0022] Figure 2A This is a flowchart illustrating the steps of another pixel value mapping method according to an embodiment of this application;
[0023] Figure 2B This is a schematic diagram of a mapping table according to an embodiment of this application;
[0024] Figure 2C This is a schematic diagram of an initialized register according to an embodiment of this application;
[0025] Figure 2D This is a schematic diagram illustrating the determination of a first sub-table from a mapping table of size 256 according to an embodiment of this application;
[0026] Figure 3 This is a flowchart illustrating the steps of another pixel value mapping method according to an embodiment of this application;
[0027] Figure 4 This is a flowchart illustrating the steps of an encoding method according to an embodiment of this application;
[0028] Figure 5 This is a flowchart illustrating the steps of a decoding method according to an embodiment of this application;
[0029] Figure 6This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0030] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.
[0031] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.
[0032] The technical terms used in this application will be explained below.
[0033] Single Instruction Multiple Data (SIMD) computation is a parallel processing technique where a single instruction can process multiple data points in parallel, significantly increasing speed compared to processing one data point per instruction. Many programs need to process large datasets, often stored in fewer than 32 bits. Examples include 8-bit pixel data in video, graphics, and image processing. In such cases, there are likely many simple, repetitive computations with little control code; SIMD excels at providing higher performance for these types of programs.
[0034] NEON: NEON is an ARM technology based on the SIMD concept. NEON combines 64-bit and 128-bit SIMD instruction sets to provide 128-bit wide vector operations.
[0035] The adaptive loop shaper (luma mapping with chroma scaling, LMCS) mainly consists of two parts: luma mapping (LM) based on an adaptive piecewise linear model, and chroma scaling (CS) based on luma residuals.
[0036] Luma mapping (LM): By making full use of the range of luminance values and photoelectric conversion characteristics, luminance mapping can improve the coding efficiency of video. It is applied at the pixel level.
[0037] Generally, when performing luminance mapping, a forward mapping table FwdLUT and a reverse mapping table InvLUT are first generated based on the forward mapping function FwdMap and the reverse mapping function InvMap. Forward and reverse mapping are then completed by querying these tables. The size and code value range of the forward and reverse mapping tables are determined by the pixel bit depth. That is, if the bit depth is 1, the table size is 1 << bit depth, where "<<" indicates shifting 1 to the left by the corresponding number of bits. For example, if the bit depth is 8 bits, then shifting 1 to the left by 8 bits corresponds to 256, so the size of the forward and reverse mapping tables is 256, and the code value range is [0, 255].
[0038] Luminance mapping can be achieved through the following process:
[0039] Based on the bit depth, a forward mapping table FwdLUT and a reverse mapping table InvLUT of size (1 << bit depth) are established using a piecewise linear model.
[0040] This paper implements a pixel-by-pixel lookup operation in C language to complete forward and reverse mapping.
[0041] The specific formula for forward mapping is:
[0042] P pwd =FwdLUT[P org Among them, P org ∈[0, 1 << bitdepth-1]
[0043] The specific formula for reverse mapping is:
[0044] P org =InvLUT[P pwd Among them, P pwd ∈[0, 1 << bitdepth-1]
[0045] Among them, P org P represents the pixel value in the original domain. pwd `bitdepth` represents the pixel value in the mapped domain, and `bitdepth` represents the pixel bit depth. Forward mapping is used to convert the original domain to the mapped domain, where operations such as intra-frame prediction and inverse transform can be performed. Backward mapping is used to convert the mapped domain back to the original domain, where operations such as loop filtering and motion compensation are performed, and the final reconstructed image can be obtained in the original domain.
[0046] However, while the above scheme transforms the pixel-by-pixel function mapping calculation into a lookup process by pre-establishing a forward mapping table FwdLUT and a reverse mapping table InvLUT, effectively improving the efficiency of numerical remapping through pixel-by-pixel lookup operations, the complexity of pixel-by-pixel lookup remains high due to the relatively weak performance of the processor.
[0047] In view of this, in order to reduce the complexity of the mapping process, this application provides another pixel value mapping scheme, which will be described below through specific embodiments.
[0048] Reference Figure 1A , Figure 1A This is a flowchart illustrating the steps of a pixel value mapping method according to an embodiment of this application. Specifically, the pixel value mapping method provided in this embodiment includes the following steps:
[0049] S101. Determine the image to be processed and the mapping table corresponding to the image to be processed.
[0050] In this embodiment, the image to be processed can be any image, such as a captured image or an image after processing the actual captured image, such as the brightness component or color component of the image; the image to be processed can be a complete image or a region of a complete image, such as an image block divided during video encoding and decoding, all of which are within the protection scope of this application.
[0051] The mapping table corresponding to the image to be processed is a mapping table determined according to a preset mapping relationship. The mapping table is used to store the pixel values before and after mapping. Specifically, the mapping table can be the aforementioned forward mapping table FwdLUT or reverse mapping table InvLUT, or it can be a color mapping table, etc., all of which are within the protection scope of this application.
[0052] For example, if the image to be processed undergoes brightness mapping, the mapping table can store the brightness values before and after mapping. Taking an 8-bit bit depth as an example, the pixel range corresponding to the brightness before mapping is 0-255, and the pixel range corresponding to the brightness after mapping is also 0-255. The mapping table determined according to the preset mapping relationship includes an index of 0-255 and the index value corresponding to each index. The index corresponds to the pixel value before mapping, and the index value corresponds to the pixel value after mapping. The specific mapping relationship can be a linear mapping relationship or a non-linear mapping relationship, etc. This embodiment does not limit this. For example, a non-linear linear mapping relationship maps pixel values of 0-100 to 0-59, maps pixel values of 101-200 to 60-200, and leaves pixel values greater than 200 unchanged.
[0053] S102. Perform a parallel lookup operation on the pixel values of multiple pixels in the image to be processed in the mapping table to obtain the mapping value corresponding to the pixel.
[0054] In this embodiment, the parallel table lookup operation can be a parallel table lookup operation based on NEON, or other parallel table lookup operations, such as a parallel table lookup operation based on C language. This embodiment does not limit this. Through the parallel table lookup operation, multiple table lookup operations can be implemented with one instruction, which reduces instruction complexity and table lookup time.
[0055] In this embodiment, parallel table lookup operations based on NEON are preferred. NEON is an ARM technology based on the SIMD concept. NEON combines 64-bit and 128-bit SIMD instruction sets to provide 128-bit wide vector operations. ARM's NEON technology is essentially an advanced Single Instruction Multiple Data (SIMD) architecture extension.
[0056] NEON can be implemented by calling an instruction set, namely NEON Assembler, or by calling a function, namely NEON Intrinsics. This embodiment does not limit the implementation.
[0057] NEON enables the CPU to process multiple data in parallel within a single instruction cycle, saving algorithm processing time. In this embodiment, a parallel lookup operation based on NEON is performed on the pixel values of multiple pixels in the image to be processed in the mapping table. Specifically, the pixel values of multiple pixels can be written into a register, and the CPU performs the same and parallel lookup operation on the pixel values of multiple pixels based on the NEON instruction. That is, a parallel lookup operation based on NEON is performed, so that multiple lookup operations can be implemented with a single instruction, reducing instruction complexity and lookup time.
[0058] S103. Determine the mapped image corresponding to the image to be processed based on the queried mapping value.
[0059] In this embodiment, the pixel value of the pixel can be replaced with the queried mapping value to obtain the mapped image corresponding to the image to be processed. This embodiment does not limit this.
[0060] Figure 1B This is a schematic diagram illustrating a pixel value mapping method according to an embodiment of this application. For the image to be processed, which can be 32*32 pixels in size, it can be divided into four 16*16 image blocks. For each image block, a parallel lookup operation can be performed on the pixel values of 16 pixels in a mapping table of 0-255, with a parallelism of 16. Specifically, refer to... Figure 1BAs shown, a parallel table lookup operation can be performed on the pixel values corresponding to a row of pixels in an image patch each time. Repeating the lookup 16 times will yield the mapping value corresponding to a pixel in an image patch; repeating it 16*4 times will yield the mapping value corresponding to a pixel in an image patch.
[0061] If one pixel is queried at a time, it would require 32*32 lookup operations to find the mapping value corresponding to the image to be processed. However, with the solution provided in this embodiment, the mapping value corresponding to the image to be processed can be found in 16*4 parallel lookup operations, which greatly improves the efficiency of the lookup and reduces the complexity.
[0062] The solution provided in this embodiment determines the image to be processed and the mapping table corresponding to the image to be processed; performs a parallel lookup operation on the pixel values of multiple pixels in the image to be processed in the mapping table to obtain the mapping value corresponding to the pixel; and finally determines the mapped image corresponding to the image to be processed based on the retrieved mapping value. Since the parallel lookup operation supports executing the parallel lookup operation in response to a single instruction, the mapping value corresponding to the pixel values of multiple pixels can be retrieved in the mapping table at once through the parallel lookup operation, which greatly reduces the instruction complexity of the lookup process and improves the lookup efficiency.
[0063] In comparison, LMCS luminance mapping uses LCUs (Large Coding Units) for mapping. An LCU is an image block of equal size divided from the image. Taking a 128x128 LCU as an example, forward mapping with one LCU requires 16384 (128x128) lookups. The decoding process requires both forward and reverse mapping, thus requiring 32768 (16384x2) lookups. However, using the scheme provided in this embodiment, if the parallelism of the parallel lookup operation is 16, forward mapping with one LCU only requires 1024 (128x128 / 16) lookups. Reverse and forward mapping, and both forward and reverse mapping, require only 2048 (1024x2) lookups. The number of lookup operations is only 1 / 16 of the original, and the number of instructions is also reduced accordingly, decreasing the instruction complexity of the lookup process and greatly improving its processing speed and efficiency.
[0064] Reference Figure 2A , Figure 2A This is a flowchart illustrating the steps of another pixel value mapping method according to an embodiment of this application. Specifically, the pixel value mapping method provided in this embodiment includes the following steps:
[0065] S201. Determine the image to be processed and the mapping table corresponding to the image to be processed.
[0066] S202, perform a comparison operation on the pixel values of the plurality of pixels to determine the maximum pixel value and the minimum pixel value.
[0067] In this embodiment, the image to be processed can be divided into several image blocks of basic size, such as 16*16 image blocks or 8*8 image blocks. Then, the pixel values of multiple pixels in the image block are compared to determine the maximum and minimum pixel values.
[0068] In this embodiment, the comparison operation can be implemented using C language or based on NEON, both of which are within the scope of protection of this application.
[0069] To save computational complexity, it is preferable to implement the comparison operation based on NEON. For example, based on the parallelism of NEON, multiple pixel values can be compared simultaneously to obtain multiple candidate maximum or minimum values as the result of one comparison. Then, the multiple candidate maximum or minimum values are compared a second time to obtain the maximum and minimum pixel values.
[0070] When determining the maximum and minimum pixel values using NEON assembly, step S202 specifically includes: initializing a maximum value register and a minimum value register, wherein the maximum value register or the minimum value register stores i pixel values, and the number of multiple pixels is n times the number of pixel values i, where n is a positive integer; dividing the pixel values of the multiple pixels into n groups of pixel values to be compared, each group of pixel values to be compared including i pixel values; performing a bitwise comparison between each group of pixel values to be compared and the pixel values stored in the maximum value register, and updating the i pixel values stored in the maximum value register according to the maximum value in the comparison result; and performing a bitwise comparison between each group of pixel values to be compared and the pixel values stored in the minimum value register, and updating the i pixel values stored in the minimum value register according to the minimum value in the comparison result; determining the maximum pixel value from the i pixel values stored in the maximum value register, and determining the minimum pixel value from the i pixel values stored in the minimum value register.
[0071] For example, the maximum value register and the minimum value register can be initialized using the vdupq_n_u8 instruction. When the bit depth is 8, the minimum value register can be initialized to 255 or to the first set of pixel values to be compared. Similarly, the maximum value register can be initialized to 0 or to the first set of pixel values to be compared.
[0072] If the minimum value register is initialized to 255 and the maximum value register is initialized to 0, then the first set of pixel values to be compared can be compared with 255, and the pixel value stored in the minimum value register can be updated according to the minimum value in the comparison result. Similarly, the first set of pixel values to be compared can be compared with 0, and the pixel value stored in the maximum value register can be updated according to the maximum value in the comparison result.
[0073] The maximum value register or minimum value register stores i pixel values. Each group of pixel values to be compared includes i pixel values. Taking the maximum value register as an example, the bit comparison is as follows: the first pixel value in the maximum value register is compared with the first pixel value in the group to be compared, the second pixel value in the maximum value register is compared with the second pixel value in the group to be compared, and so on.
[0074] After the last set of pixel values to be compared is completed, the i pixel values in the maximum value register can be compared to obtain the maximum pixel value, and the i pixel values in the minimum value register can be compared to obtain the minimum pixel value.
[0075] If the number of pixels is not n times the number of pixel values i, but is between n times and n+1 times, then the multiple pixels can be divided into n+1 groups of pixel values to be queried, where each of the n groups includes i pixels, and the number of pixels in the (n+1)th group is less than i. Each group of pixel values to be compared is compared bit-by-bit with the pixel values stored in the maximum value register, and the i pixel values stored in the maximum value register are updated according to the maximum value in the comparison result. Furthermore, each group of pixel values to be compared is compared bit-by-bit with the pixel values stored in the minimum value register, and the i pixel values stored in the minimum value register are updated according to the minimum value in the comparison result. The maximum pixel value is then determined from the pixels in the (n+1)th group and the i pixel values stored in the maximum value register, and the minimum pixel value is determined from the pixels in the (n+1)th group and the i pixel values stored in the minimum value register.
[0076] S203. Determine a first sub-table from the mapping table corresponding to the range of pixel values for the maximum pixel value and the minimum pixel value;
[0077] In this embodiment, after determining the maximum and minimum pixel values, the number of pixel values val_range can be calculated using the following formula:
[0078] val_range = max_val - min_val
[0079] Where max_val is the maximum pixel value and min_val is the minimum pixel value.
[0080] Once the number of pixel values is determined, the first sub-table corresponding to the range of pixel values for the maximum and minimum pixel values can be determined from the mapping table. That is, the starting index value of the first sub-table is the same as the minimum pixel value, and the ending index value of the first sub-table is the same as the maximum pixel value.
[0081] The solution provided in this embodiment pre-calculates the maximum and minimum pixel values, then determines the first sub-table corresponding to the pixel value range based on the maximum and minimum pixel values, and performs a lookup operation on the first sub-table in subsequent steps. This can narrow down the range of the mapping table that needs to be queried, effectively reducing the number of times the neon assembly instructions are used.
[0082] Optionally, in this embodiment, step S203 may include: determining the number of pixel values in the first sub-table based on the difference between the maximum pixel value and the minimum pixel value; and determining the first sub-table from the mapping table based on the number of pixel values, starting from the minimum pixel value.
[0083] In this embodiment, the contents of the mapping table can be loaded into the register starting from the minimum pixel value, and the loading stops after the number of indexes corresponding to the number of pixel values loaded into the first sub-table, thereby loading the first sub-table into the register. Subsequently, parallel table lookup operations are performed based on the register using NEON assembly instructions.
[0084] See Figure 2B The diagram illustrates a mapping table. When loading the mapping table, the process can begin at the position marked with `min_val` in the diagram, loading the mapping table into the register until `val_range` values have been loaded. The mapping table marked in dark in the diagram is the first sub-table loaded into the register. Of course, Figure 2B This is merely an example of a mapping table and is not intended to limit the scope of protection for mapping tables.
[0085] S204. Perform a parallel lookup operation on the pixel values of the plurality of pixels in the first sub-table to obtain the mapping value corresponding to the pixel.
[0086] Specifically, in this embodiment, if the number of multiple pixels is m times the parallelism of the parallel lookup operation, where m is a positive integer, step S204 may include: dividing the pixel values of the multiple pixels into m groups of pixel values to be queried according to the parallelism; performing a parallel lookup operation based on NEON on the pixel values to be queried in the first sub-table in units of groups to obtain the mapping value corresponding to the pixel.
[0087] If the number of pixels is not m times the parallelism of the parallel lookup operation, but is between m times and m+1 times, then the pixel values of the multiple pixels can be divided into m+1 groups of pixel values to be queried according to the parallelism. The number of pixels in each of the m groups is the same as the parallelism, and the number of pixels in the m+1 group is less than the parallelism. A parallel lookup operation based on NEON is performed on the pixel values to be queried in the first sub-table on a group-by-group basis to obtain the mapping value corresponding to the pixel.
[0088] Parallelism can be defined as the number of pixel values that can be queried in response to a single instruction. In this embodiment, the parallelism can be 2, 3, 16, etc., all of which are within the scope of protection of this application.
[0089] In this embodiment, multiple pixel values can be divided into m groups, and parallel table lookup operations can be performed in the first sub-table on a group-by-group basis, which further improves the query efficiency.
[0090] Optionally, in this embodiment, the step of performing a parallel NEON-based lookup operation on the pixel value to be queried in the first sub-table in groups to obtain the mapping value corresponding to the pixel includes: segmenting the first sub-table to obtain multiple second sub-tables; performing a parallel NEON-based lookup operation on the pixel value to be queried in the multiple second sub-tables in groups to obtain multiple sets of query results corresponding to each group of pixel values to be queried; and merging the multiple sets of query results to obtain the mapping value corresponding to each group of pixel values to be queried.
[0091] In this embodiment, since there is an upper limit to the size of the sub-table loaded by the register (the largest table that can be loaded by the register is 64), if the first sub-table is larger than the loading limit, it can be segmented into multiple second sub-tables. Segmentation ensures that the size of the second sub-tables does not exceed the loading limit. Then, parallel NEON-based lookup operations can be performed on the pixel values to be queried in multiple second sub-tables in groups, obtaining multiple sets of query results corresponding to each group of pixel values to be queried. These multiple sets of query results are then merged to obtain the mapping values corresponding to each group of pixel values to be queried. Since one pixel value to be queried corresponds to one mapping value, the mapping value corresponding to one pixel value to be queried exists within one of the multiple sets of query results. By merging the multiple sets of query results, the query result corresponding to that group of pixel values to be queried can be obtained.
[0092] S205. Determine the mapped image corresponding to the image to be processed based on the queried mapping value.
[0093] The following example illustrates the solution provided in this embodiment through a specific scenario.
[0094] For example, let's take a 16*16 pixel array with a pixel value of 16 as an example.
[0095] The 16*16 pixel values can be divided into 16 groups, with 16 pixel values in each group, and the maximum value register and the minimum value register store 16 pixel values.
[0096] The maximum value register and the minimum value register can be initialized. The initialized registers can be used as follows: Figure 2C As shown.
[0097] Then, the first group of 16 pixel values to be compared can be loaded into the register variable v_src using the vld1q_u8 instruction. The register variable v_src can be compared bitwise with the minimum value register using the vminq_u8 instruction, and the minimum value obtained from the comparison can be updated into the minimum value register. The register variable v_src can be compared bitwise with the maximum value register using the vmaxq_u8 instruction, and the maximum value obtained from the comparison can be updated into the maximum value register.
[0098] After the comparison is complete, the 16 pixel values to be compared in the second group can be loaded into the register variable v_src using the vld1q_u8 instruction, and the comparison can continue. This operation is repeated until the 16th group of pixels to be compared is completed.
[0099] Then, the maximum pixel value max_val can be found among the 8 pixel values stored in the maximum value register using the vpmax_u8 instruction, and the minimum pixel value min_val can be found among the 8 pixel values stored in the minimum value register using the vpmin_u8 instruction.
[0100] In this embodiment, after determining the maximum and minimum pixel values, the number of pixel values val_range can be calculated using the following formula:
[0101] val_range = max_val - min_val
[0102] Where max_val is the maximum pixel value and min_val is the minimum pixel value.
[0103] Then, based on the size of val_range, it can be determined whether to divide the first sub-table into multiple second sub-tables.
[0104] See Figure 2D If the bit depth is 8, the size of the mapping table is 256. The register can load a table of up to 64. Therefore, we can determine the relationship between the number of pixel values val_range and multiples of 64.
[0105] If the number of pixel values val_range ≤ 64, then the first sub-table does not need to be segmented.
[0106] Starting from min_val, you can load the number of pixel values val_range or 64 indices into the register to obtain the first sub-table tab0, and then perform parallel table lookup operations based on the loaded first sub-table tab0.
[0107] Specifically, a set of 16 pixel values to be queried can be loaded into the register variable xSrc using the vld1q_u8 instruction. Then, the difference between xSrc and min_val can be calculated using the vsubq_u8 instruction to align the pixel values to be queried with the first sub-table. Next, the 16 pixel values in Src can be used for parallel lookup operations in the first sub-table tab0 using the vqtbl4q_u8 instruction to obtain the mapping values corresponding to the 16 pixel values, which can be stored in the 128-bit register variable xMapVal0. Finally, the value of xMapVal0 is stored back into memory using the vst1q_u8 instruction. This process of loading a set of 16 pixel values to be queried into the register variable xSrc using the vld1q_u8 instruction is repeated until multiple sets of pixel values have been queried.
[0108] If the number of pixel values val_range ≤ 64*2, then the first sub-table needs to be divided into two second sub-tables.
[0109] Starting from min_val and min_val+64, 64 indices can be loaded into registers to obtain two second sub-tables, namely tab1 and tab2, and parallel table lookup operations can be performed based on the loaded second sub-tables tab1 and tab2.
[0110] Specifically, a set of 16 pixel values to be queried can be loaded into the register variable xSrc using the vld1q_u8 instruction. For the first second sub-table, the difference between xSrc and min_val is calculated using the vsubq_u8 instruction to align the pixel values to be queried with the index of the first second sub-table tab1; and the difference between xSrc and min_val+64 is calculated using the vsubq_u8 instruction to align the pixel values to be queried with the index of the second second sub-table tab2.
[0111] The `vqtbl4q_u8` command performs a parallel lookup operation on the first sub-table `tab1` of the `Src` to obtain the mapping values `xVal_1` corresponding to the 16 pixel values in the first sub-table, which is the first set of query results. Similarly, the mapping values `xVal_2` corresponding to the 16 pixel values in the second sub-table can be obtained, which is the second set of query results.
[0112] Then, by performing a bitwise OR operation or a maximum value operation, xVal_1 and xVal_2 can be merged to obtain the mapping value dst_0 of the 16 pixel values in xSrc in the first sub-table of size 128.
[0113] Finally, the value of dst_0 is stored back into memory using the vst1q_u8 instruction.
[0114] Then repeat the process of loading a set of 16 pixel values to be queried into the register variable xSrc using the vld1q_u8 instruction until multiple sets of pixel values to be queried have been queried.
[0115] If the number of pixel values val_range ≤ 64*3, then the first sub-table needs to be divided into 3 second sub-tables.
[0116] If the number of pixel values val_range > 64*3, then the first sub-table needs to be divided into 4 second sub-tables.
[0117] The calculation process for obtaining 3 or 4 second sub-tables by segmentation is similar to the process for obtaining 2 second sub-tables by segmentation described above, and will not be repeated here.
[0118] Since the size of the mapping table in this embodiment is 256, or 64*4, this embodiment judges up to 64*3 and does not continue to judge 64*4. If the size of the mapping table is other values, those skilled in the art can set other multiple judgment conditions according to their needs, which is also within the protection scope of this application.
[0119] The solution provided in this embodiment pre-calculates the maximum and minimum pixel values, then determines a first sub-table corresponding to the range of pixel values corresponding to the maximum and minimum pixel values, and performs a lookup operation on the first sub-table in subsequent steps. This can narrow down the range of the mapping table to be queried, effectively reducing the number of times neon assembly instructions are used. Furthermore, during the parallel lookup operation, the first sub-table can be segmented into multiple second sub-tables based on the difference between the maximum and minimum pixel values, and parallel lookup operations can be performed based on multiple second sub-tables, further reducing the number of times neon assembly instructions are used.
[0120] Reference Figure 3 , Figure 3 This is a flowchart illustrating the steps of another pixel value mapping method according to an embodiment of this application. Specifically, the pixel value mapping method provided in this embodiment includes the following steps:
[0121] S301. Determine the image to be processed and the mapping table corresponding to the image to be processed.
[0122] S302. Divide the mapping table into segments to obtain multiple third sub-tables;
[0123] S303. Perform parallel table lookup operations on the pixel values of the multiple pixels in the multiple third sub-tables respectively to obtain multiple sets of query results;
[0124] S304. Merge the multiple sets of query results to obtain the mapping values corresponding to the multiple pixels respectively.
[0125] S305. Determine the mapped image corresponding to the image to be processed based on the queried mapping value.
[0126] In this embodiment, if the bit depth is 8, the size of the mapping table is 256. The register can load a table of up to 64. Therefore, the mapping table can be segmented to obtain 4 third sub-tables.
[0127] Starting with 0, 0+64, 0+64*2, or 0+64*3, 64 indices can be loaded into the register to obtain 4 third sub-tables, namely tab1, tab2, tab3, and tab4. Parallel table lookup operations can then be performed based on the loaded third sub-tables tab1, tab2, tab3, and tab4.
[0128] Specifically, a set of 16 pixel values to be queried can be loaded into the register variable xSrc using the vld1q_u8 instruction.
[0129] For the first third sub-table, the difference between xSrc and 0 is calculated using the vsubq_u8 command to align the query pixel value with the index of the first third sub-table tab1. For the second third sub-table, the difference between xSrc and 0+64 is calculated using the vsubq_u8 command to align the query pixel value with the index of the second third sub-table tab2. For the third third sub-table, the difference between xSrc and 0+64*2 is calculated using the vsubq_u8 command to align the query pixel value with the index of the third third sub-table tab3. For the fourth third sub-table, the difference between xSrc and 0+64*4 is calculated using the vsubq_u8 command to align the query pixel value with the index of the fourth third sub-table tab3.
[0130] The `vqtbl4q_u8` command performs a parallel lookup operation on the 16 pixel values in the first third sub-table `tab1` of `Src`, obtaining the mapping values `xVal_1` corresponding to the 16 pixel values in the first third sub-table, which is the first set of query results. Similarly, the mapping values `xVal_2` corresponding to the 16 pixel values in the second third sub-table, which is the second set of query results, as well as the third set of query results `xVal_3` and the fourth set of query results `xVal_4`, can be obtained.
[0131] Then, by performing a bitwise OR operation or a maximum value operation, xVal_1, xVal_2, xVal_3, and xVal_4 can be merged to obtain the mapping value dst_0 of the 16 pixel values in xSrc in a mapping table of size 256.
[0132] Finally, the value of dst_0 is stored back into memory using the vst1q_u8 instruction.
[0133] Then repeat the process of loading a set of 16 pixel values to be queried into the register variable xSrc using the vld1q_u8 instruction until multiple sets of pixel values to be queried have been queried.
[0134] The solution provided in this embodiment can segment the mapping table to obtain multiple third sub-tables during the parallel table lookup operation, and perform parallel table lookup operations based on multiple third sub-tables, further reducing the number of times neon assembly instructions are used.
[0135] Reference Figure 4 , Figure 4 This is a flowchart illustrating the steps of an encoding method according to an embodiment of this application. Specifically, the method provided in this embodiment includes the following steps:
[0136] S401. Determine the image block to be processed;
[0137] S402. The luminance component of the image block to be processed is converted from the original domain to the mapped domain through a forward mapping operation, and the luminance component of the image block to be processed is converted from the mapped domain to the original domain through a reverse mapping operation, wherein at least one of the forward mapping operation and the reverse mapping operation is implemented by the method described in the above embodiments.
[0138] S403. Encode the mapped luminance components to obtain the bitstream.
[0139] The specific implementation of each step in this embodiment can be found in the corresponding descriptions of the steps and units in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.
[0140] According to the scheme provided in the embodiments of this application, the image to be processed and the mapping table corresponding to the image to be processed are determined; and a parallel lookup operation is performed on the pixel values of multiple pixels in the image to be processed in the mapping table to obtain the mapping value corresponding to the pixel; finally, the mapped image corresponding to the image to be processed is determined based on the queried mapping value. Since the parallel lookup operation supports executing the parallel lookup operation in response to a single instruction, the mapping value corresponding to the pixel values of multiple pixels can be retrieved in the mapping table at one time through the parallel lookup operation, which greatly reduces the instruction complexity of the lookup process and improves the lookup efficiency of the encoding and decoding process.
[0141] Reference Figure 5 , Figure 5 This is a flowchart illustrating the steps of a decoding method according to an embodiment of this application. Specifically, the method provided in this embodiment includes the following steps:
[0142] S501, Decode the bitstream to obtain the image block to be processed;
[0143] S502. The luminance component of the image block to be processed is converted from the original domain to the mapped domain through a forward mapping operation, and the luminance component of the image block to be processed is converted from the mapped domain to the original domain through a reverse mapping operation, wherein at least one of the forward mapping operation and the reverse mapping operation is implemented by the method described in the above embodiments.
[0144] S503. Decode the mapped luminance components to obtain video frames.
[0145] The specific implementation of each step in this embodiment can be found in the corresponding descriptions of the steps and units in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.
[0146] According to the scheme provided in the embodiments of this application, the image to be processed and the mapping table corresponding to the image to be processed are determined; and a parallel lookup operation is performed on the pixel values of multiple pixels in the image to be processed in the mapping table to obtain the mapping value corresponding to the pixel; finally, the mapped image corresponding to the image to be processed is determined based on the queried mapping value. Since the parallel lookup operation supports executing the parallel lookup operation in response to a single instruction, the mapping value corresponding to the pixel values of multiple pixels can be retrieved in the mapping table at one time through the parallel lookup operation, which greatly reduces the instruction complexity of the lookup process and improves the lookup efficiency of the encoding and decoding process.
[0147] The pixel value mapping scheme or encoding / decoding scheme provided in any of the above embodiments of this application can be executed by the video encoding end (encoder) when encoding the video stream to improve the processing efficiency of the encoding process. It can be applied to various scenarios, such as: storage and streaming of conventional video games. Specifically, the pixel value mapping scheme provided in the embodiments of this application can be used to map each image block in the video frame (e.g., forward mapping or reverse mapping), and a video bitstream can be generated based on the mapping result for storage and transmission in video streaming services or other similar applications. Another example is low-latency scenarios such as video conferencing and live video streaming. Specifically, video acquisition devices can be used to acquire conference video data, and then the method provided in the embodiments of this application can be used to map each image block in the video frame (e.g., forward mapping or reverse mapping), and a video bitstream can be generated based on the mapping result and sent to the conference terminal. The conference terminal can then process the video bitstream. Decoding yields the corresponding conference video image. Specifically, during decoding, the conference terminal can also perform mapping processing on the luminance components of the image blocks (e.g., forward mapping or reverse mapping). Similarly, in virtual reality scenarios, the method provided in this application embodiment can be used to map each image block in the video frame (e.g., forward mapping or reverse mapping), generate a video stream based on the mapping result, and send it to virtual reality-related devices (such as VR glasses). The VR device can then decode the video stream to obtain the corresponding video image, and implement corresponding VR functions based on the video image. In this case, the VR device can also perform mapping processing on the luminance components of the image blocks (e.g., forward mapping or reverse mapping).
[0148] Reference Figure 6 This document illustrates a schematic diagram of an electronic device according to an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.
[0149] like Figure 6 As shown, the electronic device may include: a processor 602, a communications interface 604, a memory 606, and a communications bus 608.
[0150] in:
[0151] The processor 602, communication interface 604, and memory 606 communicate with each other via communication bus 608.
[0152] Communication interface 604 is used for communication with other electronic devices or servers.
[0153] The processor 602 is used to execute program 610, specifically to execute the relevant steps in the above-described pixel value mapping method or encoding / decoding method embodiments.
[0154] Specifically, program 610 may include program code that includes computer operation instructions.
[0155] The processor 602 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.
[0156] Memory 606 is used to store program 610. Memory 606 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0157] Program 610 may include multiple computer instructions. Specifically, program 610 may use multiple computer instructions to cause processor 602 to execute the pixel value mapping method or encoding / decoding method described in any of the foregoing multiple method embodiments.
[0158] The specific implementation of each step in program 610 can be found in the corresponding steps and units described in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.
[0159] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk.
[0160] This application also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the methods in the above-described multiple method embodiments.
[0161] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.
[0162] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.
[0163] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.
[0164] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.
[0165] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.
Claims
1. A pixel value mapping method, comprising: determining a to-be-processed image and a mapping table corresponding to the to-be-processed image; performing comparison operations on pixel values of a plurality of pixels to determine a maximum pixel value and a minimum pixel value; performing parallel table lookup operations on the pixel values of the plurality of pixels in the to-be-processed image in the mapping table to obtain mapping values corresponding to the pixels, comprising: determining a first sub-table corresponding to a pixel value range of the maximum pixel value and the minimum pixel value from the mapping table; and performing parallel table lookup operations on the pixel values of the plurality of pixels in the first sub-table to obtain the mapping values corresponding to the pixels; determining a mapped image corresponding to the to-be-processed image according to the queried mapping values.
2. The method of claim 1, wherein, The number of the plurality of pixels is m times the parallelism degree of the parallel table lookup operations, m being a positive integer, and the performing parallel table lookup operations on the pixel values of the plurality of pixels in the first sub-table to obtain the mapping values corresponding to the pixels comprises: dividing the pixel values of the plurality of pixels into m groups of to-be-queried pixel values according to the parallelism degree; performing NEON-based parallel table lookup operations on the to-be-queried pixel values in the first sub-table in groups to obtain the mapping values corresponding to the pixels.
3. The method of claim 2, wherein, The performing NEON-based parallel table lookup operations on the to-be-queried pixel values in the first sub-table in groups to obtain the mapping values corresponding to the pixels comprises: segmenting the first sub-table to obtain a plurality of second sub-tables; performing NEON-based parallel table lookup operations on the to-be-queried pixel values in the plurality of second sub-tables in groups to obtain a plurality of groups of query results corresponding to the to-be-queried pixel values in each group; merging the plurality of groups of query results to obtain the mapping values corresponding to the to-be-queried pixel values in each group respectively.
4. The method of claim 1, wherein, The performing comparison operations on the pixel values of the plurality of pixels to determine the maximum pixel value and the minimum pixel value comprises: initializing a maximum value register and a minimum value register, the number of pixel values stored in the maximum value register or the minimum value register being i, the number of the plurality of pixels being n times the number of pixel values i, n being a positive integer; dividing the pixel values of the plurality of pixels into n groups of to-be-compared pixel values, each group of to-be-compared pixel values including i pixel values; performing bitwise comparison between each group of to-be-compared pixel values and the pixel values stored in the maximum value register, updating the i pixel values stored in the maximum value register according to the maximum value in the comparison result; and performing bitwise comparison between each group of to-be-compared pixel values and the pixel values stored in the minimum value register, updating the i pixel values stored in the minimum value register according to the minimum value in the comparison result; determining the maximum pixel value from the i pixel values stored in the maximum value register, and determining the minimum pixel value from the i pixel values stored in the minimum value register.
5. The method of claim 1, wherein, The determining a first sub-table corresponding to a pixel value range of the maximum pixel value and the minimum pixel value from the mapping table comprises: determining the number of pixel values of the first sub-table according to the difference between the maximum pixel value and the minimum pixel value. Determine the first sub-table from the mapping table according to the number of pixel values and the minimum pixel value.
6. The method of claim 1, wherein, The parallel table lookup operation is performed on the pixel values of the plurality of pixels in the mapping table to obtain the mapping values corresponding to the pixels, including: Segmenting the mapping table to obtain a plurality of third sub-tables; Performing parallel table lookup operations on the pixel values of the plurality of pixels in the plurality of third sub-tables to obtain a plurality of groups of query results; Merging the plurality of groups of query results to obtain the mapping values corresponding to the plurality of pixels respectively.
7. An encoding method, wherein, Including: Determine a to-be-processed image block; Convert the luminance component of the to-be-processed image block from the original domain to the mapping domain through a forward mapping operation, and convert the luminance component of the to-be-processed image block from the mapping domain to the original domain through a reverse mapping operation, wherein at least one of the forward mapping operation and the reverse mapping operation is implemented by the method of any one of claims 1-6; Encode according to the mapped luminance component to obtain a bitstream.
8. A decoding method, wherein, Including: Decode a bitstream to obtain a to-be-processed image block; Convert the luminance component of the to-be-processed image block from the original domain to the mapping domain through a forward mapping operation, and convert the luminance component of the to-be-processed image block from the mapping domain to the original domain through a reverse mapping operation, wherein at least one of the forward mapping operation and the reverse mapping operation is implemented by the method of any one of claims 1-6; Decode according to the mapped luminance component to obtain a video frame.
9. An electronic device comprising: A processor, a memory, a communication interface, and a communication bus, the processor, the memory, and the communication interface complete communication with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform the operation corresponding to the method in any one of claims 1-8.
10. A computer storage medium having a computer program stored thereon, the program being executed by a processor to implement the method of any one of claims 1-8.
