Data processing method, electronic device, storage medium, chip system, and computer program product

Abstract

The data processing method, the electronic device, the storage medium, the chip system and the computer program product provided by the embodiments of the present application are related to the technical field of terminals. The method can be applied to a processor. The method comprises: processing first matrix data based on an instruction set, implementing at least one of the following processes: component separation, visual geometry transformation, component sampling or color space transformation, the instruction set comprising instructions corresponding to at least one of the following: access, matrix outer product accumulation or matrix multiplication vector, the access comprising at least one of the following: horizontal access, vertical access or scattered access, the instruction corresponding to the access being used for at least one of the above processes, the instruction corresponding to the matrix outer product accumulation being used for component sampling, and the instruction corresponding to the matrix multiplication vector being used for color space conversion, thereby improving processing efficiency and reducing device power consumption. The instruction set is determined according to an SME instruction set and / or an SVE instruction set.

Description

Technical Field

[0001] This application relates to the field of terminals, and more particularly to data processing methods, electronic devices, storage media, chip systems, and computer program products. Background Technology

[0002] With the continuous development of electronic devices, their functions are becoming increasingly diverse. For example, the visual data processing functions provided by electronic devices can be used to achieve at least one of the following: video encoding / decoding, image editing, video editing, image browsing, video browsing, or image preview. Visual data can include image data corresponding to an image or video frame data corresponding to a video frame.

[0003] Among related technologies, visual data processing has relatively low processing efficiency. Summary of the Invention

[0004] This application provides a data processing method, an electronic device, a storage medium, a chip system, and a computer program product, which are applied in the field of terminal technology and can improve the processing efficiency of visual data processing and reduce device power consumption.

[0005] Firstly, embodiments of this application propose a data processing method. This method can be applied to a processor. The method may include: processing first matrix data corresponding to first visual data based on an instruction set; and outputting the processing result.

[0006] According to embodiments of this application, the instruction set may be determined based on the Extensible Single Instruction Multiple Data (SMILD) instruction set. The SMILD instruction set may include at least one of the following: the SME instruction set or the SVE instruction set. Therefore, the instruction set may be determined based on the SME instruction set and / or the SVE instruction set; that is, the instruction set may be an instruction set associated with the SME instruction set and / or an instruction set associated with the SVE instruction set.

[0007] For example, the instruction set related to the SME instruction set can be the SME instruction set itself. Optionally, the instruction set related to the SME instruction set can be obtained by adjusting the SME instruction set. Optionally, the instruction set related to the SME instruction set can include the SME instruction set and a first newly added instruction set. The instruction set related to the SVE instruction set can be the SVE instruction set itself. Optionally, the instruction set related to the SVE instruction set can be obtained by adjusting the SVE instruction set. Optionally, the instruction set related to the SVE instruction set can include the SVE instruction set and a second newly added instruction set.

[0008] The first matrix data may include at least one first element value corresponding to each of the multiple first components. The first component may be a color component of a first color space.

[0009] The instruction set can be used to control the processor to execute at least one of the following processes: storage format conversion of the first visual data, visual geometric transformation of the first visual data, sampling of the first component, or color space conversion of the first visual data. As one implementation, the storage format conversion of the first visual data can separate at least one first element value corresponding to each of the multiple first components in the first matrix data. Separating at least one first element value corresponding to each of the multiple first components in the first matrix data can be understood as rearranging at least one first element value corresponding to each of the multiple first components in the first matrix data.

[0010] The instruction set may include instructions corresponding to at least one of the following: access, matrix external accumulation, or matrix multiplication vector. Access may include at least one of the following: horizontal access (e.g., horizontal loading or horizontal storage), vertical access (e.g., vertical loading or vertical storage), or distributed access (e.g., distributed loading or distributed storage). Instructions corresponding to access may be used for at least one of storage format conversion, visual geometric transformation, sampling of a first component, or color space conversion. For example, instructions corresponding to at least one of horizontal loading, vertical loading, or distributed loading may be used to load first matrix data from memory or other matrix data obtained from the first matrix data. Instructions corresponding to matrix external accumulation may be used for sampling of the first component. For example, instructions corresponding to matrix external accumulation may be used to obtain the matrix multiplication result between two matrix data. The two matrix data may be determined based on the first matrix data. As one implementation, the two matrix data may include first sub-matrix data corresponding to the first component and matrix data for expanding the data dimension of the first sub-matrix data. Instructions corresponding to matrix multiplication vector may be used for color space conversion. As one implementation, the instruction corresponding to matrix-vector multiplication can be used to obtain the matrix-vector multiplication result between the ninth matrix data and the first vector data corresponding to each of the multiple vectors. The ninth matrix data can be determined based on color space conversion relationships. The first vector data can be determined based on the first matrix data. For example, the first vector data corresponding to a pixel can be determined based on the first element values ​​corresponding to the multiple first components of the first matrix data that correspond to the pixel.

[0011] According to embodiments of this application, by utilizing instructions corresponding to at least one of horizontal access, vertical access, or distributed access, at least one of the following can be achieved: separating the first element values ​​corresponding to each of the multiple first components (i.e., storage format conversion), visual geometric transformation of the first visual data, sampling of the first component, or color space conversion of the first visual data. Since horizontal access based on the instruction set associated with the SME instruction set can access row data, vertical access can access column data, and distributed access based on the instruction set associated with the SVE instruction set can access multiple element values ​​(e.g., first element values) simultaneously, access efficiency can be improved, thereby improving processing efficiency. Since matrix outer product is essentially the multiplication of two vector data to generate matrix data, the calculation of each element value (e.g., first element value) is independent, and matrix outer product accumulation based on the instruction set associated with the SME instruction set can support the accumulation of multiple element values, and can also complete matrix outer product and accumulation in a single cycle, parallel computing is achieved. Therefore, by utilizing instructions corresponding to matrix outer product accumulation, sampling of the first component is achieved, improving the processing efficiency of the component sampling process. Since matrix-vector multiplication based on the instruction set related to the SME instruction set provides direct multiplication and addition operations between matrix data and vector data, the color space conversion of the first visual data is realized by using the instruction corresponding to matrix-vector multiplication, thus improving the processing efficiency of the color space conversion process.

[0012] Furthermore, since matrix multiplication is converted into matrix-external accumulation, it is equivalent to loading the column data of one matrix and the row data of the other matrix only once. Therefore, the number of element value loadings is reduced, which in turn reduces the number of memory accesses, thereby saving memory bandwidth. The improved processing efficiency reduces device power consumption.

[0013] In one possible implementation, where the instruction set is used to control the processor to perform storage format conversion of the first visual data, processing the first matrix data corresponding to the first visual data based on the instruction set may include: loading the first matrix data from memory into a two-dimensional register based on the instruction set to obtain the second matrix data stored in the two-dimensional register.

[0014] The first matrix data contains alternating first element values ​​corresponding to at least two first components. The second matrix data includes second sub-matrix data corresponding to multiple first components, meaning that the first element values ​​corresponding to multiple first components in the second matrix data are arranged continuously, or that is, the first element values ​​corresponding to each first component in the second matrix data are arranged continuously.

[0015] Because the first matrix data contains alternating first element values ​​corresponding to at least two first components, and the second matrix data contains consecutive first element values ​​corresponding to multiple first components, the multiple first components can be separated based on the instruction set associated with the SME instruction set and / or the instruction set associated with the SVE instruction set. This means that each of the multiple first components has its own corresponding storage area. Since the SME instruction set and / or the SVE instruction set can accelerate matrix operations, the processing efficiency of component separation is improved.

[0016] In one possible implementation, the instruction set may include at least one first instruction corresponding to each of the plurality of first components. The plurality of first components may include first component A and a plurality of first components B. Optionally, all of the plurality of first components may be first component B. First component A may be a separated first component, i.e., first component A may be a first component with an independent storage area. At least one first element value stored in the storage area corresponding to first component A may all be the first element value corresponding to first component A. First component B may be a non-separated first component, i.e., first component B may be a first component without an independent storage area. The plurality of first element values ​​stored in the storage area corresponding to first component B may include at least one first element value corresponding to first component B and at least one first element value corresponding to other first components B.

[0017] Based on the instruction set, loading the first matrix data from memory into a two-dimensional register to obtain the second matrix data stored in the two-dimensional register may include: for any first component among a plurality of first components, based on at least one first instruction corresponding to the first component, loading at least one first object corresponding to the first component in the first matrix data from memory into the two-dimensional register to obtain the second sub-matrix data corresponding to the first component stored in the two-dimensional register.

[0018] According to an embodiment of this application, when the first component is a first component A, the first object corresponding to the first component A may include first row data, first column data, or first matrix block data corresponding to the first component A.

[0019] When the first component is a first component B, the first object may include the first element value corresponding to the first component B in the second row of data, the first element value corresponding to the first component B in the second column of data, or the first element value corresponding to the first component B in the second matrix block of data. The second row of data may be row data corresponding to multiple first components B. The second column of data may be column data corresponding to multiple first components B. The second matrix block of data may be matrix block data corresponding to multiple first components B.

[0020] Since the first component A is a separated first component, when the first component is the first component A and the first object is the first row of data, based on at least one first instruction corresponding to the first component A, at least one first row of data corresponding to the first component A in the first matrix data is loaded from memory into a two-dimensional register. At least one first row of data can be executed in parallel. Therefore, the first instruction achieves horizontal loading of at least one first row of data and allows for parallel loading of at least one first row of data, thereby improving processing efficiency. When the first component is the first component A and the first object is the first column of data, based on at least one first instruction corresponding to the first component A, at least one first column of data corresponding to the first category A in the first matrix data is loaded from memory into a two-dimensional register. At least one first column of data can be executed in parallel. Therefore, the first instruction achieves vertical loading of at least one first column of data and allows for parallel loading of at least one first column of data, thereby improving processing efficiency. When the first component is the first component A and the first object is the first matrix block data, since the first instruction achieves horizontal loading and / or vertical loading, processing efficiency is improved.

[0021] Since the first component B is an unseparated first component, and the first object includes the first element value corresponding to the first component B in the second row of data, based on at least one first instruction corresponding to the first component B, at least one first element value corresponding to the first component B in the first matrix data is loaded from memory into a two-dimensional register. At least one second row of data can be executed in parallel. Therefore, the first instruction implements the distributed loading of at least one second row of data, and the loading of at least one second row of data can be executed in parallel, thereby improving processing efficiency. Similarly, when the first component is the first component B and the first object includes the first element value corresponding to the first component B in the second column of data, based on at least one first instruction corresponding to the first component B, at least one first element value corresponding to the first component B in the first matrix data is loaded from memory into a two-dimensional register. At least one second column of data can be executed in parallel. Therefore, the first instruction implements the distributed loading of at least one second column of data, and the loading of at least one second column of data can be executed in parallel, thereby improving processing efficiency. Finally, when the first component is the first component B and the second object includes at least one first element value corresponding to the first component B in the second matrix block data, since the first instruction implements distributed loading in the horizontal direction and / or vertical direction, processing efficiency is improved.

[0022] In one possible implementation, the two-dimensional register may include at least one first region. The first instruction may include a first identifier, at least one second identifier, and at least one third identifier. The first identifier may be used to indicate a first load operation. The second identifier may be used to indicate the first region. At least one third identifier may be used to indicate a first address corresponding to each of the at least one first objects.

[0023] Loading at least one first object corresponding to the first component from memory into a two-dimensional register based on at least one first instruction corresponding to the first component can include: determining, based on at least one third identifier, a first address corresponding to each of the at least one first object for any of the at least one first instructions corresponding to the first component; retrieving the first object corresponding to each of the at least one first address from memory based on the first identifier and the first address corresponding to each of the at least one first object; and loading the first object corresponding to each of the at least one first address into a first region corresponding to each of the at least one second identifier based on the first identifier.

[0024] At least one first instruction corresponding to the first component can be executed in parallel or sequentially.

[0025] Because at least one first instruction corresponding to the first component can be executed in parallel, the speed of loading the first matrix data from memory is accelerated, thereby improving processing efficiency. Furthermore, the first instruction can also be used to execute the parallel loading of at least one first row of data, at least one first column of data, or at least one first matrix block of data, thus improving processing efficiency.

[0026] In one implementation, the processor may further include a first register. The first register may be used to store at least one fourth identifier corresponding to the first object. Of the at least one fourth identifier, the fourth identifier corresponding to the first component has a first value, and the fourth identifiers corresponding to the other first components have second values. The first value may be used to indicate that a first load operation is performed for the first element value corresponding to the first component. The second value may be used to indicate that the first load operation is not performed for the first element value corresponding to the first component. The first instruction may further include a fifth identifier. The fifth identifier may be used to indicate the first register.

[0027] Loading a first object corresponding to at least one first address into a first region corresponding to at least one second identifier based on a first identifier may include: obtaining at least one fourth identifier corresponding to at least one first object from a first register corresponding to a fifth identifier. Based on the first identifier and the at least one fourth identifier corresponding to at least one first object, the first object corresponding to at least one first address is loaded into the first region corresponding to at least one second identifier. The fourth identifier of the first object stored in the first region is a first value.

[0028] Since the first register stores at least one fourth identifier corresponding to the first object, and the fourth identifier corresponding to the first component can be a first value while the fourth identifiers corresponding to other first components can be second values, the first value is used to indicate that a first loading operation is performed for the first element value corresponding to the first component, and the second value is used to indicate that a first loading operation is not performed for the first element value corresponding to the first component. Therefore, the first element value corresponding to the first value can be loaded without loading the first element value corresponding to the second value. Thus, distributed loading of the first element value corresponding to the first component is achieved. Since distributed loading can improve processing efficiency, improved processing efficiency is achieved based on distributed loading.

[0029] In one possible implementation, the second matrix data may include second submatrix data corresponding to each of the plurality of first components.

[0030] In the case where the instruction set is used to control the processor to perform visual geometric transformations of the first visual data, and the visual geometric transformations may include visual rotation, processing the first matrix data corresponding to the first visual data based on the instruction set may include: rotating the second sub-matrix data corresponding to each of the multiple first components obtained from the two-dimensional register by a first angle, based on the instruction set, to obtain the third sub-matrix data corresponding to each of the multiple first components stored in memory. The instruction set may be used to indicate rotation instruction sets related to the SME instruction set and / or the SVE instruction set.

[0031] Since the rotation instruction set associated with the SME instruction set and / or the SVE instruction set enables the rotation of the second submatrix data corresponding to the first component by a first angle, the SME instruction set and / or the SVE instruction set can speed up matrix operations, thus improving the processing efficiency of the visual rotation process.

[0032] In one possible implementation, based on an instruction set, the second sub-matrix data corresponding to each of the multiple first components, obtained from a two-dimensional register, is rotated by a first angle to obtain third sub-matrix data corresponding to each of the multiple first components, stored in memory. This may include:

[0033] For any first component among multiple first components, when the first angle is 90°, 180°, or 270°, the second sub-matrix data corresponding to the first component is retrieved from the two-dimensional register according to a first method, and the second sub-matrix data corresponding to the first component is stored in memory according to a second method, resulting in the third sub-matrix data corresponding to the first component stored in memory. The first and second methods can be determined based on an instruction set. The first and second methods can be used to make the first angle 90°, 180°, or 270°. And / or

[0034] When the first angle is 180°, the second sub-matrix data corresponding to the first component is obtained from the two-dimensional register in the third way, the obtained second sub-matrix data is flipped, and the flipped second sub-matrix data is stored in memory in the fourth way to obtain the third sub-matrix data corresponding to the first component stored in memory.

[0035] The third and fourth modes can be determined based on the instruction set. The third, flip, and fourth modes can be used to make the first angle 180°.

[0036] Because visual rotation is achieved through flipping, horizontal reading, and vertical reading, the processing efficiency of the visual rotation process is improved.

[0037] In one possible implementation, the second submatrix data corresponding to the first component may include multiple first element values ​​corresponding to the first component. The second submatrix data may include M rows and N columns. M and N can be integers greater than 1.

[0038] When the first angle is 90°, 180°, or 270°, the instruction set may include a second instruction and a third instruction. The third instruction may be used to indicate that the second method is to store the first element value obtained in the order of the first method.

[0039] When the first angle is 90°, the second instruction can be used to indicate that the first mode is to read in parallel or sequentially, from column 0 to column N-1, with each column reading from the first element value of row M-1 to the first element value of row 0.

[0040] When the first angle is 270°, the second instruction can be used to indicate that the first mode is to read in parallel or sequentially, from column N-1 to column 0, with each column reading from the first element value of row 0 to the first element value of row M-1.

[0041] When the first angle is 180°, the second instruction can be used to indicate that the first method is to read in parallel or sequentially, from row M-1 to row 0, with each row reading from the first element value of column N-1 to the first element value of column 0, and at least one row is read in parallel or at least one row is read sequentially.

[0042] Since the first method reads columns from 0 to N-1, with each column reading from the first element of row M-1 to the first element of row 0, and at least one column is read in parallel or sequentially, vertical reading to rotate 90° is achieved based on the first method, thus improving the processing efficiency of the visual rotation process. Furthermore, if parallel reading of multiple columns is implemented based on the first method, the processing efficiency of the visual rotation process is further improved.

[0043] Since the first method reads columns N-1 to 0, and each column reads the first element from row 0 to row M-1, it achieves vertical reading for a 270° rotation, improving the processing efficiency of the visual rotation process. Furthermore, if parallel reading of multiple columns is implemented using the first method, the processing efficiency of the visual rotation process is further improved.

[0044] Since the first method reads rows M-1 to 0, and each row reads the first element from the first element of column N-1 to the first element of column 0, it achieves horizontal reading to rotate 180°, thus improving the processing efficiency of the visual rotation process. Furthermore, if parallel reading of multiple rows is implemented based on the first method, the processing efficiency of the visual rotation process is further improved.

[0045] When the first angle is 180°, the instruction set may include a fourth instruction, a fifth instruction, and a sixth instruction. The fourth instruction may be used to indicate that the third method involves reading rows M-1 to 0, with each row reading the first element value from column 0 to column N-1, and at least one row read in parallel or at least one row read sequentially. The fifth instruction may be used to indicate that the row data of the acquired second submatrix data is vector-reversed. The sixth instruction may be used to indicate that the fourth method involves storing the reversed second submatrix data sequentially.

[0046] Since the third method reads rows M-1 to 0, with each row reading from the first element of column 0 to the first element of column N-1, and at least one row is read in parallel or sequentially, horizontal reading is achieved based on this method, improving the processing efficiency of the visual rotation process. Furthermore, if parallel reading of multiple rows is implemented based on this third method, the processing efficiency of the visual rotation process is further improved. In addition, combined with vector flipping, rapid angular rotation is achieved.

[0047] In another possible implementation, for any of the multiple first components, when the first angle is 90° or 270°, the second sub-matrix data corresponding to the first component in the two-dimensional register is matrix transposed based on the instruction set. When the first angle is 90°, the transposed second sub-matrix data is column-reversed to obtain the third sub-matrix data corresponding to the first component stored in memory. When the first angle is 270°, the transposed second sub-matrix data is row-reversed to obtain the third sub-matrix data corresponding to the first component stored in memory.

[0048] With the first angle being 180°, based on the instruction set, the row-wise inversion of the second sub-matrix data corresponding to the first component in the two-dimensional register is performed, and the column-wise inversion of the row-wise inverted second sub-matrix data is performed, to obtain the third sub-matrix data corresponding to the first component stored in memory.

[0049] With the first angle being 180°, based on the instruction set, the column-reversed data of the second sub-matrix corresponding to the first component in the two-dimensional register is performed, and the row-reversed data of the second sub-matrix is ​​performed to obtain the third sub-matrix data corresponding to the first component stored in memory.

[0050] Because rotations at different angles are achieved by matrix transpose, row inversion, and column inversion associated with the SME instruction set and / or SVE instruction set, the processing efficiency of the visual rotation process is improved.

[0051] In one possible implementation, the first matrix data may include first submatrix data corresponding to each of the plurality of first components.

[0052] In an instruction set used to control the processor to perform visual geometric transformations of the first visual data, where the visual geometric transformations may include visual rotation, processing the first matrix data corresponding to the first visual data based on the instruction set may include: rotating the first sub-matrix data corresponding to each of the multiple first components obtained from a two-dimensional register by a first angle, based on the instruction set, to obtain third sub-matrix data corresponding to each of the multiple first components stored in memory. The instruction set may be used to instruct rotation instruction sets related to the SME instruction set and / or the SVE instruction set.

[0053] Since the rotation instruction set associated with the SME instruction set and / or SVE instruction set enables the rotation of the first submatrix data corresponding to the first component by a first angle, the SME instruction set and / or SVE instruction set can speed up matrix operations, thus improving the processing efficiency of the visual rotation process.

[0054] In one possible implementation, based on an instruction set, the first sub-matrix data corresponding to each of the multiple first components, obtained from a two-dimensional register, is rotated by a first angle to obtain third sub-matrix data corresponding to each of the multiple first components, stored in memory. This may include:

[0055] For any first component among multiple first components, when the first angle is 90°, 180°, or 270°, the first sub-matrix data corresponding to the first component is retrieved from the two-dimensional register according to a first method, and the first sub-matrix data corresponding to the first component is stored in memory according to a second method, resulting in the third sub-matrix data corresponding to the first component stored in memory. The first and second methods can be determined based on an instruction set. The first and second methods can be used to make the first angle 90°, 180°, or 270°. And / or

[0056] When the first angle is 180°, the first sub-matrix data corresponding to the first component is obtained from the two-dimensional register in the third way, the obtained first sub-matrix data is flipped, and the flipped first sub-matrix data is stored in memory in the fourth way to obtain the third sub-matrix data corresponding to the first component stored in memory.

[0057] The third and fourth modes can be determined based on the instruction set. The third, flip, and fourth modes can be used to make the first angle 180°.

[0058] Because visual rotation is achieved through flipping, horizontal reading, and vertical reading, the processing efficiency of the visual rotation process is improved.

[0059] In one possible implementation, the first submatrix data corresponding to the first component may include multiple first element values ​​corresponding to the first component. The first submatrix data may include M rows and N columns. M and N can be integers greater than 1.

[0060] When the first angle is 90°, 180°, or 270°, the instruction set may include a second instruction and a third instruction. The third instruction may be used to indicate that the second method is to store the first element value obtained in the order of the first method.

[0061] When the first angle is 90°, the second instruction can be used to indicate that the first mode is to read in parallel or sequentially, from column 0 to column N-1, with each column reading from the first element value of row M-1 to the first element value of row 0.

[0062] When the first angle is 270°, the second instruction can be used to indicate that the first mode is to read in parallel or sequentially, from column N-1 to column 0, with each column reading from the first element value of row 0 to the first element value of row M-1.

[0063] When the first angle is 180°, the second instruction can be used to indicate that the first method is to read in parallel or sequentially, from row M-1 to row 0, with each row reading from the first element value of column N-1 to the first element value of column 0, and at least one row is read in parallel or at least one row is read sequentially.

[0064] Since the first method reads columns from 0 to N-1, with each column reading from the first element of row M-1 to the first element of row 0, and at least one column is read in parallel or sequentially, vertical reading to rotate 90° is achieved based on the first method, thus improving the processing efficiency of the visual rotation process. Furthermore, if parallel reading of multiple columns is implemented based on the first method, the processing efficiency of the visual rotation process is further improved.

[0065] Since the first method reads columns N-1 to 0, and each column reads the first element from row 0 to row M-1, it achieves vertical reading for a 270° rotation, improving the processing efficiency of the visual rotation process. Furthermore, if parallel reading of multiple columns is implemented using the first method, the processing efficiency of the visual rotation process is further improved.

[0066] Since the first method reads rows M-1 to 0, and each row reads the first element from the first element of column N-1 to the first element of column 0, it achieves horizontal reading to rotate 180°, thus improving the processing efficiency of the visual rotation process. Furthermore, if parallel reading of multiple rows is implemented based on the first method, the processing efficiency of the visual rotation process is further improved.

[0067] When the first angle is 180°, the instruction set may include a fourth instruction, a fifth instruction, and a sixth instruction. The fourth instruction may be used to indicate that the third method involves reading rows M-1 to 0, with each row reading the first element value from column 0 to column N-1, and at least one row read in parallel or at least one row read sequentially. The fifth instruction may be used to indicate that the row data of the acquired first submatrix data is vector-reversed. The sixth instruction may be used to indicate that the fourth method involves storing the reversed first submatrix data sequentially.

[0068] Since the third method reads rows M-1 to 0, with each row reading from the first element of column 0 to the first element of column N-1, and at least one row is read in parallel or sequentially, horizontal reading is achieved based on this method, improving the processing efficiency of the visual rotation process. Furthermore, if parallel reading of multiple rows is implemented based on this third method, the processing efficiency of the visual rotation process is further improved. In addition, combined with vector flipping, rapid angular rotation is achieved.

[0069] In another possible implementation, for any of the multiple first components, when the first angle is 90° or 270°, the first sub-matrix data corresponding to the first component in the two-dimensional register is matrix transposed based on the instruction set. When the first angle is 90°, the columns of the transposed first sub-matrix data are reversed to obtain the third sub-matrix data corresponding to the first component stored in memory. When the first angle is 270°, the rows of the transposed first sub-matrix data are reversed to obtain the third sub-matrix data corresponding to the first component stored in memory.

[0070] With the first angle being 180°, based on the instruction set, the row-wise inversion of the first sub-matrix data corresponding to the first component in the two-dimensional register is performed, and the column-wise inversion of the row-wise inverted first sub-matrix data is performed, to obtain the third sub-matrix data corresponding to the first component stored in memory.

[0071] With the first angle being 180°, based on the instruction set, the column-reversed data of the first sub-matrix corresponding to the first component in the two-dimensional register is performed, and the row-reversed data of the first sub-matrix is ​​performed to obtain the third sub-matrix data corresponding to the first component stored in memory.

[0072] Because rotations at different angles are achieved by matrix transpose, row inversion, and column inversion associated with the SME instruction set and / or SVE instruction set, the processing efficiency of the visual rotation process is improved.

[0073] In one possible implementation, when the instruction set is used to control the processor to perform sampling of the first component, processing the first matrix data corresponding to the first visual data based on the instruction set may include: in a two-dimensional register, based on the instruction set, upsampling the first component C according to at least one first element value corresponding to the first component C to obtain the third matrix data corresponding to the first component C.

[0074] Because the upsampling of the first component C is achieved based on the matrix external accumulation instruction set associated with the SME instruction set, parallel computation of entire pixels or multiple pixels is realized, leveraging the hardware acceleration advantage of SME's matrix multiplication and saving memory bandwidth. Thus, while improving system performance, device power consumption is reduced.

[0075] Optionally, in the two-dimensional register, based on the instruction set, the first component C is downsampled according to at least one first element value corresponding to the first component C to obtain the third matrix data corresponding to the first component C.

[0076] The first component C can be the first component among multiple first components.

[0077] In one possible implementation, the instruction set may include a seventh instruction. The second matrix data may include second submatrix data corresponding to the first component C. Optionally, the first matrix data may include first submatrix data corresponding to the first component C.

[0078] When the instruction set is used to control the processor to perform sampling of the first component, processing the first matrix data corresponding to the first visual data based on the instruction set may include:

[0079] In the processor's two-dimensional registers, based on the seventh instruction, the third matrix data corresponding to the first component C is obtained according to the second sub-matrix data, the fourth matrix data, and the fifth matrix data corresponding to the first component C. Optionally, the third matrix data corresponding to the first component C is obtained according to the first sub-matrix data, the fourth matrix data, and the fifth matrix data corresponding to the first component C.

[0080] Optionally, the fourth and fifth matrix data can be used to expand the second sub-matrix data corresponding to the first component C by a factor of R in the first direction and / or by a factor of S in the second direction. The fourth and fifth matrix data can be determined based on the second sub-matrix data corresponding to the first component C.

[0081] Optionally, the fourth and fifth matrix data can be used to expand the second submatrix data corresponding to the first component C by a factor of R in the first direction and / or by a factor of S in the second direction.

[0082] The first direction can be horizontal and the second direction can be vertical. Optionally, the first direction can be vertical and the second direction can be horizontal. R and S can be greater than or equal to 1.

[0083] The seventh instruction can be used to instruct the external matrix accumulation between the data of the second submatrix and the data of the fourth matrix to be performed first, followed by the external matrix accumulation between the data of the sixth matrix and the data of the fifth matrix. Optionally, the seventh instruction can be used to instruct the external matrix accumulation between the data of the first submatrix and the data of the fourth matrix to be performed first, followed by the external matrix accumulation between the data of the sixth matrix and the data of the fifth matrix.

[0084] The data in the sixth matrix can be the sum of the outer matrix data between the second and fourth submatrix data. Optionally, the data in the sixth matrix can be the sum of the outer matrix data between the first and fourth submatrix data.

[0085] By accumulating matrix outer layers in the first and second directions, the number of first element values ​​corresponding to the first component C is increased. Since the matrix outer product is essentially the multiplication of two vector data (e.g., in the case where the first direction is horizontal and the second direction is vertical, the two vector data can include column vector data from the second sub-matrix and row vector data from the transposed fourth matrix, or the two vector data can include column vector data from the fifth matrix and row vector data from the sixth matrix; in the case where the first direction is vertical and the second direction is horizontal, the two vector data can include column vector data from the fourth matrix and row vector data from the second sub-matrix, or the two vector data can include column vector data from the sixth matrix and row vector data from the transposed fifth matrix) to generate matrix data (e.g., the sixth or third matrix), the calculation of each element value (e.g., the first element value) is independent. Furthermore, the matrix outer product accumulation based on the SME instruction set can support the accumulation of multiple element values ​​and can complete both the matrix outer product and accumulation in a single cycle. Therefore, parallel computation is achieved, thereby improving the processing efficiency of the component sampling process. Furthermore, since matrix multiplication is converted into matrix-external accumulation, it is equivalent to loading the column data of one matrix and the row data of the other matrix only once. This reduces the number of element loading operations, thereby reducing memory accesses and saving memory bandwidth. The improved processing efficiency reduces device power consumption. Because SMEs are relatively large, parallel processing of multiple matrix sub-blocks further improves the efficiency of the component sampling process.

[0086] In one possible implementation, the first component C can be a first component among a plurality of first components satisfying the following condition: a first number is less than a second number. The first number can be the number of first element values ​​in the second matrix data corresponding to the first component C. Optionally, the first number can be the number of first element values ​​in the first matrix data corresponding to the first component C. The second number can be the number of pixels included in the first visual data. When upsampling the first component C, a third number can be equal to the second number. The third number can be the number of element values ​​in the third matrix data corresponding to the first component C. The element values ​​can be first element values ​​or element values ​​obtained through upsampling. The instruction set can be used to indicate the matrix external accumulation instruction set associated with the SME instruction set.

[0087] Optionally, the first component C can be any of a plurality of first components. In this case, the third number can be equal to the predetermined number. The predetermined number can be configured according to actual business needs and is not limited here.

[0088] In one implementation, the fourth matrix data can be the interpolation matrix data of the second sub-matrix data in the first direction. Optionally, the fourth matrix can be the interpolation matrix data of the first sub-matrix data in the first direction. The fourth matrix data can be used to expand the data dimension of the second sub-matrix data or the first sub-matrix data by a factor of R in the first direction.

[0089] The fifth matrix data can be the interpolation matrix data of the second sub-matrix data in the second direction. Optionally, the fifth matrix data can be the interpolation matrix data of the first sub-matrix data in the second direction. The fifth matrix data can be used to expand the data dimension of the second or first sub-matrix data by a factor of S in the second direction.

[0090] In one possible implementation, R = S = 1 when the ratio between the first number and the second number is 1:4, or when the first condition is met.

[0091] In one possible implementation, R = 1 or S = 1, where the ratio between the first number and the second number is 1:2, or in the case of the second condition.

[0092] The first condition can be used to indicate that the data dimensions of the second submatrix data are doubled in both the first and second directions. Optionally, the first condition can be used to indicate that the data dimensions of the first submatrix data are doubled in both the first and second directions.

[0093] The second condition can be used to indicate whether the data dimension of the second submatrix data is doubled in the first direction or doubled in the second direction. Optionally, the second condition can be used to indicate whether the data dimension of the first submatrix data is doubled in the first direction or doubled in the second direction.

[0094] In one possible implementation, the fourth and fifth matrix data can be determined using an interpolation method. The interpolation method can include at least one of the following: linear interpolation, bilinear interpolation, bicubic interpolation, or nearest neighbor interpolation. And / or the seventh matrix data can be determined using an interpolation method.

[0095] In one possible implementation, the second object corresponding to the first component C may include third row data, third column data, or third matrix block data corresponding to the first component C. The second object may include at least one first element value A and at least one first element value B. First element value A may be a first element value that requires a third loading operation. First element value B may be a first element value that does not require a third loading operation. The number of first element values ​​A and the number of first element values ​​B may be determined according to a predetermined number.

[0096] The processor may further include a second register. The second register may be used to store at least one sixth identifier corresponding to the second object. Of the at least one sixth identifier, the sixth identifier corresponding to the first element value A has a first value, and the sixth identifier corresponding to the first element value B has a second value. The first value may be used to indicate that a third load operation for the first element value A is performed. The second value may be used to indicate that a third load operation for the first element value B is not performed.

[0097] The instruction set may include at least one tenth instruction corresponding to the first component C. The tenth instruction may include a seventh identifier, at least one eighth identifier, at least one ninth identifier, and the tenth identifier. The seventh identifier may be used to indicate a third load operation. The eighth identifier may be used to indicate a first region. At least one ninth identifier may be used to indicate a second address corresponding to at least one second object.

[0098] Based on the instruction set, the first component C is downsampled according to at least one first element value corresponding to the first component C to obtain the third matrix data corresponding to the first component C. This may include: determining a second address corresponding to at least one second object based on at least one ninth identifier; retrieving the second object corresponding to each of the at least one second address from memory based on a seventh identifier and the second address corresponding to each of the at least one second object; retrieving at least one sixth identifier corresponding to each of the at least one second object from a second register corresponding to a tenth identifier; and loading the second object corresponding to each of the at least one second address into a first region corresponding to each of the at least one eighth identifier based on the seventh identifier and the at least one tenth identifier corresponding to each of the at least one second object. The sixth identifier of the second object stored in the first region is a first value.

[0099] It can execute at least one tenth instruction corresponding to the first component C in parallel or execute at least one tenth instruction corresponding to the first component C sequentially.

[0100] Since the second register stores at least one sixth identifier corresponding to the second object, and the sixth identifier corresponding to the first element value A can be a first value and the sixth identifier corresponding to the first element value B can be a second value, the first value is used to indicate that a third loading operation for the first element value A is performed, and the second value is used to indicate that a third loading operation for the first element value B is not performed. Therefore, the first element value A corresponding to the first value can be loaded without loading the first element value B corresponding to the second value. Thus, distributed loading of the first element value corresponding to the second object is achieved, which is also called downsampling. Since distributed loading can improve processing efficiency, the processing efficiency is improved.

[0101] In one possible implementation, the first visual data may include first pixel values ​​corresponding to each of a plurality of pixels. The first pixel values ​​may include at least one first component value corresponding to each of a plurality of first components. First element values ​​may be used to indicate first component values. The first element values ​​corresponding to each of the plurality of first components in the first matrix data are arranged consecutively, meaning the first matrix data may include first sub-matrix data corresponding to each of the plurality of first components.

[0102] When the instruction set is used to control the processor to perform color space conversion of the first visual data, processing the first matrix data corresponding to the first visual data based on the instruction set may include: obtaining the second visual data in the two-dimensional registers included in the processor, based on the instruction set, according to the ninth matrix data and the first matrix data corresponding to the first visual data.

[0103] According to embodiments of this application, the second visual data may include second pixel values ​​corresponding to each of the plurality of pixels. The second pixel values ​​may include second component values ​​corresponding to each of the plurality of second components. The second components may be color components of a second color space. The ninth matrix data may be determined based on the conversion relationship between the plurality of first components and the plurality of second components.

[0104] The instruction set can be used to indicate the color space conversion instruction set related to the SME instruction set. The instruction set can be used to convert the first pixel value corresponding to a pixel into the second pixel value corresponding to the pixel based on the first matrix data corresponding to the ninth data and the first visual data.

[0105] Since the instruction set is a color space conversion instruction set related to the SME instruction set, the SME instruction set can speed up matrix operations, thus improving the processing efficiency of the color space conversion process.

[0106] In one possible implementation, the instruction set may include at least one ninth instruction.

[0107] Obtaining second visual data based on the ninth matrix data and the first matrix data corresponding to the first visual data may include: obtaining second visual data based on at least one ninth instruction, according to the ninth matrix data and the first vector data corresponding to each of the multiple pixels.

[0108] The first vector data corresponding to a pixel can be determined based on the first element values ​​of the multiple first components corresponding to the pixel in the first matrix data. The ninth instruction can be used to instruct the execution of a matrix-vector multiplication between the ninth matrix data and the first vector data.

[0109] Since matrix-vector multiplication based on the SME instruction set provides direct multiplication and addition operations between matrix data and vector data, the color space conversion of the first visual data can be achieved by using the instructions corresponding to matrix-vector multiplication, thus improving the processing efficiency of the color space conversion process.

[0110] In one possible implementation, where the first matrix data is stored in memory, the method may further include loading the first matrix data from memory into a two-dimensional register.

[0111] In one possible implementation, loading the first matrix data from memory into a two-dimensional register may include: performing a second loading operation on at least one second vector data in parallel until the first matrix data is loaded from memory into the two-dimensional register.

[0112] According to embodiments of this application, the first matrix data corresponding to the first visual data may include at least one second vector data corresponding to each of the plurality of first components. The second vector data corresponding to the first component may include at least one first element value corresponding to the first component.

[0113] Because the second loading operation for at least one second vector data can be executed in parallel, processing efficiency is improved.

[0114] In one possible implementation, if the first matrix data contains alternating first element values ​​corresponding to at least two first components or the second matrix data includes first sub-matrix data corresponding to multiple first components, and the instruction set is used to control the processor to perform color space conversion of the first visual data, processing the first matrix data corresponding to the first visual data based on the instruction set may include: obtaining the second visual data in a two-dimensional register included in the processor, based on the instruction set, according to the ninth matrix data and the second matrix data corresponding to the first visual data.

[0115] According to embodiments of this application, the second visual data may include second pixel values ​​corresponding to each of the plurality of pixels. The second pixel values ​​may include second component values ​​corresponding to each of the plurality of second components. The second components may be color components of a second color space. The ninth matrix data may be determined based on the conversion relationship between the plurality of first components and the plurality of second components.

[0116] The instruction set can be used to indicate the color space conversion instruction set related to the SME instruction set. The instruction set can be used to convert the first pixel value corresponding to a pixel into the second pixel value corresponding to the pixel, based on the second matrix data corresponding to the ninth data and the first visual data.

[0117] Since the instruction set is a color space conversion instruction set related to the SME instruction set, the SME instruction set can speed up matrix operations, thus improving the processing efficiency of the color space conversion process.

[0118] In one possible implementation, the instruction set may include at least one ninth instruction.

[0119] The second visual data is obtained based on the ninth matrix data and the second matrix data corresponding to the first visual data. This may include: obtaining the second visual data based on at least one ninth instruction, according to the ninth matrix data and the first vector data corresponding to each of the multiple pixels.

[0120] The first vector data corresponding to a pixel can be determined based on the first element values ​​of the multiple first components corresponding to the pixel in the second matrix data. The ninth instruction can be used to instruct the execution of a matrix-vector multiplication between the ninth matrix data and the first vector data.

[0121] Since matrix-vector multiplication based on the SME instruction set provides direct multiplication and addition operations between matrix data and vector data, the color space conversion of the first visual data can be achieved by using the instructions corresponding to matrix-vector multiplication, thus improving the processing efficiency of the color space conversion process.

[0122] In one possible implementation, where the second matrix data is stored in memory, the method may further include loading the second matrix data from memory into a two-dimensional register.

[0123] In one possible implementation, loading the second matrix data from memory into a two-dimensional register may include: performing a second loading operation on at least one second vector data in parallel until the second matrix data is loaded from memory into the two-dimensional register.

[0124] According to embodiments of this application, the second matrix data corresponding to the first visual data may include at least one second vector data corresponding to each of the plurality of first components. The second vector data corresponding to the first component may include at least one first element value corresponding to the first component.

[0125] Because the second loading operation for at least one second vector data can be executed in parallel, processing efficiency is improved.

[0126] SME and / or SVE enable parallel processing at the hardware level, independent of software multithreading. Two-dimensional registers can store and manipulate large-scale matrix data. Through these registers, SME and / or SVE can perform parallel computations on multiple element values ​​in a single instruction cycle. SME provides a dedicated instruction set for efficient matrix operations, allowing direct manipulation of matrix data at the hardware level, reducing instruction overhead and data transfer latency. By implementing parallel processing at the hardware level, the SME and / or SVE instruction sets significantly improve the efficiency of matrix operations, with hardware acceleration avoiding the context switching and synchronization overhead of software multithreading. Developers can utilize the SME and / or SVE instruction sets for matrix operations without explicitly managing threads, simplifying the programming model and reducing the complexity of parallel programming. The design of SME and / or SVE allows them to scale to support larger datasets and more complex matrix operations; this scalability is crucial when handling large-scale data.

[0127] The data processing method proposed in this application leverages the advantages of SME and / or SVE, including multi-vector parallel loading and computation, horizontal and vertical access to two-dimensional registers, distributed loading, vector flipping, matrix addition and accumulation, matrix multiplication by vector, or block processing. It optimizes at least one of component separation, visual geometric transformation, component sampling, or color space conversion in video or image processing, improving the processing speed and efficiency of visual data processing. Furthermore, applying the data processing method proposed in this application to image decoding is expected to improve performance by 2-4 times. Simultaneously, it reduces processor utilization and memory bandwidth, improves system performance, and reduces device power consumption. For example, image processing may include image processing for high-resolution images.

[0128] Secondly, embodiments of this application provide a data processing apparatus that can be deployed on a processor. The data processing apparatus may include: a processing module for processing first matrix data corresponding to first visual data based on an instruction set; and an output module for outputting the processing results.

[0129] According to embodiments of this application, the instruction set may be determined based on the Extensible Single Instruction Multiple Data (SMILD) instruction set. The first matrix data may include at least one first element value corresponding to each of the plurality of first components. The first components may be color components of a first color space.

[0130] The instruction set can be used to control the processor to perform at least one of the following processes: storage format conversion of first visual data, visual geometric transformation of first visual data, sampling of first component, or color space conversion of first visual data. The instruction set can include instructions corresponding to at least one of the following: access, matrix-outside accumulation, or matrix-vector multiplication. Access can include at least one of the following: horizontal access, vertical access, or distributed access. Instructions corresponding to access can be used in at least one of the above processes. Instructions corresponding to matrix-outside accumulation can be used for sampling of the first component. Instructions corresponding to matrix-vector multiplication can be used for color space conversion.

[0131] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory is used to store computer execution instructions, and the processor is used to run the computer execution instructions stored in the memory to perform the method described in the first aspect or any possible implementation of the first aspect.

[0132] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0133] Fifthly, embodiments of this application provide a computer program product including a computer program, which, when run, causes the computer to perform the method described in the first aspect or any possible implementation thereof.

[0134] Sixthly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0135] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0136] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0137] Figure 1 A schematic diagram of image data corresponding to the YUV color space provided in an embodiment of this application;

[0138] Figure 2 This is a schematic diagram of matrix external accumulation provided in an embodiment of this application;

[0139] Figure 3A A schematic diagram illustrating the color encoding format of the YUV color space provided in the embodiments of this application;

[0140] Figure 3B A schematic diagram of YU 12 provided in the embodiments of this application;

[0141] Figure 3C A schematic diagram of NV12 provided for an embodiment of this application;

[0142] Figure 3D A schematic diagram of NV 21 provided in an embodiment of this application;

[0143] Figure 4 A schematic diagram illustrating a predicate register-based loading operation provided in an embodiment of this application;

[0144] Figure 5 A schematic diagram of the ZA register provided in an embodiment of this application;

[0145] Figure 6 A schematic diagram of matrix external accumulation based on FMOPA instructions provided in an embodiment of this application;

[0146] Figure 7 A schematic diagram illustrating the principle of a data processing method based on an instruction set related to the SME instruction set and / or SVE instruction set, provided for embodiments of this application;

[0147] Figure 8 A schematic diagram illustrating the component separation process based on the instruction set related to the SME instruction set and / or SVE instruction set, provided for embodiments of this application;

[0148] Figure 9 A schematic diagram of a visual rotation process based on an instruction set related to the SME instruction set and / or SVE instruction set, provided for an embodiment of this application;

[0149] Figure 10A schematic diagram of a component sampling process based on an instruction set related to the SME instruction set and / or SVE instruction set, provided for embodiments of this application;

[0150] Figure 11 A schematic diagram illustrating the color space conversion process based on instruction sets related to the SME instruction set and / or SVE instruction set, provided for embodiments of this application;

[0151] Figure 12 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0152] Figure 13 A software structure block diagram of an electronic device provided in an embodiment of this application;

[0153] Figure 14A A schematic diagram of the first matrix data provided in the embodiments of this application;

[0154] Figure 14B A schematic diagram of the first component C and the first component D provided in the embodiments of this application;

[0155] Figure 15 A schematic diagram illustrating the principle of another data processing method based on an instruction set related to the SME instruction set and / or SVE instruction set, provided for embodiments of this application;

[0156] Figure 16A A schematic diagram illustrating the sequential loading of instruction sets based on the SME instruction set and / or SVE instruction set, provided for embodiments of this application;

[0157] Figure 16B A schematic diagram illustrating distributed loading based on instruction sets related to the SME instruction set and / or SVE instruction set, provided for embodiments of this application;

[0158] Figure 17A A schematic diagram of another visual rotation process based on an instruction set related to the SME instruction set and / or SVE instruction set provided for embodiments of this application;

[0159] Figure 17B A schematic diagram of another visual rotation process based on an instruction set related to the SME instruction set and / or SVE instruction set provided for embodiments of this application;

[0160] Figure 18A A schematic diagram of a component sampling process based on a first method provided for an embodiment of this application;

[0161] Figure 18B A schematic diagram of a component sampling process based on the second method provided in an embodiment of this application;

[0162] Figure 19A schematic diagram illustrating the parallel execution of a second loading operation on multiple second vector data, provided in an embodiment of this application;

[0163] Figure 20 A flowchart illustrating the data processing method provided in the embodiments of this application;

[0164] Figure 21 This is a structural block diagram of the data processing apparatus provided in an embodiment of this application. Detailed Implementation

[0165] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0166] To facilitate understanding of the embodiments of this application, the following points will be explained first.

[0167] I. In the embodiments of this application, "instruction" may include direct instruction, indirect instruction, explicit instruction, or implicit instruction. When describing a certain instruction information for indicating A, it can be understood that the instruction information carries A, directly indicates A, or indirectly indicates A.

[0168] II. In the embodiments of this application, " / " can indicate that the objects before and after it are in an "or" relationship. For example, "A / B" can mean A or B. "And / or" can be used to describe three relationships between the related objects. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Here, A and B can be singular or plural.

[0169] III. In the embodiments of this application, "at least one" can refer to one or more. "More than one" can refer to two or more, for example, three, four or more. Similar expressions (e.g., at least one, at least one, etc.) are similar. "At least one of the following," "one or more of the following," or similar expressions can refer to any combination of these items, and can include only a single item or a combination of multiple items. For example, at least one of a, b, or c can represent a, b, or c; a and b; a and c; b and c; a, b, and c. Wherein, a, b, and c can be singular or plural.

[0170] IV. In the embodiments of this application, the various numerical designations are merely for descriptive convenience and are not intended to limit the scope of protection of the embodiments of this application. The magnitude of the sequence numbers involved in the embodiments of this application does not imply the order of execution; the execution order of each process should be determined by its function and internal logic. For example, the terms "first," "second," "third," "fourth," and other various terminology (if present) in the specification, claims, and drawings of the embodiments of this application can be used to distinguish similar objects, rather than necessarily to describe a specific order or sequence. Wherein, such terms can be interchanged where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein, and "first," "second," "third," "fourth," etc., are not necessarily different.

[0171] V. In the embodiments of this application, the words "exemplary," "example," or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary," "example," or "for example" should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of "exemplary," "example," or "for example" is intended to present the relevant concepts in a specific manner to facilitate understanding.

[0172] VI. In the embodiments of this application, "storage" or "preservation" can refer to storage in one or more memories. These memories can be separately configured or integrated into an encoder or decoder, processor, or communication device. Alternatively, some memories can be separately configured, while others can be integrated into a decoder, processor, or communication device. The type of memory can be any form of storage medium, and this application embodiment does not limit this.

[0173] VII. In the embodiments of this application, the terms "comprising", "having", and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to which steps or units are clearly listed, but may include other steps or units that are not clearly listed or that are inherent to these processes, methods, systems, or devices.

[0174] 8. The arrows or boxes shown by dashed lines in the schematic diagrams of the accompanying drawings of the embodiments of this application may represent optional steps or optional modules.

[0175] 9. Unless otherwise specified or in case of logical conflict, the terms and / or descriptions of different embodiments of this application are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0176] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms involved in the embodiments of this application will be explained below.

[0177] 1. A color space (CS) is used to describe how colors are represented. A color space defines the type, number, and value range of color components (CC). Color components can include at least one of a luminance component (LC) or a chroma component (CC). A luminance component is a color component in the color space used to indicate luminance information. A chroma component is a color component in the color space used to indicate chroma information. Color components can be represented numerically. The numerical value of a color component can be called a color component value. A color component value can include at least one of a luminance component value or a chroma component value.

[0178] For example, a color space can include at least one of the following: RGB color space, HSV color space, HSB color space, CMYK color space, YUV color space, YCbCr color space, or Lab color space. RGB color space can be used in display devices or computer graphics. YUV and YCbCr color spaces can be used in video encoding / decoding or video transmission. YCbCr color space can be a digital version of YUV color space and can be used for digital signal processing. HSV and HSB color spaces can be used for color selection or color adjustment. CMYK color space can be used in the printing industry. Lab color space can be used for color measurement, color matching, color conversion, or image processing.

[0179] The color components of the RGB color space can include R (red), G (green), and B (blue) components. R component values ​​indicate the intensity of the red channel. G component values ​​indicate the intensity of the green channel. B component values ​​indicate the intensity of the blue channel. The luminance information of the RGB color space can be determined based on the R, G, and B component values. The chromaticity information of the RGB color space can be determined based on converted R, G, and B component values. These converted R, G, and B component values ​​can be obtained by converting the R, G, and B component values ​​to other color spaces.

[0180] The HSV color space includes H, S, and V (value, lightness or brightness) components. H component values ​​indicate the basic type of color and can be expressed in degrees. S component values ​​indicate the purity or intensity of a color. V component values ​​indicate the lightness or darkness of a color.

[0181] The HSB color space includes H (H) components, S (S) components, and B (brightness) components. B component values ​​indicate the lightness or darkness of a color. For an explanation of H and S component values, please refer to the HSV color space section above; they will not be repeated here.

[0182] The CMYK color space includes a C (cyan) component, an M (magenta) component, a Y (yellow) component, and a K (key or black) component. The C component value indicates the intensity of the cyan channel. The M component value indicates the intensity of the magenta channel. The Y component value indicates the intensity of the yellow channel. The K component value indicates the intensity of the black channel.

[0183] The color components of the YUV color space can include a Y (luma or luminance, lightness) component, a U (chroma or chrominance, chromaticity) component, and a V (chroma or chrominance, chromaticity) component. The Y component value indicates the lightness or darkness of a color. The U component value indicates the relative intensity of blue. The relative intensity of blue can be determined based on the difference between blue and lightness. The V component indicates the relative intensity of red. The relative intensity of red can be determined based on the difference between red and lightness. The U component value can be determined by the difference between the B component value of the RGB color space and the Y component value of the YUV color space. The V component value can be determined by the difference between the R component value of the RGB color space and the Y component value of the YUV color space.

[0184] The color components of the YCbCr color space can include a Y (luma or luminance, lightness) component, a Cb (chroma or chrominance, chromaticity) component, and a Cr (chroma or chrominance, chromaticity) component. Cb component values ​​are used to indicate the relative intensity of blue. Cr component values ​​are used to indicate the relative intensity of red. For explanations of Y, Cb, and Cr component values, please refer to the YUV color space section above; they will not be repeated here.

[0185] The Lab color space includes an L (lightness) component, an a (alpha) component, and a b (b) component. The L component value indicates the brightness or darkness of a color. The a component value indicates the color's position on the green-red axis. For example, an a component value greater than 0 indicates a reddish tint, while an a component value less than 0 indicates a greenish tint. The b component value indicates the color's position on the blue-yellow axis. For example, a b component value greater than 0 indicates a yellowish tint, while an a component value less than 0 indicates a bluish tint.

[0186] 2. A pixel can refer to the basic unit of an image or video frame. A pixel has positional and color information. Color information can include brightness (or lightness) information and chroma (or chrominance) information. A pixel's brightness information can be used to indicate the lightness or darkness of the pixel. Pixel brightness information reflects the intensity or brightness of light, without involving the hue of the color. A pixel's chroma information can be used to indicate the color distribution of the pixel. Chroma information can include at least one of hue information or saturation information. A pixel value can refer to the numerical representation of a pixel's color information. A pixel value can include color component values ​​corresponding to at least one color component.

[0187] 3. Visual data can refer to multimedia data presented in a visual form. Visual data can include image data corresponding to an image or video frame data corresponding to a video frame. An image can include multiple pixels. A video frame can include multiple pixels. Visual data can include the pixel values ​​corresponding to each of the multiple pixels. The size of an image or the size of a video frame can be represented by W×H. W (i.e., Width) can represent the width of the image, i.e., the number of pixels in the horizontal direction of the image. Or W can represent the width of the video frame, i.e., the number of pixels in the horizontal direction of the video frame. H (i.e., Height) can represent the height of the image, i.e., the number of pixels in the vertical direction of the image. Or H can represent the height of the video frame, i.e., the number of pixels in the vertical direction of the video frame. W and H can be integers greater than or equal to 1. The horizontal direction can refer to the pixel arrangement direction of each row in an image or video frame (e.g., from left to right). The vertical direction can refer to the pixel arrangement direction of each column in an image or video frame (e.g., from top to bottom).

[0188] Pixel position information can be represented using a coordinate system. This coordinate system can include a Cartesian coordinate system. In a Cartesian coordinate system, pixel position information can be represented using (x, y). x represents the pixel's position in the horizontal direction. y represents the pixel's position in the vertical direction. x∈{0, ..., W-1}. y∈{0, ..., H-1}.

[0189] To facilitate understanding the color information in visual data, the following example uses image data and the YUV color space. Figure 1 Please provide an explanation.

[0190] Figure 1 This is a schematic diagram of image data corresponding to the YUV color space provided in an embodiment of this application.

[0191] like Figure 1 As shown, the image size is W×H. W = 6, H = 4. The image data corresponding to the image can include pixel values ​​corresponding to each of the 24 pixels, namely P1, ..., Pf, ..., P24. f∈{1, ..., 24}. Pf can include the Yf component value corresponding to the Y component, the Uf component value corresponding to the U component, and the Vf component value corresponding to the V component. As one implementation, multiple pixels can share the same U component value and the same V component value. Each pixel can have a corresponding Y component value. For a detailed explanation of this part, please refer to the explanation of color encoding format and sampling format below. It should be noted that... Figure 1 This is merely an example and does not constitute a limitation on the embodiments of this application.

[0192] 4. Vector data refers to a data structure organized as a one-dimensional array. Vector data can include at least one element value. Vector data can be a 1×N row vector or an M×1 column vector. Both M and N can be integers greater than or equal to 1.

[0193] For example, a 1×N row vector can be u=[u1 … u n … u N ]. u n It can represent the value of the nth element. n∈{1,…,N}. An M×1 column vector can be... v m It can represent the value of the m-th element. m∈{1,…,M}.

[0194] 5. Matrix data can be a data structure organized as a two-dimensional array. Visual data can be represented using matrix data. Therefore, matrix data can include at least one element value corresponding to each of multiple color components. Element values ​​can be used to indicate color component values. Visual data can include pixel values ​​corresponding to multiple pixels. Pixel values ​​can include color component values ​​corresponding to each of multiple color components. Matrix data can include M rows and N columns. M can be equal to H. N can be equal to W.

[0195] For example, matrix data A can be represented as a mn This can represent the value of the element in the m-th row and n-th column. m can be used to indicate the row index (RI). n can be used to indicate the column index (CI). A row index can also be called a row identifier. A column index can also be called a column identifier.

[0196] Furthermore, matrix data A can be represented as M 1×N row vectors, i.e. a m: =[a m1 … a mn … a mN ]. a m: It can represent the row vector data of the m-th row. Matrix data A can also be represented as N M×1 column vectors, i.e., A = [a...]. :1 … a :n … a :N ]. a :n It can represent the column vector data of the nth column.

[0197] 6. Matrix multiplication (MM) refers to the operation of combining two matrix data into a new matrix data.

[0198] For example, matrix data A can include M rows and N columns. Matrix data A can be represented as... a mn This can represent the value of the element in the m-th row and n-th column of matrix data A. m ∈ {1, ..., M}. n ∈ {1, ..., N}. Both M and N can be integers greater than or equal to 1.

[0199] Matrix data B can include N rows and K columns. Matrix data B can be represented as... b nk This can represent the value of the element in the nth row and kth column of matrix data B. k∈{1, ...,K}. K can be an integer greater than or equal to 1.

[0200] Matrix data C can be obtained by matrix multiplication of matrix data A and matrix data B, i.e., C = A × B. Matrix data C can be represented as... The element value in matrix C can be the dot product of the element value in the m-th row of matrix A and the element value in the k-th column of matrix B. For example, the element value c in the m-th row and k-th column of matrix C... mk It can be represented as

[0201] 7. The outer product (OP) refers to the operation of obtaining a matrix from two vector data. For example, vector data u can be an M×1 column vector. Vector data u can be represented as... u m This can represent the m-th element value in vector data u, where m ∈ {1, ..., M}. Vector data v can be an N×1 row vector. Vector data v can be represented as v = [v1 ... v2]. n … v N ]. v n It can represent the value of the nth element in vector data v. n∈{1, ...,N}.

[0202] The matrix data obtained from the outer product of vector data u and vector data v can be represented as follows: The outer product of matrices can also be called the Kronecker product.

[0203] 8. Outer product accumulation (OPA) refers to the accumulation operation of multiple outer product accumulation results obtained by performing matrix addition on vector data in two matrix data.

[0204] For example, matrix data A can include M rows and N columns. Matrix data A can be represented as A = [a:1 … a: n … a: N The nth column vector data a in matrix data A. :n It can be represented as a mn It can represent column vector data a :n Let B be the value of the m-th element in the matrix. m ∈ {1, ..., M}. n ∈ {1, ..., N}. Matrix data B can contain N rows and K columns. Matrix data B can be represented as... The nth row vector data b in matrix data B n: It can be represented as b n: =[b n1 … b nk …b nK ]. b nkIt can represent row vector data b n: The value of the k-th element in the dataset. k∈{1, ...,K}.

[0205] Matrix data C can be obtained by summing the extramatrix amounts of matrix data A and matrix data B, i.e.

[0206]

[0207] Therefore, matrix multiplication can be converted into matrix external accumulation. To better understand the matrix external accumulation described in the embodiments of this application, the following example uses matrix data C as the matrix data obtained by the matrix external accumulation of matrix data A and matrix data B. Matrix data A may include M rows and N columns, and matrix data B may include N rows and K columns, with M=4, N=2, and K=4, as an example. Figure 2 An explanation is needed. It should be noted that... Figure 2 This is merely an example and does not constitute a limitation on the embodiments of this application.

[0208] Figure 2 This is a schematic diagram of matrix external accumulation provided in an embodiment of this application.

[0209] like Figure 2 As shown, the matrix outer product summation process between matrix data A and matrix data B can be as follows: Determine the outer product between the first column of matrix data A and the first row of matrix data B to obtain matrix data C1. Matrix data C1 can include 4 rows and 4 columns. Determine the outer product between the second column of matrix data A and the second row of matrix data B to obtain matrix data C2. Based on matrix data C1 and matrix data C2, obtain matrix data C.

[0210] 9. Matrix-vector multiplication (MVM) can be used to combine matrix data and vector data into a new vector data. Matrix data A can include M rows and N columns. Matrix data A can be represented as... a mn This can represent the element value in the m-th row and n-th column of matrix data A. m∈{1, ..., M}. n∈{1, ..., N}. Vector data v can be an N×1 row vector. Vector data v can be represented as... v n It can represent the value of the nth element in vector data v.

[0211] Vector data u can be obtained by matrix multiplication of matrix data A and vector data v, i.e., u = Av. Vector data u can be represented as... The element value in vector data u can be the dot product of the element value in the m-th row of matrix data A and the n-th element value in vector data v. For example, the m-th element value u in vector data u... m It can be represented as

[0212] Matrix data A can also be represented as A = [a :1 … a :n … a :N The nth column vector data a in matrix data A. :n It can be represented as a mn It can represent column vector data a :n The value of the m-th element in the vector data. The vector data u can be represented as... or

[0213] If matrix data A can be divided into submatrix data A1 and submatrix data A2, i.e., A = [A1 A2], then vector data v can be divided into v1 and v2. Then the vector data u can be represented as u = Av = A1v1 + A2v2.

[0214] 10. Row stride (RS) refers to the address offset between two adjacent rows in matrix data. Row stride can be used to determine row addresses or block addresses. In row-major storage, the row stride can be determined based on the number of element values ​​in a row and the size of those element values. Element size refers to the storage space occupied by an element value. For example, element size can refer to the number of bytes occupied by an element value.

[0215] 11. Column stride (CS) refers to the address offset between two adjacent columns in matrix data. Column stride can be used to determine column addresses or block addresses. In column-major storage, the column stride can be determined based on the number of element values ​​in the column data and the size of those element values.

[0216] 12. Offset refers to the address offset between the base address and the target address. Offsets can be used to determine row addresses, column addresses, or block addresses, etc. Offsets can include row offsets or column offsets. A row offset refers to the address offset between the base address and the target row address. A column offset refers to the address offset between the target row address and the target element address. The target row address can refer to the base address of the target row data. The target element address can be used to indicate the address of the target element value. Row offsets can be determined based on row index and row step. Column offsets can be determined based on column index and element size.

[0217] In row-major storage, the offset can be determined based on the row index, row step, column index, and element size. In column-major storage, the offset can be determined based on the column index, column step, row index, and element size.

[0218] 13. A row address (RA) can refer to the base address of a row of data. The row address can be determined based on the base address, row index, and row step size. Optionally, the row address can be determined based on the base address and row offset. The row index can be used to indicate the index of the row data. The base address can refer to the starting address of the matrix data.

[0219] 14. A column address (CA) refers to the base address of a column of data. The column address can be determined based on the base address, column index, and column step size. A column index can be used to indicate the index of the column data.

[0220] 15. A tile address (TA) can refer to the base address of a matrix block of data. In row-major storage, the tile address can be determined based on the base address, the starting row index of the matrix block, the row step size, the starting column index of the matrix block, and the element size. In column-major storage, the tile address can be determined based on the base address, the starting column index of the matrix block, the column step size, the starting row index of the matrix block, and the element size.

[0221] 16. A color encoding format (CEF) can be a specific implementation of a color space. A color encoding format defines how color information within a color space is represented; that is, it defines how color components are represented.

[0222] 17. Sampling format (SF) can be used to describe the sampling rate of the chrominance component relative to the luminance component. For example, the sampling format of the YUV color space can be used to describe the sampling rate of the U and V components relative to the Y component.

[0223] The sampling format can be represented in the form J:A:B. J represents the number of samples of the luminance component (e.g., the Y component) in the horizontal direction. A represents the number of samples of the chrominance components (e.g., the U and V components) in the horizontal direction in the first row. B represents the number of samples of the chrominance components (e.g., the U and V components) in the horizontal direction in the second row. This representation describes the sampling relationship between the luminance and chrominance components in the horizontal and vertical directions. The first row can refer to a row in an image or video frame. This row can be an odd-numbered row, for example, the 3rd row. The second row can refer to the row immediately following the first row in an image or video frame. The next row can be an even-numbered row, for example, the 4th row. A and B can be used to determine the sampling method of the chrominance components in the vertical direction. For example, if A = B, then in the vertical direction, the number of samples of the chrominance components is the same as the number of samples of the luminance components. If B = 0, then in the vertical direction, the number of samples of the chrominance components is half the number of samples of the luminance components. This sampling format can reduce the amount of data without significantly affecting visual quality. Reducing the amount of data is beneficial for video compression and transmission. Sampling formats can also be called subsampling formats, color sampling formats, or chroma sampling, etc.

[0224] For J, for example, if J=4, it means that in the horizontal direction, each pixel samples a luminance component value.

[0225] For A, for example, if A = 4, it means that in the horizontal direction, the number of samples for the chroma component in the first row is the same as the number of samples for the luma component. If A = 2, it means that in the horizontal direction, every 2 pixels in the first row share one chroma component value. If A = 1, it means that in the horizontal direction, every 4 pixels in the first row share one chroma component value.

[0226] For the value of B, for example, if B = 4, it means that in the horizontal direction, the number of samples for the chroma component in the second row is the same as the number of samples for the luma component. If B = 2, it means that in the horizontal direction, every two pixels in the second row share one chroma component value. If B = 0, it means that in the horizontal direction, no chroma component is sampled in the second row.

[0227] Therefore, if J:A:B = 4:4:4, it means that the number of samples for the luminance and chrominance components is the same in both the horizontal and vertical directions; that is, each luminance component value has a corresponding chrominance component value. For example, each Y component value has corresponding U and V component values. Thus, each pixel has corresponding Y, U, and V component values. If J:A:B = 4:2:2, it means that in the horizontal direction, the number of samples for the luminance components is twice the number of samples for the chrominance components, and in the vertical direction, the number of samples for the luminance components is the same as the number of samples for the chrominance components; that is, every two luminance component values ​​have corresponding chrominance component values. For example, every two Y component values ​​have corresponding U and V component values. Thus, each pixel has a corresponding Y component value, and every two horizontal pixels have corresponding U and V component values. If J:A:B = 4:2:0, it means that in both the horizontal and vertical directions, the number of samples for the luminance components is twice the number of samples for the chrominance components. For example, in the horizontal direction, every two Y component values ​​have corresponding U and V component values. In the vertical direction, every two Y component values ​​have corresponding U and V component values. Thus, each pixel has a corresponding Y component value, and every four pixels (i.e., a 2×2 pixel block) has corresponding U and V component values.

[0228] 18. A storage area (SA) can refer to a contiguous or non-contiguous space within a storage medium used to store data. Storage media can include memory or registers.

[0229] 19. An array is a data structure that can be used to store elements of the same type. An array occupies a contiguous block of storage space; that is, the elements of an array are stored consecutively within that block.

[0230] 20. A plane can refer to a two-dimensional data structure or a one-dimensional data structure that unfolds one dimension of multidimensional data. A plane can be a two-dimensional array used to store color component values.

[0231] 21. Interleaved storage (IS) refers to storing the color component values ​​corresponding to multiple color components alternately in a first order within the same storage area. Therefore, a storage area can store the color component values ​​corresponding to multiple color components in an alternating first order. If visual data is represented by matrix data, and the matrix data is stored using interleaved storage, then the element values ​​corresponding to multiple color components in the matrix data are arranged alternately. Two adjacent element values ​​in the matrix data may correspond to different color components. Adjacent element values ​​can be from the same row or the same column. If two adjacent element values ​​are from the same row, then the element values ​​in the same column can correspond to the same color component. If two adjacent element values ​​are from the same column, then the element values ​​in the same row can correspond to the same color component.

[0232] Alternating storage can include packed storage (PS). Packed storage refers to storing the color component values ​​corresponding to each of the color components in the same storage area in a first-order sequence. For example, packed storage can refer to storing the color component values ​​corresponding to each of the color components in the same array in a first-order sequence. Alternating storage can also be called interleaved storage.

[0233] Since the color component values ​​corresponding to multiple color components are stored alternately in the same storage area, the physical addresses of the related data are contiguous, which can improve access efficiency and reduce the number of accesses.

[0234] 22. Non-interleaved storage (NIS) refers to storing the color component values ​​corresponding to each of multiple color components in their respective storage areas. Thus, each color component has its own independent storage area. The storage area corresponding to a color component stores multiple color component values ​​consecutively; that is, the multiple color component values ​​corresponding to that color component are arranged either non-interleaved or consecutively within the storage area corresponding to that color component. For example, the color component values ​​corresponding to each of multiple color components are stored in their respective planes (or arrays). Each color component value has its own corresponding plane (or array). If visual data is represented using matrix data, and the matrix data is stored using non-interleaved storage, then the element values ​​corresponding to each of the multiple color components in the matrix data are arranged consecutively. Matrix data can include sub-matrix data corresponding to each of the multiple color components. Non-interleaved storage can also be called planar storage (PS).

[0235] Because multiple color component values ​​corresponding to a color component are stored contiguously, it facilitates individual access to a single color component, simplifies data processing, and is suitable for parallel processing. For example, if processing of a specific color component is required, the color component value corresponding to that component can be accessed directly.

[0236] 23. Semi-planar storage (SPS) can refer to a situation where at least one color component among multiple color components has an independent storage area, and the color component values ​​corresponding to other color components are stored alternately in the same storage area in a second order. Other color components can refer to the color components other than at least one color component among the multiple color components. For example, at least one color component among multiple color components has a corresponding plane (or array). Other color components have corresponding planes (or arrays).

[0237] 24. Separated color components refer to color components that have independent storage areas among multiple color components. Non-separated color components refer to color components that do not have independent storage areas among multiple color components. Therefore, color components in alternating storage can be called non-separated color components. Color components in non-alternating storage can be called separated color components. Other color components in half-plane storage can be called non-separated color components.

[0238] 25. Storage format (SF) refers to the arrangement of visual data stored on a storage medium (e.g., memory). For example, storage format defines the arrangement of pixel values. Pixel values ​​may include color component values ​​corresponding to multiple color components. Storage format may include at least one of the following: planar storage, half-planar storage, or packed storage. For explanations of planar storage, half-planar storage, and packed storage, please refer to the corresponding sections above; they will not be repeated here. Furthermore, the storage precision of each color component can be configured according to actual business needs and is not limited here. For example, the storage precision can be 1 byte.

[0239] To facilitate a better understanding of the storage format, the following explanation uses the storage method of the YUV sampling format as an example, in conjunction with Table 1. The storage format of the YUV sampling format can include at least one of the following: planar storage, half-planar storage, or packed storage. For example, for YUV 420, planar storage can be called YUV 420P. For YUV 422, planar storage can be called YUV422P. For YUV 420, half-planar storage can be called YUV 420SP. It should be noted that Table 1 is merely an illustrative example and does not constitute a limitation on the embodiments of this application.

[0240]

[0241]

[0242] Table 1

[0243] As shown in Table 1, YUV 420P can include YU 12 (or I420) or YV 12. The storage order of YU 12 is Y-plane → U-plane → V-plane, meaning YU 12 can be stored in the order of Y component value → U component value → V component value. The storage order of YV 12 is Y-plane → V-plane → U-plane, meaning YV 12 can be stored in the order of Y component value → V component value → U component value. The Y-plane can store the Y component value of a pixel. The U-plane can store the U component value of a pixel. The V-plane can store the V component value of a pixel.

[0244] The YUV 420SP may include NV 12 or NV 21. NV 12 can be stored in a manner where the Y component values ​​are stored independently, and the U and V component values ​​are stored alternately. NV 21 is stored in a manner where the Y component values ​​are stored independently, and the V and U component values ​​are stored alternately.

[0245] YUYV can be stored in a way that alternates between Y component values, U component values, and V component values. UYVY can be stored in a way that alternates between U component values, Y component values, and V component values.

[0246] To better understand color encoding formats, sampling formats, and storage formats, the following example uses a YUV color space image, combined with... Figure 3A , Figure 3B , Figure 3C and Figure 3D Further explanation is needed. Specifically, Figure 3A The color encoding format used to describe the YUV color space. Figure 3A Based on this, taking the YUV420 storage formats YU 12, NV 12, and NV 21 as examples, combined with... Figure 3B , Figure 3C and Figure 3D The storage format is described. Figures 3A-3D In this context, Y can be used to indicate the Y component value. U can be used to indicate the U component value. V can be used to indicate the V component value. Figures 3A-3D This is merely an illustrative example and does not constitute a limitation on the embodiments of this application.

[0247] Figure 3A This is a schematic diagram of the color encoding format of the YUV color space provided in the embodiments of this application.

[0248] like Figure 3AAs shown, it has 24 pixels. The sampling format of the YUV color space can include at least one of the following: J:A:B=4:4:4, J:A:B=4:2:2, or J:A:B=4:2:0, etc. Therefore, the color encoding format of the YUV color space can include at least one of the following: YUV 444, YUV 422, or YUV 420.

[0249] For YUV 444, in the horizontal direction, each pixel samples one Y component value, one U component value, and one V component value. Thus, each pixel has corresponding Y component value, U component value, and V component value.

[0250] For YUV 422, in the horizontal direction, a U component value and a V component value are sampled every other pixel. Each pixel is sampled with one Y component value. Thus, each pixel has a corresponding Y component value. Every two horizontal pixels share the same U component value and the same V component value.

[0251] For YUV 420, in the horizontal direction, each pixel samples one Y component value. In the first row, a U component value and a V component value are sampled every other pixel. The second row does not sample U and V component values. Thus, each pixel has a corresponding Y component value. Every four pixels (i.e., a 2×2 pixel block) share the same U component value and the same V component value.

[0252] exist Figure 3A On this basis, Figure 3B This is a schematic diagram of YU 12 provided in an embodiment of this application.

[0253] like Figure 3B As shown, the storage order of YU 12 is Y plane → U plane → V plane, meaning YU 12 can be stored in the order of Y component value → U component value → V component value. The Y plane can include Y1 component values ​​to Y24 component values. The U plane can include U1 component values ​​to U6 component values. The V plane can include V1 component values ​​to V6 component values.

[0254] Based on the explanation of YUV 420 above, each pixel has a corresponding Y component value. Every four pixels share the same U and V component values. Therefore, Figure 3B In this model, the Y1, Y2, Y7, and Y8 component values ​​share the U1 and V1 component values. Similarly, every four Y component values ​​share the same U and V component values, which will not be elaborated further here.

[0255] exist Figure 3A On this basis, Figure 3C This is a schematic diagram of NV12 provided in an embodiment of this application.

[0256] like Figure 3CAs shown, the storage order of NV 12 is Y plane → UV plane. U component values ​​and V component values ​​are stored alternately in the UV plane. The Y plane can include Y1 to Y24 component values. The UV plane can include U1, V1, U2, V2, U3, V3, U4, V4, U5, V5, U6, and V6 component values.

[0257] For the relationship between the Y component values ​​and the U and V component values, please refer to [the section on...]. Figure 3B The explanation will not be repeated here.

[0258] exist Figure 3A On this basis, Figure 3D This is a schematic diagram of NV 21 provided in an embodiment of this application.

[0259] like Figure 3D As shown, the storage order of NV 21 is Y plane → UV plane. In the UV plane, V component values ​​and U component values ​​are stored alternately. Figure 3C The difference lies in the UV plane. Figure 3D The V component values ​​and U component values ​​are stored alternately. Figure 3C The U component values ​​and V component values ​​are stored alternately. (This is for...) Figure 3D Further instructions can be found in the section on Figure 3C The explanation will not be repeated here.

[0260] 26. A data element can refer to the smallest unit of data processed in a computer. A data element can be a value or a variable. At least one of the following can be performed on a data element: arithmetic operation, logical operation, comparison operation, or bitwise operation.

[0261] 27. An instruction is the basic unit of a computer program, used to instruct electronic devices to perform a predetermined operation. An instruction can include an opcode and operands. The opcode indicates the type of operation performed by the instruction. Operands indicate the data or location of the data to be operated on.

[0262] 28. The instruction format (IF) defines the encoding method of an instruction. The instruction format may include at least one of the following fields: opcode, operand, addressing mode (AM), or condition code (CC). The addressing mode can be used to indicate how the operand is obtained.

[0263] 29. An instruction set (IS) can include complex instruction set computer (CISC) or reduced instruction set computer (RISC), etc.

[0264] 30. RISC can refer to a processor architecture design philosophy. RISC architecture can improve processor performance and efficiency by simplifying the instruction set and optimizing hardware design. RISC architecture can be based on load and store instructions, meaning only load and store instructions can access memory, while other instructions can only operate on registers. RISC architecture can provide general-purpose registers to reduce memory access. RISC architecture can include at least one of the following: ARM (advanced RISC machines) architecture, MIPS (microprocessor without interlocked pipeline stages) architecture, RISC-V (reduce instruction set computer fifth generation) architecture, or SPARC (scalable processor architecture), etc.

[0265] 31. A processor (central processing unit, CPU) is responsible for executing instructions and processing data. As one implementation, the processor can be based on the ARM architecture. The processor may include registers. Registers are used to store data that the processor needs to access quickly while executing instructions. Registers may include at least one of the following: general-purpose registers (GPRs) or special-purpose registers (SPRs). General-purpose registers can be used to store at least one of the following: temporary data, addresses, or intermediate calculation results. Special-purpose registers can be used to implement predetermined functions.

[0266] As one implementation, general-purpose registers may include at least one of the following: a base register (BR), an index register (IR), or an offset register (OR). The base register can be used to store the base address of a memory address and can be used in conjunction with other registers to generate a valid address. For example, other registers may include at least one of an index register or an offset register. The base address can also be called the starting address. The index register can be used to store an offset or index value and can be used in conjunction with the base register to generate a valid address. The offset register can be used to store a fixed offset and can be used in conjunction with the base register to generate a valid address. As another implementation, special-purpose registers may include at least one of the following: a base register, an index register, or an offset register. This application does not limit the form of the base register, index register, or offset register.

[0267] 32. Single Instruction Multiple Data (SIMD) is a parallel computing technique that allows a single instruction to process multiple data points simultaneously. Multiple data points refer to the ability to process multiple data points in parallel. SIMD can be used to accelerate data-intensive tasks. A SIMD instruction set can include at least one of the following: SVE (scalable vector extension) instruction set, SVE2 instruction set, SME (scalable matrix extension) instruction set, or NEON (or advanced SIMD) instruction set, etc. Variable vector extension can also be called scalable vector extension or scalable matrix extension. Variable matrix extension can also be called scalable matrix extension or scalable matrix extension. For explanations of the SVE, SVE2, and SME instruction sets, please refer to the explanations of the above sections below; they will not be repeated here.

[0268] 33. Single Instruction Multiple Operations (SIMO) is an instruction-level parallelism design paradigm that allows a single instruction to trigger multiple independent operations. These operations can be homogeneous or heterogeneous. Homogeneous operations refer to operations of the same type, while heterogeneous operations refer to operations of different types. For example, a heterogeneous operation could be simultaneously loading data and determining an address. SIMO instruction sets may include SME instruction sets, etc.

[0269] 34. Vector registers (VR) can be used to store and process multiple data sets. Vector registers can support the SIMD instruction set. As one implementation, vector registers can include vector registers that support the SVE instruction set.

[0270] 35. Two-dimensional registers (2D registers) can be used to store two-dimensional data. For example, two-dimensional data can include matrix data. A two-dimensional register can include a region of at least one two-dimensional register. Regions of different two-dimensional registers can not overlap. Two-dimensional registers can also be called matrix registers (MR). As one implementation, two-dimensional registers can include two-dimensional registers that support the SME instruction set. As another implementation, two-dimensional registers can be on-chip caches for the processor.

[0271] 36. A predicate register (PR) can be used to store Boolean values ​​for data filtering. The Boolean value corresponding to a data element indicates whether an operation needs to be performed on that data element. The number of predicate registers can be determined based on the hardware implementation. Predicate registers are also called prediction registers.

[0272] As one implementation method, a Boolean value of "1" can be used to indicate that an operation needs to be performed on the data element. A Boolean value of "0" can be used to indicate that no operation needs to be performed on the data element; that is, a Boolean value of "0" can indicate that the operation on the data element needs to be skipped. Therefore, a Boolean value of 1 corresponding to a data element can be used to indicate that an operation needs to be performed on that data element, while a Boolean value of "0" can be used to indicate that no operation needs to be performed on that data element. The values ​​of the Boolean values ​​can be configured according to actual business needs and are not limited here.

[0273] To facilitate a better understanding of the predicate register described in the embodiments of this application, the following example illustrates how to use the predicate register to load the color component value corresponding to the target color component from memory into the first vector register in the first data. Figure 4 Please provide an explanation.

[0274] Figure 4 This is a schematic diagram of a predicate register-based loading operation provided in an embodiment of this application. Figure 4 In this context, U can be used to indicate the U component value. V can be used to indicate the V component value. Figure 4 This is merely an example and does not constitute a limitation on the embodiments of this application.

[0275] like Figure 4As shown, the first data may include multiple U component values ​​corresponding to the U component and multiple V component values ​​corresponding to the V component. The multiple U component values ​​corresponding to the U component may include U1, U2, and U3 component values. The multiple V component values ​​corresponding to the V component may include V1, V2, and V3 component values. The U component values ​​and V component values ​​are arranged alternately. The predicate register can be used to store Boolean values ​​corresponding to each of the multiple component values. A Boolean value of "0" can be used to indicate that no loading operation is required for the component value. A Boolean value of "1" can be used to indicate that a loading operation is required for the component value.

[0276] To load multiple U component values ​​corresponding to the U component in the first data from memory into the first vector register, based on the above description of Boolean values, the Boolean values ​​corresponding to each of the multiple U component values ​​in the predicate register can be configured as "1". The Boolean values ​​corresponding to each of the multiple V component values ​​can be configured as "0", thus allowing the predicate register to store "101010".

[0277] In this case, based on the "101010" stored in the predicate register, a Boolean value of "1" corresponds to the U component value, and a Boolean value of "0" corresponds to the V component value. A loading operation is performed for the U component value, but not for the V component value. That is, loading operations are performed for the U1, U2, and U3 component values, but not for the V1, V2, and V3 component values. This allows the predicate register to load multiple U component values ​​corresponding to the U component from memory into the first vector register. The first vector register can store second data including the U1, U2, and U3 component values.

[0278] To load multiple V component values ​​corresponding to the V component in the first data from memory into the first vector register, a similar method to that described above for loading multiple U component values ​​corresponding to the U component in the first data from memory into the first vector register can be used. In this case, the predicate register can store "010101". The first vector register can store the second data including the V1 component value, V2 component value, and V3 component value. Further explanations can be found in the corresponding sections above and will not be repeated here.

[0279] 37. The SVE register set may include at least one of the following: at least one vector register, at least one predicate register, or at least one scalar register. For example, multiple vector registers may include Z0 (i.e., vector register zero)-Z31. Multiple predicate registers may include P0-P15. Multiple scalar registers may include V0-V31.

[0280] 38. The SME register set may include at least one of the following: at least one two-dimensional register, at least one vector register, and at least one predicate register. For example, multiple vector registers may include Z0-Z31. As one implementation, the vector registers may be used as inputs to row and / or column data for matrix external accumulation. The ZA register may be used to store the results of matrix external accumulation. Multiple predicate registers may include P0-P15. The number and storage capacity of the two-dimensional registers may be determined according to the hardware implementation. The two-dimensional registers may include ZA (i.e., ZEray Array) registers. The ZA register may be an expandable two-dimensional register. The ZA register may be used to store matrix data or support at least one of matrix operations. The size of the ZA register may be determined according to the scalable vectorlength (or streaming vectorlength, SVL). For example, the size of the ZA register may be SVL×SVL. The ZA register may be divided into multiple slices. Slices may include horizontal slices or vertical slices.

[0281] To facilitate a better understanding of the ZA register described in the embodiments of this application, the following is combined with... Figure 5 Please provide an explanation.

[0282] Figure 5 This is a schematic diagram of the ZA register provided in an embodiment of this application.

[0283] like Figure 5 As shown, the size of the ZA register can be SVL×SVL.

[0284] 39. The SVE instruction set is a type of SMID instruction set that can be used to accelerate vector operations. The SVE instruction set may include at least one of the following: load instructions, store instructions, arithmetic operation instructions, predicate operation instructions, logical operation instructions, comparison instructions, shift instructions, broadcast instructions, reduction operation instructions, or other instructions.

[0285] Load instructions can be used to load data elements from memory into a vector register. For example, load instructions may include at least one of the following: LD1, LD2, LD3, or LD4. The LD1 instruction can be used to load data elements from memory into a vector register. The LD2, LD3, or LD4 instructions can be used to interleave multiple data elements from memory into a vector register.

[0286] Store instructions can be used to store data elements from a vector register to memory. For example, store instructions may include at least one of the following: ST1, ST2, ST3, or ST4. The ST1 instruction can be used to store data elements from a vector register to memory. The ST2, ST3, or ST4 instructions can be used to interleave multiple data elements from a vector register into memory.

[0287] Other instructions may include the reverse (REV) instruction. The reverse instruction can be used to reverse the order of data elements in a vector register.

[0288] 40. The SVE2 instruction set can be an extension of the SVE instruction set, that is, the SVE2 instruction set adds new instructions on the basis of the SVE instruction set.

[0289] For load instructions, SVE2 adds at least one of the following: SL (scatterload) instruction, GL (gatherload) instruction, or conditional load instruction. The scatterload instruction can be used to load multiple data elements from memory non-contiguously to a vector register; that is, it can load data elements from multiple non-contiguous memory addresses to a vector register. The scatterload instruction can be applied to non-contiguous data access or irregular data loading. The gatherload instruction can be used to load multiple data elements from memory contiguously to a vector register; that is, it can load data elements from multiple contiguous memory addresses to a vector register. The conditional load instruction can be used to selectively load data elements to a vector register based on a condition mask (PM).

[0290] For store instructions, SVE2 adds at least one of the following: GS (gather store) instruction, SS (scatter store) instruction, or conditional store instruction. The scatter store instruction can be used to store data elements from a vector register to multiple non-contiguous addresses in memory. The conditional store instruction can be used to selectively store data elements from a vector register to memory based on a condition mask.

[0291] For arithmetic operation instructions, the new instructions in SVE2 may include at least one of the following: matrix multiplication accumulation instruction or matrix transpose instruction, etc.

[0292] 41. The SME instruction set is a matrix extension instruction set developed based on the SVE and SVE2 instruction sets. The SME instruction set can be used to accelerate matrix operations.

[0293] The SME instruction set may include at least one of the following: matrix loading instructions, matrix storage instructions, matrix multiplication instructions, matrix accumulation instructions, matrix-vector operation instructions, matrix block operation instructions, matrix parallel operation instructions, matrix operation instructions, matrix reduction instructions, streaming matrix operation instructions, matrix mask operation instructions, or matrix conditional operation instructions, etc.

[0294] Matrix load instructions can load matrix data from memory into a two-dimensional register. Matrix store instructions can be used to load matrix data from a two-dimensional register into memory.

[0295] Matrix multiplication instructions can include at least one of the following: outer product instruction, outer product and accumulate (OPA) instruction, or matrix multiplication (MM) instruction. The outer product instruction can be used to implement outer product. The outer product and accumulate instruction can be used to implement outer product and accumulate. The matrix multiplication instruction can be used to implement matrix multiplication. For explanations of outer product, outer product and accumulate, and matrix multiplication, please refer to the corresponding sections above; they will not be repeated here.

[0296] The matrix multiply outer product and accumulate instruction can include at least one of the following: FMOPA (floating-point multiply outer product and accumulate), SMOPA (signed integer multiply outer product and accumulate), or UMOPA (unsigned integer multiply outer product and accumulate), etc.

[0297] For example, the FMOPA instruction can be FMOPA ZA0.S, Pn / M, Pm / M, Zn.S, Zm.S. Optionally, the FMOPA instruction can be FMOPA ZA0.D, Pn / M, Pm / M, Zn.D, Zm.D. FMOPA can be used to indicate matrix external accumulation. ZA0 can be used to indicate the ZA register. .S can identify single-precision floating-point. .D can represent double-precision floating-point. Pn can be used to indicate the predicate register corresponding to Zn. Pm can be used to indicate the predicate register corresponding to Zm. Zn can be used to indicate the second vector register. The second vector register can be used to store row vector data or row data. Zm can be used to indicate the third vector register. The third vector register can be used to store column vector data or column data.

[0298] Matrix accumulation instructions may include at least one of the following: matrix external accumulation instruction, matrix multiplication accumulation instruction, or block matrix multiplication accumulation instruction, etc.

[0299] Matrix-vector manipulation instructions may include at least one of the following: matrix-vector multiplication (MVM) instruction or vector outer product (VOP) instruction, etc.

[0300] Matrix block operation instructions may include at least one of the following: matrix block loading instructions or matrix block storage instructions, etc. Matrix operation instructions may include matrix transpose instructions, matrix addition instructions, matrix subtraction instructions, or element-wise operation instructions, etc.

[0301] To better understand the matrix outside accumulation plus instruction, the following will combine... Figure 6 Please provide an explanation.

[0302] Figure 6 This is a schematic diagram of matrix external accumulation based on FMOPA instructions provided in an embodiment of this application.

[0303] like Figure 6 As shown, the FMOPA instruction can be FMOPA ZA0.S, Pn / M, Pm / M, Zn.S, Zm.S. ZA0 can be used to indicate the ZA register. Pn can be used to indicate the predicate register corresponding to Zn. Pm can be used to indicate the predicate register corresponding to Zm. Zn can be used to indicate the second vector register. The second vector register can be used to store row vector data. Zm can be used to indicate the third vector register. The third vector register can be used to store column vector data.

[0304] Row vector data can be [b0 b1 b3 b4]. Pm can be used to store the Boolean values ​​corresponding to multiple elements of the row vector data, i.e.,

[1110] . Column vector data can be... Pn can be used to store the Boolean values ​​corresponding to multiple element values ​​of a column vector, i.e. A Boolean value of "1" indicates that a matrix outer product operation on element-values ​​should be performed. A Boolean value of "0" indicates that a matrix outer product operation on element-values ​​should not be performed. Therefore, based on the Boolean values ​​stored in Pm and Pn respectively, the matrix outer product summation between the row vector data and the column vector data is determined, resulting in the matrix outer product summation. This result is then stored in the ZA register corresponding to ZA0.

[0305] It should be noted that, Figure 6 This is merely an example and does not constitute a limitation on the embodiments of this application.

[0306] 42. Row reversal refers to reversing the order of rows in a matrix, but keeping the order of the elements within each row unchanged. Column reversal refers to reversing the order of columns in a matrix, but keeping the order of the elements within each column unchanged.

[0307] 43. Visual geometric transformation (VGT) refers to geometric transformations applied to image or video frames. Visual geometric transformations can include at least one of the following: visual rotation, visual scaling, or visual cropping.

[0308] 44. Color space conversion refers to the process of converting an image or video frame from one color space to another. Color space conversion can be achieved using conversion formulas. Conversion formulas can include at least one of linear transformation formulas or nonlinear transformation formulas.

[0309] 45. Interpolation methods can refer to methods for estimating unknown data using known data. Interpolation methods can include at least one of the following: linear interpolation methods, polynomial interpolation methods, spline interpolation methods, nearest neighbor interpolation methods, bilinear interpolation methods, or bicubic interpolation methods, etc.

[0310] The inventive concept of the embodiments of this application will now be described with reference to the accompanying drawings.

[0311] To facilitate understanding of the inventive concept described below with reference to the embodiments of this application, some terms related to the inventive concept will be explained based on the above explanation of terminology.

[0312] A first color space may include multiple first components. A first component can be used to indicate the color components of the first color space. Each first component has a corresponding first component value. For example, the first color space may be the YUV color space, and the multiple first components may include a Y component, a U component, and a V component. A Y component value can refer to the color component value corresponding to the Y component. A U component value can refer to the color component value corresponding to the U component. A V component value can refer to the color component value corresponding to the V component.

[0313] Based on whether a first component has an independent storage area, it can be divided into separated first components and non-separated first components. A separated first component can refer to a first component with an independent storage area. A non-separated first component can refer to a first component without an independent storage area. First component A can be a separated first component among multiple first components, meaning first component A can be a first component with an independent storage area among multiple first components. At least one first element value stored in the storage area corresponding to first component A is the first element value corresponding to first component A. First component B can be a non-separated first component among multiple first components, meaning first component B can be a first component without an independent storage area among multiple first components. The multiple first element values ​​stored in the storage area corresponding to first component B can include at least one first element value corresponding to first component B and at least one first element value corresponding to other first components B.

[0314] First visual data can refer to visual data corresponding to a first color space. First visual data can include first pixel values ​​corresponding to multiple pixels. First pixel values ​​can include first component values ​​corresponding to multiple first components. For example, the first color space can be the YUV color space. First pixel values ​​can include Y component values ​​corresponding to the Y component, U component values ​​corresponding to the U component, and V component values ​​corresponding to the V component. Y component values, U component values, and V component values ​​can be referred to as first component values.

[0315] The first visual data may have corresponding first matrix data. The first matrix data may include at least one first element value corresponding to each of the plurality of first components. The first element value may be used to indicate the first component value. If the plurality of first components are all first components A, then the first matrix data may include first sub-matrix data corresponding to each of the plurality of first components.

[0316] A first component C can be defined as one of multiple first components that satisfies the following condition: The first number is less than the second number. The first number can be the number of first element values ​​corresponding to first component C in the second matrix data or the first matrix data. The second data can be the number of pixels included in the first visual data. Therefore, if the first number is less than the second number, the first component can be called first component C. Furthermore, the first element values ​​corresponding to first component C can be the same for different pixels; that is, multiple pixels can share the same first element value. If a first component B satisfies the condition that the first number is less than the second number, then the first component B can be called first component C.

[0317] Furthermore, the first component C can be any of the multiple first components. For example, the multiple first components can include Y components, U components, and V components, so the first component C can be the Y component, the U component, or the V component.

[0318] The second matrix data can be obtained by component separation of the first matrix data. The first component values ​​corresponding to each of the multiple first components in the second matrix data are arranged continuously or non-alternatingly. The second matrix data may include second sub-matrix data corresponding to each of the multiple first components; that is, at least one first element value in the second sub-matrix data corresponding to a first component is also corresponding to a first component. All of the aforementioned multiple first components are first component A.

[0319] The instruction set can be determined based on the SME instruction set; that is, the instruction set can be an instruction set related to the SME instruction set. For example, the instruction set can be the SME instruction set. Optionally, the instruction set can be obtained by adjusting the SME instruction set. Optionally, the instruction set can include the SME instruction set and a first newly added instruction set.

[0320] The instruction set can be determined based on the SVE instruction set; that is, the instruction set can be an instruction set related to the SVE instruction set. For example, the instruction set can be the SVE instruction set. Optionally, the instruction set can be obtained by adjusting the SME instruction set. Optionally, the instruction set can include the SVE instruction set and a second newly added instruction set.

[0321] The inventive concept of the embodiments of this application will be described below.

[0322] With the continuous development of electronic devices, their functions are becoming increasingly diverse. For example, the visual data processing functions provided by electronic devices can be used to achieve at least one of the following: video encoding / decoding, image editing, video editing, image browsing, video browsing, or image preview. Visual data processing functions may include at least one of the following: component separation, visual geometric transformation, component sampling, or color space conversion.

[0323] Component separation can be used to separate the first components of a first color space, allowing for a continuous arrangement of the first component values ​​corresponding to each of the multiple first components, meaning that each first component has its own corresponding storage area. For example, component separation can be used to convert the storage format of fourth data from NV12 to YU12. In YU12, the Y, U, and V components each have their own corresponding storage areas. Component separation can be used to implement storage format conversion.

[0324] Visual geometric transformation functions can be used to perform geometric transformations on image or video frames. For example, visual geometric transformation functions can include visual rotation or visual scaling.

[0325] Component sampling can be used for both upsampling and downsampling. Upsampling can be used to add a corresponding first component value to a first component C when multiple pixels share the same first component value, so that each pixel has its own corresponding first component value for that first component C. Since multiple pixels share the same first component value, the number of first component values ​​corresponding to that first component C is less than the number of pixels included in the first visual data. Since each pixel has its own corresponding first component value for that first component C, the number of first component values ​​corresponding to that first component C is equal to the number of pixels. For example, for the fifth data, where the sampling format is YUV 420, the first component C can be either a U component or a V component. Component sampling can be used to add U component values ​​and V component values ​​to ensure that each pixel has its own corresponding U component value and V component value. This can be combined with the above description regarding... Figure 3A To understand component sampling, refer to the following explanation.

[0326] Furthermore, upsampling can also be used for image magnification, i.e., when it is necessary to increase the size of an image or video frame. In this case, the first component C can be any of the multiple first components, meaning that upsampling can be performed on any of the multiple first components to achieve image magnification.

[0327] Downsampling can be used to reduce the number of first component values ​​corresponding to a first component C, thereby compressing the data. For example, if you need to convert YUV 444 to YUV 420, you need to reduce the number of U component values ​​corresponding to the U component and the number of V component values ​​corresponding to the V component.

[0328] Furthermore, downsampling can also be used for image downsampling, i.e., reducing the size of an image or video frame. In this case, the first component C can be any of a plurality of first components; that is, downsampling can be performed on any of the plurality of first components to achieve image downsampling.

[0329] The color space conversion function can be used to convert first visual data corresponding to the first color space into second visual data corresponding to the second color space.

[0330] The following explains why electronic devices need to provide the above functions.

[0331] For component separation, the camera frame's output visual data can be stored in a half-plane format, but some image processing interfaces support planar storage instead of half-plane storage. Planar storage requires multiple color components to have their own corresponding storage areas. Therefore, in order to utilize image processing interfaces, electronic devices need to provide component separation functionality. Furthermore, if color space conversion or visual geometry conversion is involved, the separated color components can be directly accessed without deinterleaving, thus improving processing efficiency.

[0332] For visual geometric transformation functions, for example, users may have the need to switch between landscape and portrait modes and / or image scaling. In order to meet the above needs, electronic devices need to provide visual geometric transformation functions.

[0333] Regarding component sampling, if the number of color component values ​​corresponding to a color component is less than the number of pixels—for example, if the number of U component values ​​corresponding to the U component is less than the number of pixels, and the number of V component values ​​corresponding to the V component is less than the number of pixels—the lack of chromaticity information may lead to color blurring or edge artifacts. In such cases, electronic devices need to provide component sampling functionality. Furthermore, if video encoding is performed under these circumstances, the lack of chromaticity information may result in insufficient color width or motion estimation errors. In these situations, electronic devices also need to provide component sampling functionality.

[0334] When color space conversion is involved, such as the conversion between YUV and RGB color spaces, the YUV color space uses the YUV 420 color encoding format, meaning each pixel has a corresponding Y component value, and every four pixels share the same U and V component values. However, to reduce the probability of color distortion or edge blurring, the RGB color space requires each pixel to have its own corresponding R, G, and B component values. In this case, upsampling of the U and V component values ​​is necessary, requiring electronic devices to provide component sampling functionality. Furthermore, for some deep learning models, to improve model performance, the model input may require each pixel to have its own corresponding color component value. In this case, electronic devices also need to provide component sampling functionality.

[0335] If image scaling is involved, which can include enlarging or shrinking the image, then for image enlargement, the number of color component values ​​corresponding to each of the multiple color components can be increased; for image shrinking, the number of color component values ​​corresponding to each of the multiple color components can be decreased. In this case, the electronic device also needs to provide component sampling functionality.

[0336] Furthermore, there is a need for data compression. To address this, electronic devices also need to provide component sampling capabilities.

[0337] Regarding color space conversion, different application scenarios have different application and algorithm requirements. For example, since display devices use RGB subpixels to generate colors, the output of the display device requires the use of the RGB color space. In this case, if the color space corresponding to the first visual data is another color space, such as the YUV color space, then it is necessary to convert from the YUV color space to the RGB color space. Therefore, electronic devices need to provide color space conversion functionality.

[0338] The output of visual sensors and video encoding require the use of the YUV color space. If the color space corresponding to the first visual data is another color space, such as the RGB color space, then it is necessary to convert from the RGB color space to the YUV color space. Therefore, electronic devices need to provide color space conversion functions.

[0339] For some deep learning models, the input is required to use the RGB color space. If the color space corresponding to the first visual data is another color space, such as the YUV color space, then it is necessary to convert from the YUV color space to the RGB color space. Therefore, electronic devices need to provide color space conversion functions.

[0340] Visual data processing can be implemented using a processor based on serial instructions, but this method is inefficient. To improve efficiency, visual data can be represented as matrix data; therefore, we can consider how to improve the processing efficiency of matrix data.

[0341] To improve the processing efficiency of matrix data, it was found that accelerating matrix operations can be achieved. Furthermore, it was discovered that the SME instruction set and / or SVE instruction set support multiple matrix operations, which can improve processing efficiency. Therefore, embodiments of this application propose a data processing method based on instruction sets related to the SME instruction set and / or SVE instruction set. The following is a combination of... Figure 7 Please provide an explanation.

[0342] Figure 7 This is a schematic diagram illustrating the principle of a data processing method based on an instruction set related to the SME instruction set and / or the SVE instruction set, provided for an embodiment of this application.

[0343] like Figure 7 As shown, the first matrix data corresponding to the first visual data can be processed based on the instruction set associated with the SME instruction set and / or the SVE instruction set.

[0344] The instruction set may be determined based on the Extensible Single Instruction Multiple Data (SVE) instruction set. The Extensible Single Instruction Multiple Data (SVE) instruction set may include at least one of the following: the SME instruction set or the SVE instruction set. Therefore, the instruction set may be determined based on the SME instruction set and / or the SVE instruction set; that is, the instruction set may be an instruction set associated with the SME instruction set and / or an instruction set associated with the SVE instruction set.

[0345] For example, the instruction set related to the SME instruction set can be the SME instruction set itself. Optionally, the instruction set related to the SME instruction set can be obtained by adjusting the SME instruction set. Optionally, the instruction set related to the SME instruction set can include the SME instruction set and a first newly added instruction set. The instruction set related to the SVE instruction set can be the SVE instruction set itself. Optionally, the instruction set related to the SVE instruction set can be obtained by adjusting the SVE instruction set. Optionally, the instruction set related to the SVE instruction set can include the SVE instruction set and a second newly added instruction set.

[0346] The instruction set can be used to control the processor to execute at least one of the following processes: component separation process, visual geometric transformation process, component sampling process, or color space conversion process.

[0347] The instruction set may include instructions corresponding to at least one of the following: access, matrix outside accumulation, or matrix multiplication by vector. Access may include at least one of the following: horizontal access, vertical access, or distributed access. For example, horizontal access may include horizontal loading or horizontal storage. Vertical access may include vertical loading or vertical storage. Distributed access may include distributed loading or distributed storage. Horizontal access may also be row-based access. Vertical access may also be column-based access.

[0348] The instructions corresponding to access can be used to implement at least one of component separation (i.e., storage format conversion), visual geometric transformation, sampling of the first component, or color space conversion. For example, the instructions corresponding to at least one of horizontal loading, vertical loading, or distributed loading can be used to load first matrix data or other matrix data obtained from the first matrix data from memory. The instructions corresponding to matrix external accumulation can be used for sampling of the first component. For example, the instructions corresponding to matrix external accumulation can be used to obtain the matrix multiplication result between two matrix data. The two matrix data can be determined based on the first matrix data. As one implementation, the two matrix data may include first sub-matrix data corresponding to the first component C and matrix data for expanding the data dimension of the first sub-matrix data. The instructions corresponding to matrix multiplication by vector can be used for color space conversion. As one implementation, the instructions corresponding to matrix multiplication by vector can be used to obtain the matrix multiplication by vector result between ninth matrix data and first vector data corresponding to multiple vectors. The ninth matrix data can be determined based on color space conversion relationships. The first vector data can be determined based on the first matrix data. For example, the first vector data corresponding to a pixel can be determined based on the first element values ​​corresponding to the multiple first components in the first matrix data that correspond to the pixel.

[0349] The component separation process can be used to implement component separation functionality. For example, the component separation process can be used to separate at least one first element value corresponding to each of multiple first components, that is, to rearrange at least one first element value corresponding to each of multiple first components in the first matrix data, so as to load the first element value corresponding to the first component in the first matrix data from memory into a two-dimensional register. The two-dimensional register stores second matrix data corresponding to the first matrix data. The first element values ​​corresponding to each of the multiple first components in the second matrix data are arranged consecutively, that is, each of the multiple first components has an independent storage area or the second matrix data includes second sub-matrix data corresponding to each of the multiple first components. The component separation function can realize storage format conversion. For example, as described above, the first element value corresponding to the first component in the first matrix data can be loaded from memory into a two-dimensional register using instructions corresponding to at least one of horizontal loading, vertical loading, or distributed loading.

[0350] The visual geometric transformation process implements geometric transformation functionality. For example, the visual geometric transformation process can be used to perform geometric transformations on first matrix data or second matrix data corresponding to first visual data. The visual geometric transformation process can include at least one of a visual rotation process or a visual scaling process. For example, the visual rotation process can be used to rotate the first matrix data by a first angle. The first angle can include 90°, 180°, or 270°, etc. For example, as described above, the first matrix data can be rotated by a first angle using instructions corresponding to at least one of horizontal or vertical access.

[0351] Component sampling procedures can be used to implement component sampling functionality. For example, a component sampling procedure can be used to execute a sampling procedure for a first component. For example, as mentioned above, a sampling procedure for the first component can be executed using instructions corresponding to matrix external accumulation.

[0352] A color space conversion process can be used to implement color space conversion functions. For example, as mentioned above, instructions corresponding to matrix-vector multiplication can be used to convert first visual data corresponding to a first color space into second visual data corresponding to a second color space. The instruction set can include instructions corresponding to at least one of the following: access, matrix external accumulation, or matrix-vector multiplication. Access can include at least one of the following: horizontal access, vertical access, or distributed access. For example, horizontal access can include horizontal loading or horizontal storage. Vertical access can include vertical loading or vertical storage. Distributed access can include distributed loading or distributed storage. Horizontal access can also be called row-by-row access. Vertical access can also be called instance-by-instance access.

[0353] The instructions corresponding to access can be used to implement at least one of component separation (i.e., storage format conversion), visual geometric transformation, sampling of the first component, or color space conversion. For example, the instructions corresponding to at least one of horizontal loading, vertical loading, or distributed loading can be used to load first matrix data or other matrix data obtained from the first matrix data from memory. The instructions corresponding to matrix external accumulation can be used for sampling of the first component. For example, the instructions corresponding to matrix external accumulation can be used to obtain the matrix multiplication result between two matrix data. The two matrix data can be determined based on the first matrix data. As one implementation, the two matrix data may include first sub-matrix data corresponding to the first component C and matrix data for expanding the data dimension of the first sub-matrix data. The instructions corresponding to matrix multiplication by vector can be used for color space conversion. As one implementation, the instructions corresponding to matrix multiplication by vector can be used to obtain the matrix multiplication by vector result between ninth matrix data and first vector data corresponding to multiple vectors. The ninth matrix data can be determined based on color space conversion relationships. The first vector data can be determined based on the first matrix data. For example, the first vector data corresponding to a pixel can be determined based on the first element values ​​corresponding to the multiple first components in the first matrix data that correspond to the pixel.

[0354] The following section explains how to implement component separation, visual geometric transformation, component sampling, and color space conversion processes based on instruction sets related to the SME and / or SVE instruction sets.

[0355] Regarding how to implement component separation based on instruction sets related to the SME and / or SVE instruction sets, it was found that vector loading can be implemented in a row-by-row, column-by-column, or block-by-block manner based on instruction sets related to the SME and / or SVE instruction sets. This allows the first element value corresponding to the first component in the first matrix data to be loaded from memory into a two-dimensional register. Vector loading can include continuous loading or scattered loading. Scattered loading can be used to indicate loading multiple non-contiguous first element values ​​from memory into the two-dimensional register.

[0356] Therefore, for the first component A, continuous loading can be achieved in a row-by-row, column-by-column, or block-by-block manner based on instruction sets related to the SME instruction set and / or SVE instruction set. The instruction set can be a continuous loading instruction set related to the SME instruction set and / or SVE instruction set. For the first component B, distributed loading can be achieved in a row-by-row, column-by-column, or block-by-block manner based on instruction sets related to the SME instruction set and / or SVE instruction set. The instruction set can be a distributed loading instruction set related to the SME instruction set and / or SVE instruction set. Row-by-row loading can include parallel loading or sequential loading. Column-by-column loading can include parallel loading or sequential loading. Block-by-block loading can include parallel loading or sequential loading. Parallel loading can further improve processing efficiency. Continuous loading in a row-by-row manner can also be called horizontal loading. Continuous loading in a column-by-column manner can also be called vertical loading.

[0357] To better understand the component separation process described in the embodiments of this application, the following example uses the YUV color space as the first color space and the NV12 storage format of the YUV color space. Figure 8 Let me explain. In this case, the Y component is the first component A. Figure 8 In this context, Y can be used to indicate the Y component value. U can be used to indicate the U component value. V can be used to indicate the V component value.

[0358] Figure 8 This is a schematic diagram illustrating the component separation process based on the instruction set associated with the SME instruction set and / or SVE instruction set, provided for embodiments of this application.

[0359] like Figure 8 As shown, the first matrix data may include multiple Y component values ​​corresponding to the Y component, multiple U component values ​​corresponding to the U component, and multiple V component values ​​corresponding to the V component. The multiple Y component values ​​corresponding to the Y component are arranged consecutively. The multiple U component values ​​corresponding to the U component and the multiple V component values ​​corresponding to the V component are arranged alternately.

[0360] For the Y component, multiple Y component values ​​corresponding to the Y component in the first matrix data can be loaded from memory into a two-dimensional register based on the sequential loading of instruction sets related to the SME or SVE instruction sets. Sequential loading can include sequential loading by row, sequential loading by column, or sequential loading by block. During the linking and loading process, execution can be parallel or sequential.

[0361] For the U component, multiple U component values ​​corresponding to the U component in the first matrix data can be loaded from memory into a two-dimensional register using scatter loading based on instruction sets related to the SME or SVE instruction sets. Scatter loading can include row-wise, column-wise, or block-wise scatter loading. During scatter loading, execution can be parallel or sequential. To implement scatter loading, predicate registers can be used, or they can be omitted.

[0362] For the V component, multiple V component values ​​corresponding to the V component in the first matrix data can be loaded from memory into a two-dimensional register based on the distributed loading of instruction sets related to the SME or SVE instruction sets. Further explanations regarding the V component can be found in the explanations for the U component above, and will not be repeated here.

[0363] It should be noted that, Figure 8 This is merely an example and does not constitute a limitation on the embodiments of this application. Furthermore, for further explanation of how to implement the component separation process based on the SME instruction set and / or SVE instruction set, please refer to the following description of the component separation process, for example, regarding... Figure 16A and Figure 16B Explanation.

[0364] The following explanation uses the visual rotation process as an example to illustrate how to implement the visual geometric transformation process based on the instruction sets related to the SME instruction set and / or SVE instruction set.

[0365] Based on the instruction set, the second sub-matrix data corresponding to each of the multiple first components obtained from the two-dimensional register can be rotated by a first angle to obtain the third sub-matrix data corresponding to each of the multiple first components stored in memory. Optionally, if the first matrix data may include the first sub-matrix data corresponding to each of the multiple first components, the first sub-matrix data corresponding to each of the multiple first components obtained from the two-dimensional register can be rotated by a first angle based on the instruction set to obtain the third sub-matrix data corresponding to each of the multiple first components stored in memory.

[0366] The instruction set can be used to indicate the rotation instruction set associated with the SME instruction set and / or the SVE instruction set, that is, the instruction set can be the SME instruction set and / or the SVE instruction set associated with achieving the rotation angle.

[0367] To better understand the visual rotation process described in the embodiments of this application, the following example uses the first component as the U component and the first angle as 90°, 180°, or 270°, combined with... Figure 9 Please provide an explanation. Figure 9The 'U' in the text can be used to indicate the U component value (i.e., the first element value corresponding to the U component). The 90°, 180°, and 270° mentioned in the embodiments of this application can refer to angles in a clockwise direction, i.e., 90°, 180°, and 270° clockwise.

[0368] Figure 9 This is a schematic diagram of a visual rotation process based on the SME instruction set provided in an embodiment of this application.

[0369] like Figure 9 As shown, when the first angle is 90° or 270°, the second sub-matrix data corresponding to the U component can be rotated by 90° or 270° based on parallel vertical reading or sequential vertical reading of the instruction set related to the SME instruction set and / or SVE instruction set, to obtain the third sub-matrix data corresponding to the U component. Parallel vertical reading can refer to the ability to read multiple columns of data in parallel.

[0370] With the first angle being 180°, parallel horizontal reading or sequential horizontal reading based on the instruction set related to the SME instruction set and / or SVE instruction set, combined with vector flipping, can rotate the second sub-matrix data corresponding to the U component by 180° to obtain the third sub-matrix data corresponding to the U component.

[0371] It should be noted that, Figure 9 This is merely an example and does not constitute a limitation on the embodiments of this application. Furthermore, for further explanation of how to implement the visual rotation process based on instruction sets related to the SME instruction set and / or SVE instruction set, please refer to the following description of the visual rotation process, for example, regarding... Figure 17A and Figure 17B Explanation.

[0372] Regarding how to implement a component sampling process based on instruction sets related to the SME instruction set and / or SVE instruction set, it was found that for upsampling, it is possible to use the matrix external accumulation instruction set related to the SME instruction set. Based on at least one first element value corresponding to the first component C, the first component C is upsampled to obtain the third matrix data corresponding to the first component C. The third number can be the number of element values ​​in the third matrix data corresponding to the first component C. The element values ​​in the third matrix data corresponding to the first component C can be the first element value or the element value obtained after upsampling.

[0373] If the first component C satisfies the condition described above that the first number is less than the second number and the first component C is upsampled, the third number can be equal to the second number.

[0374] The instruction set can be used to instruct the data of the second submatrix corresponding to the first component C to be expanded by a factor of R in the first direction and / or by a factor of S in the second direction. Optionally, the instruction set can be used to instruct the data of the first submatrix corresponding to the first component C to be expanded by a factor of R in the first direction and / or by a factor of S in the second direction. R and S can be greater than or equal to 1. The first direction can be a vertical direction and the second direction can be a horizontal direction. Optionally, the first direction can be a horizontal direction and the second direction can be a vertical direction.

[0375] To expand the data dimension of the second sub-matrix data or the first sub-matrix data corresponding to the first component C by a factor of R in the first direction, it was found that a fourth matrix data determined by an interpolation method can be used. The fourth matrix data can be the interpolation matrix data in the first direction corresponding to the second or first sub-matrix data. The fourth matrix data can be used to expand the data dimension of the second or first sub-matrix data by a factor of R in the first direction. If the first direction is horizontal, the fourth matrix data can also be called horizontal interpolation matrix data. If the first direction is vertical, the fourth matrix data can also be called vertical interpolation matrix data.

[0376] Similarly, to expand the data dimension of the second sub-matrix data or the first sub-matrix data corresponding to the first component C by a factor of S in the second direction, it is found that this can be achieved using the fifth matrix data determined by interpolation methods. The fifth matrix data can be the interpolation matrix data in the second direction corresponding to the second or first sub-matrix data. The fifth matrix data can be used to expand the data dimension of the second or first sub-matrix data by a factor of S in the second direction. If the second direction is vertical, the fifth matrix data can also be called the vertical interpolation matrix data. If the second direction is horizontal, the fifth matrix data can also be called the horizontal interpolation matrix data.

[0377] To better understand the component sampling process described in the embodiments of this application, the following example uses the YUV color space as the first color space and the YUV color space color encoding format as YUV 420. Figure 10 This will be explained further. In this case, the U and V components are the first component, C. Furthermore, for an explanation of the color encoding format YUV 420 for the YUV color space, please refer to the above. Figure 3A The details regarding YUV 420 will not be repeated here. Figure 10 The value of U can be used to indicate the value of the U component.

[0378] Figure 10This is a schematic diagram of a component sampling process based on an instruction set related to the SME instruction set and / or SVE instruction set, provided for an embodiment of this application.

[0379] like Figure 10 As shown, since every four pixels share the same U and V component values, to ensure that each pixel has its own corresponding U and V component values, the second sub-matrix data corresponding to the U component can be doubled in both the horizontal and vertical directions. Alternatively, the first sub-matrix data corresponding to the U component can be doubled in both the horizontal and vertical directions. This yields the third matrix data corresponding to the U component. Similarly, the third matrix data corresponding to the V component can be obtained. The following explanation uses the method of obtaining the third matrix data corresponding to the U component as an example.

[0380] To achieve a doubling of both the horizontal and vertical data dimensions, it was found that this can be achieved by either accumulating the data outside the matrix between the second submatrix and the interpolation matrix, or by accumulating the data outside the matrix between the first submatrix and the interpolation matrix. The interpolation matrix data can include both horizontal and vertical interpolation matrix data.

[0381] To achieve a doubling of both the horizontal and vertical data dimensions based on the matrix extramatrix accumulation between the interpolation matrix data, the following method can be used.

[0382] The first method is based on first expanding horizontally and then vertically. It involves performing an extra-matrix accumulation between the second sub-matrix data and the horizontal interpolation matrix data, thus expanding the horizontal data dimension of the second sub-matrix data by a factor of R = 1. Then, it performs an extra-matrix accumulation between the horizontally expanded matrix data and the vertical interpolation matrix data, expanding the vertical data dimension of the second sub-matrix data by a factor of S = 1. This achieves doubling both the horizontal and vertical data dimensions of the second sub-matrix data. Similarly, a similar method can be used to double both the horizontal and vertical data dimensions of the first sub-matrix data.

[0383] The second method is based on first expanding vertically and then horizontally. First, perform extra-matrix accumulation between the second sub-matrix data and the vertical interpolation matrix data to expand the vertical dimension of the second sub-matrix data by a factor of S = 1. Then, perform extra-matrix accumulation between the vertically expanded matrix data and the horizontal interpolation matrix data to expand the horizontal dimension of the second sub-matrix data by a factor of R = 1. This achieves doubling both the horizontal and vertical dimensions of the second sub-matrix data. Similarly, a similar method can be used to double both the horizontal and vertical dimensions of the first sub-matrix data.

[0384] It should be noted that, Figure 10 This is merely an example and does not constitute a limitation on the embodiments of this application. Furthermore, for further explanation of how to implement the component sampling procedure based on the instruction sets related to the SME instruction set and / or SVE instruction set, please refer to the following description of the component sampling procedure, for example, regarding... Figure 18A and Figure 18B Explanation.

[0385] Regarding how to implement a component sampling process based on instruction sets related to the SME instruction set and / or SVE instruction set, it was found that downsampling can be achieved based on conditional loading. For example, based on the instruction set, the first component C is downsampled according to at least one first element value corresponding to the first component C to obtain third matrix data corresponding to the first component C. The third number can be equal to a predetermined number. The predetermined number can be configured according to actual business needs and is not limited here. The instruction set can be used to implement conditional loading. The first element value included in the third matrix data corresponding to the first component C is the first element value that needs to be subjected to the third loading operation. The first matrix data or second matrix data corresponding to the first component C includes the first element value that needs to be subjected to the third loading operation and the first element value that does not need to be subjected to the third loading operation.

[0386] Regarding how to implement a color space conversion process based on instruction sets related to the SME instruction set and / or SVE instruction set, it was found that it can be achieved by matrix-vector multiplication based on instruction sets related to the SME instruction set. For example, based on the instruction set, according to the ninth matrix data and the first vector data corresponding to each of the multiple pixels, the second visual data is obtained, thereby converting the first pixel value corresponding to the pixel into the second pixel value corresponding to the pixel.

[0387] The first matrix data may include first sub-matrix data corresponding to each of the multiple first components, that is, the first element values ​​corresponding to each of the multiple first components in the first matrix data are arranged consecutively, or the first element values ​​corresponding to each first component in the first matrix data are arranged consecutively. The first vector data corresponding to a pixel may be determined based on the first element values ​​corresponding to each of the multiple first components in the first matrix data corresponding to the pixel. For example, the first vector data corresponding to a pixel may include the third element values ​​corresponding to each of the multiple first components corresponding to the pixel. The third element value corresponding to the first component of the pixel may be determined based on the first element value corresponding to the first component of the pixel. The instruction set may be used to instruct the execution of matrix-external accumulation between the ninth matrix data and the first vector data. The ninth matrix data may be determined based on the transformation relationship between the multiple first components and the multiple second components.

[0388] When the first matrix data contains alternating first element values ​​corresponding to at least two first components, or when the second matrix data includes second sub-matrix data corresponding to multiple first components, second visual data can be obtained based on the instruction set, according to the ninth matrix data and the first vector data corresponding to multiple pixels. This allows the conversion of the first pixel value corresponding to a pixel into a second pixel value. The first vector data corresponding to a pixel can be determined based on the first element values ​​corresponding to the multiple first components in the second matrix data.

[0389] To better understand the color space conversion process described in the embodiments of this application, the following example uses YUV color space as the first color space, RGB color space as the second color space, first visual data as first image data, and second visual data as second image data. Figure 11 To explain. Regarding Figure 11 For an explanation of the first image data corresponding to the YUV color space, please refer to the above text. Figure 1 The explanation of image data corresponding to the YUV color space will not be repeated here.

[0390] Figure 11 This is a schematic diagram illustrating the color space conversion process based on instruction sets related to the SME instruction set and / or SVE instruction set, as provided in an embodiment of this application.

[0391] like Figure 11As shown, the second image data may include pixel values ​​corresponding to each of the 24 pixels, namely P25, ..., Pf, ..., P48, where f ∈ {25, ..., 48}. Pf may include the Rf component value corresponding to the R component, the Gf component value corresponding to the G component, and the Bf component value corresponding to the B component. The color component values ​​in the RGB color space may include the Rf component value corresponding to the R component, the Gf component value corresponding to the G component, and the Bf component value corresponding to the B component.

[0392] To convert color component values ​​from the YUV color space to the RGB color space, a matrix-vector multiplication can be implemented using instructions related to the SME instruction set. The matrix in the matrix-vector multiplication can be the transformation matrix data. The vector in the matrix-vector multiplication can be YUV vector data. The YUV vector data corresponding to a pixel can include the Y, U, and V component values ​​corresponding to that pixel. The YUV vector data can be the first vector data.

[0393] It should be noted that, Figure 11 This is merely an example and does not constitute a limitation on the embodiments of this application. Furthermore, for further explanation of how to implement the color space conversion process based on instruction sets related to the SME instruction set and / or SVE instruction set, please refer to the following description of the color space conversion process.

[0394] The inventive concept of the embodiments of this application has been described above. The data processing method provided by the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0395] First, combined Figure 12 and Figure 13 The electronic equipment to which the data processing method is applicable will be described. Next, an example will be used... Figure 12 and Figure 13 Taking the electronic device shown as an example, in conjunction with the accompanying drawings (e.g., Figure 14A , Figure 14B , Figure 15 , Figure 16A , Figure 16B , Figure 17A , Figure 17B , Figure 18A , Figure 18B , Figure 19 and Figure 20 The data processing methods provided in the embodiments of this application will be specifically described in terms of their application scenarios and usage scenarios.

[0396] It should be noted that the embodiments of this application can be implemented independently or in combination with each other, and the same or similar concepts or processes will not be described again in some embodiments.

[0397] The data processing method provided in this application can be used in electronic devices. Electronic devices may include handheld devices or vehicle-mounted devices with image processing capabilities. For example, some electronic devices include mobile phones, tablets, PDAs, laptops, mobile internet devices (MIDs), virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control (IC), wireless terminals in self-driving (SD), wireless terminals in remote medical surgery (RMS), wireless terminals in smart grids (SD), wireless terminals in transportation safety (TS), wireless terminals in smart cities (SC), wireless terminals in smart homes (SH), cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, in-vehicle devices, terminal devices in 5G networks, or future evolution of public land mobile communication networks (PLANs). Terminal devices in a network (PLMN), etc., are not limited to this in the embodiments of this application.

[0398] By way of example and not limitation, in this embodiment, the electronic device can also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, or shoes. Wearable devices are portable devices worn directly on the body or integrated into a user's clothing or accessories. Wearable devices are not merely hardware devices; they achieve powerful functions through software support, data interaction, and cloud interaction. Broadly defined, wearable smart devices include those with comprehensive functions, large size, and the ability to perform complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those focused on a specific application function that require interaction with other devices such as smartphones, such as various smart bracelets or smart jewelry for vital sign monitoring.

[0399] The electronic devices in the embodiments of this application may also be referred to as terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0400] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0401] Electronic device 1200 may include processor 1210, external memory interface 1220, internal memory 1221, universal serial bus (USB) interface 1230, charging management module 1240, power management module 1241, battery 1242, antenna 1, antenna 2, mobile communication module 1250, wireless communication module 1260, audio module 1270, speaker 1270A, receiver 1270B, microphone 1270C, headphone jack 1270D, sensor module 1280, button 1290, motor 1291, indicator 1292, camera 1293, display screen 1294, and subscriber identification module (SIM) card interface 1295, etc. The sensor module 1280 may include a pressure sensor 1280A, a gyroscope sensor 1280B, a barometric pressure sensor 1280C, a magnetic sensor 1280D, an accelerometer sensor 1280E, a distance sensor 1280F, a proximity sensor 1280G, a fingerprint sensor 1280H, a temperature sensor 1280J, a touch sensor 1280K, an ambient light sensor 1280L, or a bone conduction sensor 1280M, etc.

[0402] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 1200. In other embodiments of this application, the electronic device 1200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0403] Processor 1210 may include one or more processing units. For example, processor 1210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor, a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0404] The processor 1210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 1210 is a cache memory. This memory can store instructions or data that the processor 1210 has just used or that are used repeatedly. If the processor 1210 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 1210, and thus improves the efficiency of the system.

[0405] In some instances, the memory of this application embodiment may store instructions and data for implementing the data processing method described in the application embodiment, and when the processor is running, the data processing method described in the application embodiment can be implemented.

[0406] Internal memory 1221 can be used to store computer executable program code, which includes instructions. Internal memory 1221 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (e.g., data processing functions, etc.). The data storage area may store data created during the use of electronic device 1200. Furthermore, internal memory 1221 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 1210 executes various functional applications and data processing of electronic device 1200 by running instructions stored in internal memory 1221 and / or instructions stored in memory disposed in processor 1210. In this embodiment, internal memory 1221 can be used to implement the instructions and data of the data processing method described in this embodiment.

[0407] The software system of the electronic device 1200 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example, combined with... Figure 13 Please provide an explanation.

[0408] Figure 13 This is a software structure block diagram of an electronic device provided in an embodiment of this application.

[0409] A layered architecture divides software into multiple layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system may include an application layer, a Java layer, a Java Native Interface (JNI) layer, a native layer, and a hardware layer.

[0410] The application layer may include a series of application packages. For example, an application package may include at least one of a gallery application, an instant messaging application, a browser, an image editing application, or a camera application.

[0411] The Java layer can include audio and video encoding / decoding classes.

[0412] The Java Native Interface layer acts as a bridge between Java code and native code (C / C++), allowing Java to call native functions and vice versa.

[0413] JNI functions declare Java methods using the `native` keyword and implement these methods in the native layer.

[0414] The native layer can be written in C / C++ for high-performance computing, hardware operation, etc. Native libraries can be compiled into .so files and interact with the Java layer through the JNI layer. The native layer can support web-based instant messaging, LibAOM, Libyuv, X265, or acceleration libraries. LibAOM is an open-source software library used for encoding and decoding AV1 (Aomedia Video 1) video format. Libyuv can be used to implement YUV data format conversion, rotation, scaling, or mirroring. X265 is a library for encoding video into high-efficiency video encoding formats. The data processing methods proposed in this application can be deployed in the acceleration library to improve processing efficiency and reduce device power consumption.

[0415] The hardware layer may include a central processing unit (CPU). The CPU can be used to execute the data processing methods described in the embodiments of this application.

[0416] It should be noted that the embodiments of this application are only illustrated using the Android system. In other operating systems, if the functions implemented by each functional module are similar to those in the embodiments of this application, the image processing method described in the embodiments of this application can also be implemented.

[0417] The following is combined with Figure 14A , Figure 14B , Figure 15 , Figure 16A , Figure 16B , Figure 17A , Figure 17B , Figure 18A , Figure 18B , Figure 19 The data processing method provided in the embodiments of this application will be described. Among them, Figure 14A and Figure 14B This is used to explain certain terms involved in the embodiments of this application. Figure 15 This describes, as a whole, the data processing methods based on instruction sets related to the SME instruction set and / or SVE instruction set. Figures 16A-16B The process for separating components based on instruction sets related to the SME instruction set and / or SVE instruction set is described. Figures 17A-17B This is used to describe the visual rotation process based on the instruction set associated with the SME instruction set and / or the SVE instruction set. Figures 18A-18B This describes the component sampling process based on instruction sets related to the SME instruction set and / or SVE instruction set. Figure 19 This is a description of a second loading operation involving multiple pixels in a color conversion process.

[0418] To facilitate understanding, the following supplementary explanations are provided on the first matrix data, the second matrix data, and the first component C, in conjunction with the above text, as well as explanations of the first storage format, the second storage format, the first component D, the first condition, the second condition, and the second visual data mentioned below.

[0419] The first storage format can be half-plane storage or packed storage. The second storage format can be planar storage. Half-plane storage can refer to at least one of multiple first components having an independent storage area, with the first component values ​​corresponding to other first components alternately stored in the same storage area. Packed storage can refer to alternating the storage of the first component values ​​corresponding to each of the first components in the same storage area. Planar storage can refer to multiple first components each having an independent storage area. The storage area corresponding to each first component stores multiple first component values ​​corresponding to that first component, and these multiple first component values ​​are stored consecutively. This can be combined... Figure 3B , Figure 3C and Figure 3D Understand the first and second storage formats.

[0420] If the first matrix data contains alternating first element values ​​corresponding to at least two first components, then the storage format of the first matrix data is a first storage format. In other words, if the storage format of the first matrix data is a first storage format, then the storage format of the first matrix data contains alternating first element values ​​corresponding to at least two first components.

[0421] If the first element values ​​corresponding to each of the multiple first components in the first matrix data are arranged continuously or non-alternatingly, then the storage format of the first matrix data is the second storage format. In other words, if the storage format of the first matrix data is the second storage format, then the first element values ​​corresponding to each of the multiple first components in the first matrix data are arranged continuously or non-alternatingly.

[0422] If a first component A exists among multiple first components, then the first matrix data may include first sub-matrix data A corresponding to the first component A, meaning that at least one first element value included in the first sub-matrix data A corresponds to the first component A. When the first component is the first component A, the first sub-matrix data A may include at least one first element value corresponding to each of at least one first object. A first object may include first row data, first column data, or first matrix block data. First row data may be used to indicate row data corresponding to the first component A. First column data may be used to indicate column data corresponding to the first component A. First matrix block data may be used to indicate matrix block data corresponding to the first component A.

[0423] If multiple first components B exist among the multiple first components, the first matrix data may include at least one first element value corresponding to each of the multiple first components B. The first matrix data may include at least one first object corresponding to each of the multiple first components B. For any first object among the at least one first object, the first object may include at least one first element value corresponding to each of the multiple first components B, and the first components B corresponding to two adjacent first element values ​​in the first object are different. The second object may include second row data, second column data, or second matrix block data. The second row data may be used to indicate row data corresponding to the multiple first components B. The second column data may be used to indicate column data corresponding to the multiple first components B. The second matrix block data may be used to indicate matrix block data corresponding to the multiple first components B. Thus, the second row data may include at least one first element value corresponding to each of the multiple first components B, and the first components B corresponding to two adjacent first element values ​​in the second row data are different. The second column data may include at least one first element value corresponding to each of the multiple first components B, and the first components B corresponding to two adjacent first element values ​​in the second column data are different. The second matrix block data may include at least one first element value corresponding to each of the multiple first components B, and the first element values ​​of different first components B are alternately arranged in the second matrix block data.

[0424] For any first component B among a plurality of first components B, and for any first object among at least one first object, the first object includes at least one first element value corresponding to the first component B and at least one first element value corresponding to other first components B. Adjacent first element values ​​corresponding to the first component B in the first object are arranged alternately. The memory addresses of the at least one first element corresponding to the first component B in the first object are not contiguous. Other first components B may refer to first components B other than the first component B among the plurality of first components B.

[0425] Therefore, for any second row of data in at least one second row of data, the second row of data may include at least one first element value corresponding to the first component B and at least one first element value corresponding to other first components B. Adjacent first element values ​​corresponding to the first component B in the second row of data are arranged alternately. The memory addresses of the at least one first element value corresponding to the first component B in the first row of data are not contiguous. For any second column of data in at least one second column of data, the second column of data may include at least one first element value corresponding to the first component B and at least one first element value corresponding to other first components B. The at least one first element value corresponding to the first component B in the second column of data is arranged alternately. The memory addresses of the at least one first element value corresponding to the first component B in the second column of data are not contiguous. For any second matrix block of data in at least one second matrix block of data, the second matrix block of data may include at least one first element value corresponding to the first component B and at least one first element value corresponding to other first components B. Adjacent first element values ​​corresponding to the first component B in the second matrix block of data are arranged alternately. Adjacent first element values ​​may refer to first element values ​​located in the same row of the second matrix block of data. In the second matrix block data, the first element value located in the same column corresponds to the same first component B. Optionally, two adjacent first element values ​​can refer to the first element values ​​located in the same column of the second matrix block data. The first element values ​​located in the same row of the second matrix block data also correspond to the same first component B.

[0426] To facilitate a better understanding of the first object, first component A, first component B, first component C, and first component D described in the embodiments of this application, the following will use the YUV color space as the first color space, the YUV color space color encoding format as YUV 420, the first visual data as image data, and the storage format of the first matrix data as NV 12 as an example, combined with... Figure 14A and Figure 14B Please provide an explanation. Furthermore... Figure 14B For explanations regarding image data, please refer to the section above. Figure 1 The explanation, Figure 14B For a description of NV 12, please refer to the description of Figure 3 above, which will not be repeated here. Figure 14A and Figure 14B The Y component can be used to indicate the Y component value. The U component can be used to indicate the U component value. The V component can be used to indicate the V component value.

[0427] Figure 14A This is a schematic diagram of the first matrix data provided in an embodiment of this application.

[0428] like Figure 14AAs shown, the first matrix data can include M rows and N columns. M = 6. N = 6. Since the U component values ​​and V component values ​​are arranged alternately, the U component and V component constitute the first component B. Since the Y component values ​​are arranged continuously, the Y component constitutes the first component A.

[0429] When the first component is a Y component, the first matrix data may include at least one first object corresponding to the Y component. The first object may include first row data, first column data, or first matrix block data. For example, when the first object is first row data, the first row data may be the first, second, third, or fourth row data of the first matrix data. When the first object is first column data, the first column data may be the first, second, third, fourth, fifth, or sixth column data of the first matrix data. When the first object is first matrix block data, for example, the first matrix block data may be composed of the third and fourth rows of the first matrix data. The first matrix block data may also be composed in other ways, which are not limited in this embodiment.

[0430] When the first component is a U component or a V component, the first matrix data may include at least one first object corresponding to the U component and the V component. The first object may include at least one U component value (i.e., a first element value) corresponding to the U component and at least one V component value (i.e., a first element value) corresponding to the V component. The second object may include second row data, second column data, and second matrix block data. The second row data may be the 5th or 6th row data of the first matrix data. The second matrix block data may be composed of the 5th and 6th rows data of the first matrix data. The first matrix block data may also be composed in other ways, which are not limited in this embodiment. If the first matrix data is matrix transposed, the second column data may be the 5th or 6th column data of the transposed first matrix data. Figure 14A The second column of data is not shown.

[0431] It should be noted that Figure 14 is merely an example and does not constitute a limitation on the embodiments of this application.

[0432] The first condition can be used to indicate that the data dimensions of the second submatrix data are doubled in both the first and second directions. Optionally, the first condition can be used to indicate that the data dimensions of the first submatrix data are doubled in both the first and second directions.

[0433] The second condition can be used to indicate whether the data dimension of the second submatrix data is doubled in the first direction or doubled in the second direction. Optionally, the second condition can be used to indicate whether the data dimension of the first submatrix data is doubled in the first direction or doubled in the second direction.

[0434] The second matrix data can be obtained by separating the components of the first matrix data. Therefore, the storage format of the second matrix data can be the second storage format.

[0435] A first component D can be defined as one of multiple first components that satisfies the following conditions: The fourth number can be equal to the second number. The fourth number can be the number of first element values ​​corresponding to first component D in the second matrix data or the first matrix data. Therefore, if the fourth number is equal to the second number, the first component can be called first component D. Furthermore, the first element values ​​corresponding to first component D can be different for different pixels; that is, the first element values ​​corresponding to first component D can correspond one-to-one with pixels.

[0436] Furthermore, if the first component A satisfies the condition that the fourth number is the same as the second number, then the first component A can be called the first component D.

[0437] For example, if the first color space is the YUV color space, and the color encoding format of the YUV color space is YUV420, the storage format of the first matrix data can be a first storage format. The first matrix data may include at least one first element value corresponding to the Y component, at least one first element value corresponding to the U component, and at least one first element value corresponding to the V component. The Y component can be called first component A. The U component and V component can be called first component B.

[0438] Since the number of first element values ​​corresponding to the Y component is equal to the number of pixels included in the first visual data (i.e., the third number is equal to the second number), the Y component can be called the first component D. Since the number of first element values ​​corresponding to the U component is less than the number of pixels included in the first visual data (i.e., the first number is less than the second number), the U component can be called the first component C. Since the number of first element values ​​corresponding to the V component is less than the number of pixels included in the first visual data (i.e., the first number is less than the second number), the V component can be called the first component C.

[0439] Figure 14B A schematic diagram of the first component C and the first component D provided in the embodiments of this application.

[0440] like Figure 14B As shown, since the number of Y component values ​​(i.e., the first element values ​​corresponding to the Y component) is equal to the number of pixels included in the image data, the Y component is the first component D. Since the number of U component values ​​(i.e., the first element values ​​corresponding to the U component) is less than the number of pixels included in the image data, the U component is the first component C. Since the number of V component values ​​(i.e., the first element values ​​corresponding to the V component) is less than the number of pixels included in the image data, the V component is the first component C.

[0441] Second visual data can refer to visual data corresponding to a second color space. A second color space can include multiple second components. Each second component can be a color component of the second color space. Each second component has a corresponding second component value. Second visual data can include second pixel values ​​corresponding to multiple pixels. Second pixel values ​​can include second component values ​​corresponding to each of the multiple second components. For example, the second color space can be the RGB color space. The multiple second components can include R components, G components, and B components. Second pixel values ​​can include the R component value corresponding to the R component, the G component value corresponding to the G component, and the B component value corresponding to the B component. The R component value, G component value, and B component value can be referred to as second component values.

[0442] The second visual data may have corresponding tenth matrix data. The tenth matrix data may include at least one second element value corresponding to each of the multiple second components. The second element value may be used to indicate the second component value.

[0443] Figure 15 This is a schematic diagram illustrating the principle of another data processing method based on an instruction set related to the SME instruction set and / or SVE instruction set, provided for embodiments of this application. It should be noted that this can be combined with... Figure 7 understand.

[0444] like Figure 15 As shown, the first matrix data corresponding to the first visual data can be processed based on the instruction set associated with the SME instruction set and / or the SVE instruction set to realize at least one of the component separation process, visual geometry transformation process, component sampling component or color space transformation process.

[0445] As one implementation method, component separation process, visual geometry transformation process, component sampling process, or color space transformation process can be executed based on the instruction set.

[0446] As another implementation, the following sequence can be executed based on the instruction set: component separation process → visual geometry transformation process, component separation process → component sampling process, component separation process → color space transformation process, visual geometry transformation process → component sampling process, visual geometry transformation process → color space transformation process, or component sampling process → color space transformation process.

[0447] As another implementation, the following order can be executed based on the instruction set: component separation process → visual geometric transformation process → component sampling process, component separation process → visual geometric transformation process → color space conversion process, component separation process → component sampling process → color space conversion process, or visual geometric transformation process → component sampling process → color space conversion process.

[0448] As another implementation method, it can be executed in the following order based on the instruction set: component separation process → visual geometric transformation process → component sampling process → color space conversion process.

[0449] For the component separation process, please refer to the following section. Figures 16A-16B The explanation below details the visual geometric transformation process. Figures 17A-17B The following explanation details the component sampling process. Figures 18A-18B Explanation.

[0450] The component separation process, visual geometric transformation process, component sampling process, and color space conversion process are described below with reference to the accompanying drawings. These processes can be executed by a processor. The processor may include a two-dimensional register, a first register, a second register, and a third register.

[0451] For the component separation process, at least one first object corresponding to the first component in the first matrix data can be loaded from memory into a two-dimensional register based on at least one first instruction corresponding to the first component, so as to obtain the second matrix data stored in the two-dimensional register.

[0452] According to embodiments of this application, when the first component is first component A, the first object corresponding to first component A may include first row data corresponding to first component A. Optionally, the first object corresponding to first component A may include first column data corresponding to first component A. Optionally, the first object corresponding to first component A may include first matrix block data corresponding to first component A. First component A can be used to indicate a separated first component among multiple first components. For descriptions of first component A, the first object corresponding to first component A, the first row data, the first column data, and the first matrix block data, please refer to the corresponding sections above, and they will not be repeated here.

[0453] When the first component is component B, the first object may include at least one first element value corresponding to component B in the second row of data. The memory addresses of the at least one first element value corresponding to component B in the second row of data are not contiguous. Optionally, the first object may include at least one first element value corresponding to component B in the second column of data. The memory addresses of the at least one first element value corresponding to component B in the second column of data are not contiguous. Optionally, the first object may include at least one first element value corresponding to component B in the second matrix block data. The memory addresses of the at least one first element value corresponding to component B in the second matrix block data are not contiguous.

[0454] Regarding how to load at least one first object corresponding to the first component from memory into a two-dimensional register based on at least one first instruction corresponding to the first component, and obtain the second matrix data stored in the two-dimensional register, the following section discusses... Figures 16A-16B Please provide an explanation.

[0455] When the first component is the first component A, based on at least one first instruction corresponding to the first component A, at least one first row data, at least one first column data, or at least one first matrix block data corresponding to the first component A are loaded from memory into a two-dimensional register to obtain the second sub-matrix data corresponding to the first component A stored in the two-dimensional register.

[0456] The first element value in the second submatrix data corresponding to the first component A is either arranged continuously or non-alternatingly.

[0457] When the first object includes a first row of data, the first instruction can be a horizontal loading instruction related to the SME instruction set. At least one first instruction can be used to instruct the loading of at least one first row of data in parallel or sequentially. When the first object includes a first column of data, the first instruction can be a vertical loading instruction related to the SME instruction set. At least one instruction can be used to instruct the loading of at least one first column of data in parallel or sequentially. When the first object includes first matrix block data, the first instruction can be a block loading instruction related to the SME instruction set. At least one first instruction can be used to instruct the loading of at least one first matrix block of data in parallel or sequentially.

[0458] When the first component is the first component B, based on at least one first instruction corresponding to the first component B, at least one first object corresponding to the first component B is loaded from memory into a two-dimensional register to obtain the second sub-matrix data corresponding to the first component B stored in the two-dimensional register.

[0459] The first element value in the second submatrix data corresponding to the first component B is either arranged continuously or non-alternatingly.

[0460] The first instruction can be a scatter load instruction associated with the SME instruction set and / or the SVE instruction set. When the first object includes a first element value corresponding to the first component B in the second row of data, at least one first instruction can be used to instruct the loading of at least one first element value corresponding to the first component B in at least one second row of data from memory into a two-dimensional register. At least one second row of data can be executed in parallel or sequentially. When the first object includes a first element value corresponding to the first component B in the second column of data, at least one first instruction can be used to instruct the loading of at least one first element value corresponding to the first component B in at least one second column of data from memory into a two-dimensional register. At least one second column of data can be executed in parallel or sequentially. When the first object includes a first element value corresponding to the first component B in the second matrix block of data, at least one first instruction can be used to instruct the loading of at least one first element value corresponding to the first component B in at least one second matrix block of data from memory into a two-dimensional register. At least one second matrix block of data can be executed in parallel or sequentially.

[0461] The following describes how to load at least one first object corresponding to the first component from memory into a two-dimensional register based on at least one first instruction corresponding to the first component.

[0462] First, the relationship between the first object and the first instruction is explained, including how to determine the first address corresponding to the first object and how to control whether the first element value is loaded. Then, the continuous loading of the first component A and the distributed loading of the first component B are explained separately.

[0463] The relationship between the first object and the first instruction is explained below.

[0464] A first instruction corresponding to a first component may include at least one. At least one first instruction corresponding to a first component can be used to process at least one first object corresponding to the first component. When the first component is first component A, the first object may include first row data, first column data, or first matrix block data. When the first component is first component B, the first object may include second row data, second column data, or second matrix block data. One instruction may correspond to P first objects, meaning one instruction can be used to process P first objects. The P first objects may be all or part of at least one first object. P may be an integer greater than or equal to 1 and less than or equal to Q. Q may be the number of first objects corresponding to the first component. Q may be an integer greater than or equal to 1. Thus, the relationship between a first instruction and a first object can be one-to-one, meaning at least one first object has its own corresponding first instruction. Optionally, the relationship between a first instruction and a first object can be one-to-many, meaning P first objects correspond to the same first instruction.

[0465] Furthermore, different first components B can have the same first instruction, that is, at least one first object corresponding to each of the multiple first components B can be processed using at least one first instruction.

[0466] The following explains how to determine the first address corresponding to the first object.

[0467] When the first object is the first row of data, the first address corresponding to the first object can be the address of the first row of data. The address of the first row of data can be used to indicate the base address (or starting address) of that first row of data. The address of the first row of data can be determined based on the first base address, the first row index, and the first row step size. The first base address can be the base address (or starting address) of the first submatrix data A or the base address of the first matrix data. The first row index can be the row index corresponding to the first row of data. The first row step size can be the address offset between two adjacent rows in the first submatrix data A. Optionally, the address of the first row of data can be determined based on the first base address and the first offset. The first offset can be determined based on the first row index and the first row step size. Optionally, the address of the first row of data can be determined based on the first base address, the first row index, the first number, and the first element size. The first number can be the number of first element values ​​included in the first row of data. The first element size can be used to indicate the number of bytes occupied by the first element value.

[0468] When the first object is the first column of data, the first address corresponding to the first object can be the first column address corresponding to the first column of data. The first column address corresponding to the first column of data can be used to indicate the base address (or starting address) of that first column of data. The first column address corresponding to the first column of data can be determined based on the first base address, the first column index, and the first column step size. The first column index can be the column index corresponding to that first column of data. The first column step size can be the address offset between two adjacent columns in the first submatrix data A.

[0469] When the first object is a first matrix block of data, the first address corresponding to the first object can be the first block address corresponding to the first matrix block of data. The first block address corresponding to the first matrix block of data can be used to indicate the base address (or starting address) of that first block of data. The first block address corresponding to the first matrix block of data can be determined based on the first base address, the second row index, the first row step size, the second column index, and the size of the first element. The second row index can be the starting row index of the first sub-matrix data A. The second column index can be the starting column index of the first sub-matrix data A. Optionally, the first block address corresponding to the first matrix block of data can be determined based on the first base address, the second column index, the first column step size, the second row index, and the size of the first element.

[0470] The following explains how to control whether the first element value is loaded.

[0471] As one implementation, based on the combination of a fourth identifier and an address, that is, after determining the memory address of the first element value for which the first load operation needs to be performed, the value of the first identifier can also be used to indicate which first element values ​​among the at least one first element value corresponding to the first object are valid element values ​​and which are invalid element values, so that the first load operation is performed on the valid element values ​​without performing the first load operation on the invalid element values. The fourth identifier can be a first value or a second value. A first value can be used to indicate that the first load operation is performed on the first element value corresponding to the first component. If the fourth identifier corresponding to the first element value corresponding to the first component is a first value, it indicates that the first element value is a valid element value, and therefore, the first load operation needs to be performed on the first element value. A second value can be used to indicate that the first load operation is not performed on the first element value corresponding to the first component. If the fourth identifier corresponding to the first element value corresponding to the first component is a second value, it indicates that the first element value is an invalid element value, and therefore, the first load operation does not need to be performed on the first element value. There can be at least one fourth identifier corresponding to the first object. At least one fourth identifier can be stored in a first register. The first register can be the predicate register described above. For example, the first register can store at least one fourth identifier corresponding to the first object. The fourth identifier can be understood as a Boolean value. As one implementation, the first value can be "1" and the second value can be "0".

[0472] For any first component among multiple first components, when the first component is first component A, the first element value in the first object corresponding to first component A all correspond to first component A. That is, the first element value included in the first row of data all corresponds to first component A. The first element value included in the first column of data all corresponds to first component A. The first element value included in the first matrix block of data all corresponds to first component A. Since the first element value corresponding to first component A needs to be subjected to the first loading operation, the fourth identifier corresponding to first component A is the first value.

[0473] When the first component is a first component B, the first element value in the first object corresponding to the first component B is either corresponding to the first component B or corresponding to other first components B. That is, the first object may include at least one first element value corresponding to the first component B and at least one first element value corresponding to other first components B. The second row of data may include at least one first element value corresponding to the first component B and at least one first element value corresponding to other first components B. The second column of data may include at least one first element value corresponding to the first component B and at least one first element value corresponding to other first components B. The second matrix block data may include at least one first element value corresponding to the first component B and at least one first element value corresponding to other first components B. Since the first element value corresponding to the first component B needs to undergo a first loading operation, while the first element value corresponding to other first components B does not need to undergo a first loading operation, in at least one fourth identifier corresponding to the first component B, the fourth identifier corresponding to the first component B is a first value and the fourth identifier corresponding to other first components B is a second value. As one implementation, the first value can be 1. The second value can be 0. Optionally, the first value can be 0. The second value can be 1.

[0474] As an alternative implementation, the address-based approach determines the memory address of the first element value for which the first load operation needs to be performed without using a condition mask.

[0475] The following explanation addresses the continuous loading of the first component A and the distributed loading of the first component B. Since continuous loading can be divided into row-based continuous loading and column-based continuous loading, and distributed loading can be divided into row-based distributed loading and column-based distributed loading, the explanation will focus on the row-based and column-based methods.

[0476] The following section explains the continuous loading of the first component A.

[0477] 1. Continuous loading in row-by-row mode

[0478] In one implementation, the first object may include a first row of data. The two-dimensional register may include a first region corresponding to at least one first row of data. The first region corresponding to the first row of data can be used to store that first row of data. The first region may be a portion of the two-dimensional register. Different first regions may not overlap.

[0479] The first instruction may include a first identifier, at least one second identifier, and at least one third identifier. The first identifier may be used to indicate a first load operation. For example, the first load operation may be a horizontal load operation. The second identifier may be used to indicate a first region. The at least one second identifier may include a second identifier corresponding to each of the at least one first region. The at least one third identifier may be used to determine a first address corresponding to a first object.

[0480] Therefore, a first address corresponding to each of the at least one first row of data can be determined based on at least one third identifier. Based on the first identifier and the first address corresponding to each of the at least one first row of data, the first row of data corresponding to each of the at least one first address is retrieved from memory. For example, for any first row of data among the at least one first row of data, the first row of data corresponding to that first address can be retrieved from memory based on the first address corresponding to that first row of data. Based on the first identifier, the first row of data corresponding to each of the at least one first address is loaded from memory into a first region corresponding to each of the at least one second identifier. For example, for any first address among the at least one first address, the first row of data corresponding to that first address is loaded from memory into a first region corresponding to the second identifier.

[0481] To better understand the row-by-row continuous loading described in the embodiments of this application, the following example uses the first component A as the Y component, combined with... Figure 16A Please provide an explanation. Figure 16A Y can be used to indicate the value of the Y component (i.e., the value of the first element corresponding to the Y component).

[0482] Figure 16A This is a schematic diagram illustrating the sequential loading of instruction sets based on the SME instruction set and / or SVE instruction set, provided for embodiments of this application.

[0483] like Figure 16A As shown, the first matrix data may include the first sub-matrix data corresponding to the Y component. The first sub-matrix data corresponding to the Y component may include M rows and N columns. M = 4. N = 8. The first row of data can be the 1st, 2nd, 3rd, or 4th row of the first sub-matrix data. The first column of data can be the 1st, 2nd, 3rd, 4th, 5th, 6th, 7th, and 8th columns of the first sub-matrix data.

[0484] It should be noted that, Figure 16A This is merely an example and does not constitute a limitation on the embodiments of this application.

[0485] To load multiple first element values ​​corresponding to the Y component from memory into a two-dimensional register, the following method can be used.

[0486] As one implementation, multiple first-row data corresponding to the Y component can be loaded from memory into a two-dimensional register based on at least one horizontal load instruction. The at least one first horizontal load instruction can be used to instruct parallel loading or sequential loading of at least one first-row data.

[0487] As an alternative implementation, multiple first column data corresponding to the Y component can be loaded from memory into a two-dimensional register based on at least one vertical load instruction. The at least one vertical load instruction can be used to instruct parallel loading or sequential loading of at least one first column data.

[0488] As another implementation, the processor may further include a first register. The first register may be used to store at least one fourth identifier corresponding to the first row of data. Of the at least one fourth identifier, the fourth identifier corresponding to the first component A has a first value, and the fourth identifiers corresponding to the other first components have second values. The first value is used to indicate that a first load operation is performed for the first element value corresponding to the first component A. The second value is used to indicate that a first load operation is not performed for the first element values ​​corresponding to the other first components. In this case, all fourth identifiers corresponding to the first row of data have the first value.

[0489] The first instruction may also include a fifth identifier. The fifth identifier may be used to indicate the first register.

[0490] Therefore, a first address corresponding to each of the at least one first row of data can be determined based on at least one third identifier. At least one fourth identifier corresponding to each of the at least one first row of data is obtained from the first register corresponding to the fifth identifier. Based on the first identifier and the at least one fourth identifier corresponding to each of the at least one first object, the first row of data corresponding to each of the at least one first address is loaded into a first region corresponding to each of the at least one second identifier. For example, the first row of data with the fourth identifier being a first value is loaded into the first region corresponding to the second identifier.

[0491] In addition, it is possible to perform consecutive loading of multiple first rows of data in parallel or sequentially perform consecutive loading of the first row of data.

[0492] 2. Continuous loading in column mode

[0493] In one implementation, the first object may include a first column of data. The two-dimensional register may include a first region corresponding to at least one column of data. The first region corresponding to the first column of data can be used to store that first column of data.

[0494] The first instruction may include a first identifier, at least one second identifier, and at least one third identifier. The first identifier may be used to indicate a first load operation. For example, the first load operation may be a vertical load operation. At least one third identifier may be used to determine a first address corresponding to the first column of data.

[0495] Therefore, based on at least one third identifier, a first address corresponding to each of the at least one first column of data can be determined. Based on the first identifier and the first address corresponding to each of the at least one first column of data, the first column of data corresponding to each of the at least one first address is retrieved from memory. Based on the first identifier, the first column of data corresponding to each of the at least one first address is loaded from memory into a first region corresponding to each of the at least one second identifier. It should be noted that for the second identifier, how to retrieve the first column of data corresponding to each of the at least one first address from memory, and how to load the data from memory into the first region corresponding to the second identifier, please refer to the above explanation of continuous loading in row-by-row mode, which will not be repeated here.

[0496] As another implementation, the processor may further include a first register. The first register may be used to store at least one fourth identifier corresponding to the first column of data. Of the at least one fourth identifier, the fourth identifier corresponding to the first component A has a first value, and the fourth identifiers corresponding to the other first components have second values. The first value is used to indicate that a first load operation is performed for the first element value corresponding to the first component A. The second value is used to indicate that a first load operation is not performed for the first element values ​​corresponding to the other first components. In this case, all fourth identifiers corresponding to the first row of data have the first value.

[0497] The first instruction may also include a fifth identifier. The fifth identifier may be used to indicate the first register.

[0498] Therefore, a first address corresponding to each of the at least one first column of data can be determined based on at least one third identifier. At least one fourth identifier corresponding to each of the at least one first column of data is obtained from the first register corresponding to the fifth identifier. Based on the first identifier and the at least one fourth identifier corresponding to each of the at least one first object, the first column of data corresponding to each of the at least one first address is loaded into a first region corresponding to each of the at least one second identifier. For example, the first column of data with the fourth identifier being a first value is loaded into the first region corresponding to the second identifier.

[0499] In addition, it is possible to perform consecutive loading of multiple first column data in parallel or sequentially perform consecutive loading of first column data.

[0500] The following section explains the distributed loading of the first component B.

[0501] 1. Distributed loading by row

[0502] In one implementation, the processor may further include a first register. The first register may be used to store at least one fourth identifier corresponding to the second row of data. Of the at least one fourth identifier, the fourth identifier corresponding to the first component B has a first value, and the fourth identifiers corresponding to other first components B have second values. The first value is used to indicate that a first load operation is performed for the first element value corresponding to the first component B. The second value is used to indicate that a first load operation is not performed for the first element value corresponding to other first components B. In this case, the fourth identifier corresponding to the second row of data is either the first value or the second value.

[0503] The first instruction may also include a fifth identifier. The fifth identifier may be used to indicate the first register.

[0504] Therefore, a first address corresponding to each of the at least one second row of data can be determined based on at least one third identifier. At least one fourth identifier corresponding to each of the at least one second row of data is obtained from the first register corresponding to the fifth identifier. Based on the first identifier and the at least one fourth identifier corresponding to each of the at least one first object, the second row of data corresponding to each of the at least one first address is loaded into the first region corresponding to each of the at least one second identifier. For example, the second row of data with the fourth identifier being a first value is loaded into the first region corresponding to the second identifier.

[0505] In addition, it is possible to perform consecutive loading of multiple second rows of data in parallel or sequentially.

[0506] To better understand the row-by-row distributed loading described in the embodiments of this application, the following examples will be used, where the first component B is the U component and the other first components B are V components, or the first component B is the V component and the other first components B are U components. Figure 16B Please provide an explanation. Figure 16B In this context, U can be used to indicate the U component value (i.e., the first element value corresponding to the U component). V can be used to indicate the V component value (i.e., the first element value corresponding to the V component).

[0507] Figure 16B This is a schematic diagram illustrating the distributed loading of instruction sets based on the SME instruction set and / or SVE instruction set, provided for embodiments of this application.

[0508] like Figure 16BAs shown, the second row of data can include multiple U component values ​​corresponding to the U component and multiple V component values ​​corresponding to the V component. The multiple U component values ​​corresponding to the U component can include U00, U02, U04, U06, U20, U22, U24, and U26 component values. The multiple V component values ​​corresponding to the V component can include V00, V02, V04, V06, V20, V22, V24, and V26 component values. The U component values ​​and V component values ​​are arranged alternately.

[0509] The predicate register can be used to store a fourth identifier corresponding to each of the multiple component values. If the fourth identifier is a first value, the first value can be used to indicate that no loading operation is required for the component value. If the fourth identifier is a second value, the second value can be used to indicate that a loading operation is required for the component value. As one implementation, the first value can be "1" and the second value can be "0".

[0510] To load multiple U component values ​​corresponding to the U component in the second row of data from memory into the two-dimensional register, based on the above description of the value of the fourth identifier, the fourth identifier corresponding to each of the multiple U component values ​​in the predicate register can be configured to the first value "1". The fourth identifier corresponding to each of the multiple V component values ​​can be configured to the second value "0", thus the predicate register can store "10101010".

[0511] In this scenario, based on the "10101010" stored in the predicate register, the first value "1" corresponds to the U component value, and the second value "0" corresponds to the V component value. The first load operation for the U component value is executed, but the first load operation for the V component value is not executed. That is, the first load operations for the U00, U02, U04, U06, U20, U22, U24, and U26 component values ​​are executed, but the first load operations for the V00, V02, V04, V06, V20, V22, V24, and V26 component values ​​are not executed. This allows the predicate register to load multiple U component values ​​corresponding to the U component from memory into the two-dimensional register. The fourth identifier stored in the predicate register can be reused; that is, the same set of fourth identifiers can be used for each second row of data. Furthermore, parallel loading of multiple second rows of data can be performed.

[0512] To load multiple V component values ​​corresponding to the V component in the first data row from memory into the two-dimensional register, a similar method to that described above for loading multiple U component values ​​corresponding to the U component in the second data row from memory into the two-dimensional register can be used. In this case, the predicate register can store "01010101". Other explanations can be found in the corresponding sections above and will not be repeated here.

[0513] It should be noted that, Figure 16B This is merely an example and does not constitute a limitation on the embodiments of this application.

[0514] 2. Distributed loading in column mode

[0515] In one implementation, the processor may further include a first register. The first register may be used to store at least one fourth identifier corresponding to the second column of data. Of the at least one fourth identifier, the fourth identifier corresponding to the first component B has a first value, and the fourth identifiers corresponding to other first components B have second values. The first value is used to indicate that a first load operation is performed for the first element value corresponding to the first component B. The second value is used to indicate that a first load operation is not performed for the first element value corresponding to other first components B. In this case, the fourth identifier corresponding to the second column of data is either the first value or the first value.

[0516] The first instruction may also include a fifth identifier. The fifth identifier may be used to indicate the first register.

[0517] Therefore, a first address corresponding to each of the at least one second column of data can be determined based on at least one third identifier. At least one fourth identifier corresponding to each of the at least one second column of data is obtained from the first register corresponding to the fifth identifier. Based on the first identifier and the at least one fourth identifier corresponding to each of the at least one first object, the second column of data corresponding to each of the at least one first address is loaded into the first region corresponding to each of the at least one second identifier. For example, the second column of data with the fourth identifier as the first value is loaded into the first region corresponding to the second identifier.

[0518] In addition, it is possible to perform consecutive loading of multiple second column data in parallel or sequentially perform consecutive loading of second column data.

[0519] The component separation process has been described above. Based on this process, the second matrix data corresponding to the first matrix data is stored in a two-dimensional register. The second matrix data may include second sub-matrix data corresponding to each of the multiple first components. The second sub-matrix data corresponding to each first component may include multiple first element values ​​corresponding to the first component. The first element values ​​included in the second sub-matrix data may be arranged continuously or non-alternatingly. The storage format of the second sub-matrix data can be a second storage format, i.e., planar storage. The second matrix data may include M rows and N columns. M and N can be integers greater than 1.

[0520] The following is combined with Figure 17A and Figure 17B The visual rotation process is described below. It should be noted that the visual rotation process can be implemented based on the component separation process. Optionally, the visual rotation process can be implemented independently.

[0521] The visual rotation process can be implemented in the following way.

[0522] The first approach is a visual rotation process based on the second matrix data.

[0523] Based on the instruction set, the second sub-matrix data obtained from the two-dimensional register and corresponding to each of the multiple first components can be rotated by a first angle to obtain the third sub-matrix data stored in memory and corresponding to each of the multiple first components.

[0524] According to embodiments of this application, the instruction set can be used to indicate rotation instruction sets related to the SME instruction set and / or SVE instruction set; that is, the instruction set can be the SME instruction set and / or SVE instruction set related to implementing rotation angles. The storage format of the third submatrix data in memory can be configured according to actual business needs and is not limited here. For example, the third submatrix data can be stored in row-major order. Optionally, the third submatrix data can be stored in column-major order.

[0525] The following explanation addresses how to implement rotation of the first angle based on the instruction set when the first angle is 90°, 180°, or 270°. Specifically, it explains how to rotate the second sub-matrix data corresponding to any one of multiple first components by the first angle to obtain the third sub-matrix data corresponding to that first component.

[0526] For cases where the first angle is 90°, 180°, or 270°, it can be achieved in the following way.

[0527] As one implementation method, the second submatrix data corresponding to the first component is obtained from the two-dimensional register in the first manner, and the second submatrix data corresponding to the first component is stored in memory in the second manner to obtain the third submatrix data corresponding to the first component.

[0528] The first and second methods can be determined according to the instruction set. The first and second methods can be used to make the first angle 90°, 180°, or 270°, that is, the first and second methods can be used to achieve a rotation angle of 90°, 180°, or 270° between the second sub-matrix data corresponding to the first component and the third sub-matrix data corresponding to the first component. The instruction set may include a second instruction and a third instruction.

[0529] When the first angle is 90°, the second instruction can be used to indicate that the first method is to read in parallel or sequentially, from column 0 to column (N-1), with each column reading from the first element value of row (M-1) to the first element value of row 0. Specifically, the second instruction can indicate that the first method is a column-priority reverse vertical reading method, a column-priority reverse access method, a column-priority row-priority reverse access method, or a column-wise reverse access method, and that at least one column can be read in parallel or sequentially. The first element value of row (M-1) in each column can refer to the last first element in that column. The first element value of row 0 in each column can refer to the first first element value in that column.

[0530] The third instruction can be used to instruct the second method to store the first element values ​​obtained according to the first method in the order of the first element values, and to store them as N rows and M columns of third sub-matrix data. Thus, it is possible to rotate the second sub-matrix data corresponding to the first component by 90° to obtain the third sub-matrix data corresponding to the first component. The third sub-matrix data can include N rows and M columns. The nth row of the third sub-matrix data can be the reverse of the nth column of the second sub-matrix data. n can be an integer greater than or equal to 0 and less than or equal to N-1.

[0531] When the first angle is 270°, the second instruction can be used to indicate that the first mode is to read in parallel or sequentially, from column N-1 to column 0, with each column reading from the first element value of row 0 to the first element value of row M-1. That is, the second instruction can be used to indicate that the first mode is to read in reverse column order vertically or in reverse column order row priority access, and at least one row can be read in parallel or at least one row can be read sequentially.

[0532] Similarly, the third instruction can be used to instruct the second method to store the first element values ​​obtained according to the first method in order, and to store them as a third submatrix of N rows and M columns. Thus, it is possible to rotate the second submatrix data corresponding to the first component by 270° to obtain the third submatrix data corresponding to the first component. The third submatrix data can include N rows and M columns. The nth row of the third submatrix data can be the (N-1)-nth column of the second submatrix data. n can be an integer greater than or equal to 0 and less than or equal to N-1.

[0533] With the first angle being 180°, the second instruction can be used to indicate that the first method is to read in parallel or sequentially, from row M-1 to row 0, with each row reading from the first element value of column N-1 to the first element value of column 0, and at least one row being read in parallel or sequentially. The first element value of column N-1 in each row can refer to the last first element value in each row. The first element value of column 0 in each row can refer to the first first element value in each row.

[0534] The third instruction can be used to instruct the second method to store the first element values ​​obtained according to the first method in order, and to store them as a third sub-matrix data with M rows and N columns. Thus, it is possible to rotate the second sub-matrix data corresponding to the first component by 180° to obtain the third sub-matrix data corresponding to the first component. The third sub-matrix data may include M rows and N columns.

[0535] For the case where the first angle is 180°, it can also be implemented in the following way. As another implementation method, the second sub-matrix data corresponding to the first component is obtained from the two-dimensional register according to the third method, the obtained second sub-matrix data is flipped, and the flipped second sub-matrix data is stored in memory according to the fourth method.

[0536] The third and fourth methods can be determined based on the instruction set. The third, flip, and fourth methods can be used to make the first angle 180°, that is, the third, flip, and fourth methods can be used to achieve a rotation angle of 180° between the second submatrix data corresponding to the first component and the third submatrix data corresponding to the first component. The instruction set may include a fourth instruction, a fifth instruction, and a sixth instruction.

[0537] The fourth instruction can be used to indicate that the third method is to read in parallel or sequentially, according to rows M-1 to 0, each row according to the first element value from column 0 to column N-1, and at least one row is read in parallel or at least one row is read sequentially. That is, the fourth instruction can be used to indicate that the third method is to read in reverse horizontal order, and at least one row is read in parallel or at least one row is read sequentially.

[0538] The fifth instruction can be used to instruct the row data of the acquired second submatrix to be vector-reversed. The sixth instruction can be used to instruct the fourth method to store the reversed second submatrix data sequentially.

[0539] To better understand the visual rotation process based on the instruction set related to the SME instruction set and / or SVE instruction set described in the embodiments of this application, the following uses the second sub-matrix data corresponding to the U component as an example, combined with... Figure 17A Please provide an explanation. Figure 17A U can be used to indicate the value of the U component (i.e., the value of the first element corresponding to the U component).

[0540] Figure 17A This is a schematic diagram of another visual rotation process based on the SME instruction set and / or SVE instruction set provided for an embodiment of this application.

[0541] like Figure 17A As shown, with the first angle at 90°, the data can be read from column 0 to column 3, with each column reading the U component values ​​from row 1 to row 0, and at least one column read in parallel or sequentially. The U component values ​​obtained are then stored sequentially in a 4x2 array, thus obtaining the third submatrix data corresponding to the U components. The rotation angle between the first and third submatrix data corresponding to the U components is 90°. The third submatrix data corresponding to the U components can be stored in memory using a row-major storage method.

[0542] With the first angle at 180°, the data can be read from row 1 to row 0, with each row reading the U component values ​​from column 0 to column 3, and at least one row read in parallel or sequentially. Then, the row data of the second submatrix is ​​vector-flipped, and the flipped second submatrix is ​​stored sequentially. This yields the third submatrix data corresponding to the U components. The rotation angle between the first and third submatrix data corresponding to the U components is 180°. The third submatrix data corresponding to the U components can be stored in memory using a row-major storage method.

[0543] With the first angle being 270°, the data can be read from column 3 to column 0, with each column reading the U component value from row 0 to row 1, and at least one column read in parallel or sequentially. The U component values ​​obtained are then stored sequentially in a 4x2 column format, thus obtaining the third submatrix data corresponding to the U components. The rotation angle between the first and third submatrix data corresponding to the U components is 270°. The third submatrix data corresponding to the U components can be stored in memory using a row-major storage method.

[0544] It should be noted that, Figure 17A This is merely an example and does not constitute a limitation on the embodiments of this application.

[0545] For cases where the first angle is 90° or 270°, it can also be achieved in the following way.

[0546] As another implementation, based on the instruction set, the second submatrix data corresponding to the first component in the two-dimensional register is transposed. When the first angle is 90°, the transposed second submatrix data is column-reversed to obtain the third submatrix data corresponding to the first component, stored in memory. When the first angle is 270°, the transposed second submatrix data is row-reversed to obtain the third submatrix data corresponding to the first component, stored in memory.

[0547] When the first angle is 180°, it can also be achieved in the following way.

[0548] As another implementation method, based on the instruction set, the row-wise inversion of the second sub-matrix data corresponding to the first component in the two-dimensional register is performed, and the column-wise inversion of the row-wise inverted second sub-matrix data is performed to obtain the third sub-matrix data corresponding to the first component stored in memory.

[0549] As another implementation method, based on the instruction set, the column-reversed data of the second sub-matrix corresponding to the first component in the two-dimensional register is reversed, and the row-reversed data of the second sub-matrix is ​​reversed to obtain the third sub-matrix data corresponding to the first component stored in memory.

[0550] To better understand the visual rotation process based on the instruction set related to the SME instruction set and / or SVE instruction set described in the embodiments of this application, the following uses the second sub-matrix data corresponding to the U component as an example, combined with... Figure 17B Please provide an explanation. Figure 17B U can be used to indicate the value of the U component (i.e., the value of the first element corresponding to the U component).

[0551] Figure 17BThis is a schematic diagram of another visual rotation process based on an instruction set related to the SME instruction set, provided for an embodiment of this application.

[0552] like Figure 17B As shown, with the first angle being 90°, the second submatrix data corresponding to the U component can be transposed, and the columns of the transposed second submatrix data can be reversed to obtain the third submatrix data corresponding to the U component. The rotation angle between the first and third submatrix data corresponding to the U component is 90°. The third submatrix data corresponding to the U component can be stored in memory using a row-major storage method.

[0553] With the first angle being 180°, the second submatrix data corresponding to the U component can be row-reversed, and the row-reversed second submatrix data can be column-reversed, or the second submatrix data corresponding to the U component can be column-reversed, and the column-reversed second submatrix data can be row-reversed. This yields the third submatrix data corresponding to the U component. The rotation angle between the first and third submatrix data corresponding to the U component is 180°. The third submatrix data corresponding to the U component can be stored in memory using a row-major storage method.

[0554] With the first angle being 270°, the second submatrix data corresponding to the U component can be transposed, and then the rows of the transposed second submatrix data can be reversed to obtain the third submatrix data corresponding to the U component. The rotation angle between the first and third submatrix data corresponding to the U component is 270°. The third submatrix data corresponding to the U component can be stored in memory using a row-major storage method.

[0555] It should be noted that, Figure 17B This is merely an example and does not constitute a limitation on the embodiments of this application.

[0556] The second approach is a visual rotation process based on the first matrix data. The first matrix data may include first sub-matrix data corresponding to each of the multiple first components.

[0557] Based on the instruction set, the first submatrix data corresponding to each of the multiple first components, obtained from the two-dimensional register, can be rotated by a first angle to obtain the third submatrix data corresponding to each of the multiple first components, stored in memory. The instruction set can be used to indicate the rotation instruction set related to the SME instruction set and / or the SVE instruction set.

[0558] To obtain the third submatrix data corresponding to each of the multiple first components, which is obtained from the two-dimensional register and rotated by a first angle based on the instruction set, the following method can be used.

[0559] As one implementation, for any one of the multiple first components, when the first angle is 90°, 180°, or 270°, the first sub-matrix data corresponding to the first component is retrieved from the two-dimensional register according to a first method, and the first sub-matrix data corresponding to the first component is stored in memory according to a second method, resulting in the third sub-matrix data corresponding to the first component stored in memory. The first and second methods can be determined based on the instruction set. The first and second methods can be used to make the first angle 90°, 180°, or 270°.

[0560] When the first angle is 180°, the first sub-matrix data corresponding to the first component is obtained from the two-dimensional register in the third way, the obtained first sub-matrix data is flipped, and the flipped first sub-matrix data is stored in memory in the fourth way to obtain the third sub-matrix data corresponding to the first component stored in memory.

[0561] The third and fourth modes can be determined based on the instruction set. The third, flip, and fourth modes can be used to make the first angle 180°.

[0562] In one implementation, the data of the first submatrix corresponding to the first component may include multiple first element values ​​corresponding to the first component. The data of the first submatrix may include M rows and N columns. M and N can be integers greater than 1.

[0563] When the first angle is 90°, 180°, or 270°, the instruction set may include a second instruction and a third instruction. The third instruction may be used to indicate that the second method is to store the first element value obtained in the order of the first method.

[0564] When the first angle is 90°, the second instruction can be used to indicate that the first mode is to read in parallel or sequentially, from column 0 to column N-1, with each column reading from the first element value of row M-1 to the first element value of row 0.

[0565] When the first angle is 270°, the second instruction can be used to indicate that the first mode is to read in parallel or sequentially, from column N-1 to column 0, with each column reading from the first element value of row 0 to the first element value of row M-1.

[0566] When the first angle is 180°, the second instruction can be used to indicate that the first method is to read in parallel or sequentially, from row M-1 to row 0, with each row reading from the first element value of column N-1 to the first element value of column 0, and at least one row is read in parallel or at least one row is read sequentially.

[0567] When the first angle is 180°, the instruction set may include a fourth instruction, a fifth instruction, and a sixth instruction. The fourth instruction may be used to indicate that the third method involves reading rows M-1 to 0, with each row reading the first element value from column N-1 to column 0, and at least one row read in parallel or at least one row read sequentially. The fifth instruction may be used to indicate that the row data of the acquired first submatrix data is vector-reversed. The sixth instruction may be used to indicate that the fourth method involves sequentially storing the reversed first submatrix data.

[0568] As another implementation, for any of the multiple first components, when the first angle is 90° or 270°, the first sub-matrix data corresponding to the first component in the two-dimensional register is matrix transposed based on the instruction set. When the first angle is 90°, the columns of the transposed first sub-matrix data are reversed to obtain the third sub-matrix data corresponding to the first component stored in memory. When the first angle is 270°, the rows of the transposed first sub-matrix data are reversed to obtain the third sub-matrix data corresponding to the first component stored in memory.

[0569] With the first angle being 180°, based on the instruction set, the row-wise inversion of the first sub-matrix data corresponding to the first component in the two-dimensional register is performed, and the column-wise inversion of the row-wise inverted first sub-matrix data is performed, to obtain the third sub-matrix data corresponding to the first component stored in memory.

[0570] With the first angle being 180°, based on the instruction set, the column-reversed data of the first sub-matrix corresponding to the first component in the two-dimensional register is performed, and the row-reversed data of the first sub-matrix is ​​performed to obtain the third sub-matrix data corresponding to the first component stored in memory.

[0571] It should be noted that the explanation of the second method can be found in the explanation of the first method above. For example, you can replace "second matrix data" with "first matrix data" and "second submatrix data" with "first submatrix data" in the explanation of the first method. This will not be elaborated on here.

[0572] The above describes the visual rotation process. The component sampling process will be explained below. It should be noted that the component sampling process can be implemented independently. Optionally, the component sampling process can be implemented based on the component separation process. Optionally, the component sampling process can be implemented based on the visual rotation process. Optionally, the component sampling process can be implemented based on both the component separation process and the visual rotation process.

[0573] The second matrix data may include second sub-matrix data corresponding to the first component C and second sub-matrix data corresponding to the first component D. The first number may be the number of first element values ​​in the second sub-matrix data corresponding to the first component C. The second number may be the number of pixels included in the first visual data. The fourth number may be the number of first element values ​​in the second sub-matrix data corresponding to the first component D.

[0574] In one implementation, the first number can be less than the second number. The fourth number can be equal to the second number. In another implementation, the first number can be less than the second number or equal to the second number. For explanations of the first component C, the first component D, the first number, the second number, and the third number, please refer to the corresponding sections above, which will not be repeated here.

[0575] Upsampling in the component sampling process can be achieved through certain methods.

[0576] The first method is a component sampling process based on the second matrix data.

[0577] In the two-dimensional register, based on the seventh instruction, the third matrix data corresponding to the first component C is obtained according to the second sub-matrix data, the fourth matrix data, and the fifth matrix data corresponding to the first component C.

[0578] The fourth and fifth matrix data can be used to expand the data dimension of the second submatrix data by a factor of R in the first direction and / or by a factor of S in the second direction.

[0579] As one implementation, the fourth matrix data can be the interpolation matrix data in the first direction corresponding to the second sub-matrix data. The fourth matrix data can be used to expand the data dimension of the second sub-matrix data by a factor of R in the first direction. The fourth matrix data can be determined using an interpolation method. The interpolation method can include at least one of the following: linear interpolation, bilinear interpolation, bicubic interpolation, or nearest neighbor interpolation, etc. The first direction can be vertical or horizontal.

[0580] The fifth matrix data can be the interpolation matrix data in the second direction corresponding to the second sub-matrix data. The fifth matrix data can be used to expand the data dimension of the second sub-matrix data by a factor of S in the second direction. The fifth matrix data can be determined using an interpolation method. The second direction can be horizontal or vertical.

[0581] The seventh instruction can be used to instruct the execution of the external matrix accumulation between the second submatrix data and the fourth matrix data first, followed by the external matrix accumulation between the sixth matrix data and the fifth matrix data. The sixth matrix data can be the external matrix accumulation between the second submatrix data and the fourth matrix data. Alternatively, the sixth matrix data can be the matrix obtained by expanding the data dimension of the second submatrix data corresponding to the first component C by a factor of R in the first direction.

[0582] Therefore, the extramatrix accumulation between the data of the second submatrix and the data of the fourth matrix is ​​first determined to obtain the data of the sixth matrix. Then, the extramatrix accumulation between the data of the sixth matrix and the data of the fifth matrix is ​​determined to obtain the data of the third matrix corresponding to the first component C.

[0583] The following example, using the second submatrix corresponding to the first component C, which comprises M rows and N columns, with the first direction being horizontal and the second direction being vertical, illustrates how to obtain the data for the fourth and fifth matrices. M and N can be integers greater than 1.

[0584] Since the M×N second submatrix data needs to be expanded into the M×2N sixth matrix data through the fourth matrix data, that is, the M×N second submatrix data is doubled in the horizontal data dimension through the fourth matrix data to obtain the M×2N sixth matrix data, the fourth matrix data can be 2N×N. The second submatrix data, the fourth matrix data, and the sixth matrix data can satisfy the following formula (1).

[0585]

[0586] Here, S can represent the data of the second sub-matrix corresponding to the first component C. D1 can represent the data of the sixth matrix. H can represent the data of the fourth matrix. T It can represent the transpose of the fourth matrix data. It can represent the outer product of matrices.

[0587] The data in the fourth matrix can satisfy the following formulas (2)-(4).

[0588]

[0589] Here, H can represent the fourth matrix data. j can represent the column index in the sixth matrix data. j can also represent the row index in the fourth matrix data. j∈{0,1,......,2N-1}. t can represent the column index of the second sub-matrix data corresponding to the first component C. t can also represent the column index in the fourth matrix data. t∈{0,1,......,N-1}. s1 can represent mapping j to the coordinate space of the second sub-matrix data. This can represent the left-hand index corresponding to j. The left-hand index can be used to indicate the column index in the second submatrix data corresponding to the value of the first element. It can represent rounding down. dx can represent the integer part of s. dx can represent the fractional part of s. `j` can represent the right-hand index corresponding to `j`. The right-hand index can be used to indicate the column index corresponding to the first element value in the second submatrix data. `1-dx` can represent the weight corresponding to the first element value on the left. The first element value on the left can be used to indicate the first element value corresponding to the left-hand index. `dx` can represent the weight corresponding to the first element value on the right. The first element value on the right can be used to indicate the first element value corresponding to the right-hand index.

[0590] According to the above formulas (2)-(4), in In this case, the data of the fourth matrix can be represented as shown in equation (5).

[0591]

[0592] For example, when N=2, the above equation (5) can be expressed as the following equation (6).

[0593]

[0594] The data in the fifth matrix can satisfy the following formulas (7)-(9).

[0595]

[0596] Where V can represent the fifth matrix data. k can represent the row index of the sixth matrix data. k can also represent the row index of the fifth matrix data. k∈{0,1,......,2M-1}. l can represent the row index of the second sub-matrix data corresponding to the first component C. l can also represent the column index of the fifth matrix data. l∈{0,1,......,M-1}. s2 can represent mapping k to the coordinate space of the second sub-matrix data. This can represent the lower index corresponding to k. The lower index can be used to indicate the row index in the second submatrix data corresponding to the value of the first element. It can represent rounding down. can represent the integer part of s². dy can represent the fractional part of s². `k` can represent the index above `k`. The index above `k` can be used to indicate the row index corresponding to the first element value in the second submatrix data. `1-dy` can represent the weight corresponding to the first element value below `k`. The first element value below `k` can be used to indicate the first element value corresponding to the index below `k`. `dy` can represent the weight corresponding to the first element value above `k`. The first element value above `k` can be used to indicate the first element value corresponding to the index above `k`.

[0597] According to the above formulas (7)-(9), in In this case, the data of the fifth matrix can be represented as shown in equation (10).

[0598]

[0599] For example, when M=2, the above equation (10) can be expressed as the following equation (11).

[0600]

[0601] For example, when M=4, the above equation (10) can be expressed as the following equation (12).

[0602]

[0603] The data in the third matrix can satisfy the following formula (13).

[0604]

[0605] Here, D2 can represent the data of the third matrix. V can represent the data of the fifth matrix. D1 can represent the data of the sixth matrix. It can represent the outer product of matrices.

[0606] The above explains how to obtain the data for the fifth and sixth matrices when the first direction is horizontal and the second direction is vertical. A similar method can be used to obtain the data for the fifth and sixth matrices when the first direction is vertical and the second direction is horizontal, which will not be elaborated upon here.

[0607] To facilitate a better understanding of the component sampling process based on the instruction set related to the SME instruction set and / or SVE instruction set described in the embodiments of this application, the following... Figure 10 Based on, combined Figure 18A and Figure 18B ,right Figure 10 The first and second methods will be further explained.

[0608] Figure 18A This is a schematic diagram of a component sampling process based on a first method provided in an embodiment of this application.

[0609] like Figure 18A As shown, the horizontal interpolation matrix data (i.e., the fourth matrix data) satisfies formula (6) mentioned above. The vertical interpolation matrix data (i.e., the fifth matrix data) satisfies formula (11) mentioned above. The sixth matrix data satisfies formula (1) mentioned above. The third matrix data satisfies formula (13) mentioned above.

[0610] By first performing the matrix outside accumulation between the second submatrix data and the horizontal interpolation matrix data to obtain the sixth matrix data, and then performing the matrix outside accumulation between the sixth matrix data and the vertical interpolation matrix data, the data dimension of the second submatrix data in the horizontal direction is doubled, and then the data dimension of the second submatrix data in the vertical direction is doubled.

[0611] Figure 18B This is a schematic diagram of a component sampling process based on the second method provided in an embodiment of this application.

[0612] like Figure 18B As shown, with Figure 18A The difference is that, Figure 18B The process involves first performing the outer product of the second submatrix data and the vertical interpolation matrix data to obtain the sixth matrix data, and then performing the outer product of the sixth matrix data and the horizontal interpolation matrix data. This effectively doubles the data dimension of the second submatrix data in the vertical direction, and then doubles the data dimension of the second submatrix data in the horizontal direction.

[0613] It should be noted that, Figure 18A and Figure 18B This is merely an example and does not constitute a limitation on the embodiments of this application.

[0614] The second approach is based on a component sampling process using the first matrix data. The first matrix data may include first sub-matrix data corresponding to each of the multiple first components.

[0615] In the two-dimensional register, based on the seventh instruction, the third matrix data corresponding to the first component C is obtained according to the first sub-matrix data, the fourth matrix data, and the fifth matrix data corresponding to the first component C.

[0616] The fourth and fifth matrix data can be used to expand the data dimension of the first submatrix data by a factor of R in the first direction and / or by a factor of S in the second direction.

[0617] As one implementation, the fourth matrix data can be the interpolation matrix data in the first direction corresponding to the first sub-matrix data. The fourth matrix data can be used to expand the data dimension of the first sub-matrix data by a factor of R in the first direction. The fourth matrix data can be determined using an interpolation method. The interpolation method can include at least one of the following: linear interpolation, bilinear interpolation, bicubic interpolation, or nearest neighbor interpolation, etc. The first direction can be vertical or horizontal.

[0618] The fifth matrix data can be the interpolation matrix data in the second direction corresponding to the first sub-matrix data. The fifth matrix data can be used to expand the data dimension of the first sub-matrix data by a factor of S in the second direction. The fifth matrix data can be determined using an interpolation method. The second direction can be horizontal or vertical.

[0619] The seventh instruction can be used to instruct the execution of the external matrix accumulation between the first submatrix data and the fourth matrix data first, followed by the external matrix accumulation between the sixth matrix data and the fifth matrix data. The sixth matrix data can be the external matrix accumulation between the first submatrix data and the fourth matrix data. Alternatively, the sixth matrix data can be the matrix obtained by expanding the data dimension of the first submatrix data corresponding to the first component C by a factor of R in the first direction.

[0620] Therefore, the extramatrix accumulation between the data of the first submatrix and the data of the fourth matrix is ​​first determined to obtain the data of the sixth matrix. Then, the extramatrix accumulation between the data of the sixth matrix and the data of the fifth matrix is ​​determined to obtain the data of the third matrix corresponding to the first component C.

[0621] For downsampling in the component sampling process.

[0622] The second object corresponding to the first component C may include a third row of data, a third column of data, or a third matrix block of data corresponding to the first component C. The second object may include at least one first element value A and at least one first element value B; that is, if the second object is a third row of data, the third row of data may include at least one first element value A and at least one first element value B. If the second object is a third column of data, the third column of data may include at least one first element value A and at least one first element value B. If the second object is a third matrix block of data, the third matrix block of data may include at least one first element value A and at least one first element value B. First element value A may be a first element value that requires a third loading operation. First element value B may be a first element value that does not require a third loading operation. The number of first element values ​​A and the number of first element values ​​B may be determined according to a predetermined number.

[0623] The processor may further include a second register. The second register may be used to store at least one sixth identifier corresponding to the second object. Of the at least one sixth identifier, the sixth identifier corresponding to the first element value A has a first value, and the sixth identifier corresponding to the first element value B has a second value. The first value may be used to indicate that a third load operation for the first element value A is performed. The second value may be used to indicate that a third load operation for the first element value B is not performed.

[0624] The instruction set may include at least one tenth instruction corresponding to the first component C. The tenth instruction may include a seventh identifier, at least one eighth identifier, at least one ninth identifier, and the tenth identifier. The seventh identifier may be used to indicate a third load operation. The eighth identifier may be used to indicate a first region. At least one ninth identifier may be used to indicate a second address corresponding to at least one second object.

[0625] Therefore, a second address corresponding to each of the at least one second object can be determined based on at least one ninth identifier. Based on the seventh identifier and the second address corresponding to each of the at least one second object, the second object corresponding to each of the at least one second address is retrieved from memory. At least one sixth identifier corresponding to each of the at least one second object is retrieved from the second register corresponding to the tenth identifier. Based on the seventh identifier and the at least one tenth identifier corresponding to each of the at least one second object, the second object corresponding to each of the at least one second address is loaded into a first region corresponding to each of the at least one eighth identifier. The sixth identifier of the second object stored in the first region is a first value.

[0626] In addition, at least one tenth instruction corresponding to the first component C can be executed in parallel or sequentially.

[0627] The above describes the component sampling process. The following section explains the color space conversion process. It should be noted that the color space conversion process can be implemented independently. Optionally, the color space conversion process can be implemented based on the component separation process. Optionally, the color space conversion process can be implemented based on the visual rotation process. Optionally, the color space conversion process can be implemented based on both the component separation and visual rotation processes. Optionally, the color space conversion process can be implemented based on both the component separation and component sampling processes. Optionally, the color space conversion process can be implemented based on a combination of component separation, visual rotation, and component sampling processes.

[0628] The color space conversion process can be used to convert first visual data corresponding to a first color space into second visual data corresponding to a second color space.

[0629] The following explains how to convert the first visual data corresponding to the first color space into the second visual data corresponding to the second color space.

[0630] As one implementation, when the first matrix data includes first sub-matrix data corresponding to each of the multiple first components, or when the first element values ​​corresponding to each of the multiple first components in the first matrix data are arranged consecutively, or when the first element values ​​corresponding to each first component in the first matrix data are arranged consecutively, the second visual data is obtained in a two-dimensional register based on the instruction set, according to the ninth matrix data and the first matrix data corresponding to the first visual data. The ninth matrix data can be determined based on the transformation relationship between the multiple first components and the multiple second components.

[0631] The instruction set can be used to indicate color space conversion instruction sets related to the SME instruction set and / or SVE instruction set. The instruction set can be used to convert a first pixel value corresponding to a pixel into a second pixel value corresponding to the pixel based on the ninth matrix data and the first matrix data corresponding to the first visual data. The first element value can be used to indicate a first component value. Therefore, the instruction set can be used to convert the first component values ​​corresponding to each of the plurality of first components corresponding to a pixel into second component values ​​corresponding to each of the plurality of second components corresponding to the pixel, based on the ninth matrix data and the first matrix data corresponding to the first visual data.

[0632] As another implementation, in the case where there are at least two first element values ​​corresponding to each of the first components in the first matrix data arranged alternately, or in the case where the second matrix data includes second sub-matrix data corresponding to each of the multiple first components, the second visual data is obtained in the two-dimensional register based on the instruction set, according to the ninth matrix data and the second matrix data corresponding to the first visual data.

[0633] The instruction set can be used to convert a first pixel value corresponding to a pixel into a second pixel value corresponding to the pixel, based on the ninth matrix data and the second matrix data corresponding to the first visual data. The first element value can be used to indicate the first component value. Therefore, the instruction set can be used to convert the first component values ​​corresponding to multiple first components of a pixel into the second component values ​​corresponding to multiple second components of the pixel, based on the ninth matrix data and the second matrix data corresponding to the first visual data.

[0634] If the first matrix data is stored in memory, it needs to be loaded from memory into a two-dimensional register. For example, a second load operation for at least one second vector data can be performed in parallel until the first matrix data is loaded from memory into the two-dimensional register.

[0635] The first matrix data corresponding to the first visual data may include at least one second vector data corresponding to each of the plurality of first components. The second vector data corresponding to the first component may include at least one first element value corresponding to the first component. That is, for any first component among the plurality of first components, there is at least one second vector data corresponding to that first component. The second vector data may include at least one first element value corresponding to that first component.

[0636] When the second matrix data is stored in memory, it needs to be loaded from memory into a two-dimensional register. For example, a second load operation for at least one second vector data can be performed in parallel until the second matrix data is loaded from memory into the two-dimensional register.

[0637] The second matrix data corresponding to the first visual data may include at least one second vector data corresponding to each of the plurality of first components. The second vector data corresponding to a first component may include at least one first element value corresponding to the first component. That is, for any first component among the plurality of first components, there is at least one second vector data corresponding to that first component. The second vector data may include at least one first element value corresponding to that first component.

[0638] To better understand the parallel execution of the second loading operation for at least one second vector data described in the embodiments of this application, the following example uses the YUV color space as the first color space and the first visual data including the first pixel values ​​corresponding to each of the 16 pixels. Figure 19 Please provide an explanation. Figure 19 The Y component can be used to indicate the Y component value (i.e., the first element value corresponding to the Y component). The U component can be used to indicate the U component value (i.e., the first element value corresponding to the U component). The V component can be used to indicate the V component value (i.e., the first element value corresponding to the V component).

[0639] Figure 19 This is a schematic diagram illustrating the parallel execution of a second loading operation on multiple second vector data, as provided in an embodiment of this application.

[0640] like Figure 19 As shown, the two second vector data corresponding to the Y component can include the data in row 0 and row 1 of the first matrix data. Row 0 and row 1 can be the second vector data corresponding to the Y component. The two second vector data corresponding to the U component can include the data in row 2 and row 3 of the first matrix data. Row 2 and row 3 can be the second vector data corresponding to the U component. The two second vector data corresponding to the V component can include the data in row 4 and row 5 of the first matrix data. Row 4 and row 5 can be the second vector data corresponding to the V component.

[0641] It can execute the second loading operation for multiple second vector data in parallel, thereby loading multiple second vector data from memory into a two-dimensional register.

[0642] It should be noted that, Figure 19 This is merely an example and does not constitute a limitation on the embodiments of this application.

[0643] The following explanation pertains to the data in the ninth matrix.

[0644] As one implementation, when the first color space is the YUV color space and the second color space is the RGB color space, the multiple first components may include Y, U, and V components. The multiple second components may include R, G, and B components. The conversion relationship between the multiple first components and the multiple second components can satisfy the following formula (14). It should be noted that, depending on the color space standard, luminance range, chromaticity range, and application scenario, the conversion relationship between the YUV color space and the RGB color space can take many forms. For ease of understanding, this application embodiment illustrates one of these forms, but does not constitute a limitation on the embodiments of this application.

[0645]

[0646] According to formula (14), the data of the ninth matrix can satisfy the following formulas (15)-(18).

[0647] M = [m1 m2 m3] (15)

[0648]

[0649]

[0650] Where M can represent the data of the ninth matrix. m1 can represent the influence of the Y component on RGB. m2 can represent the influence of the U component on RGB. m3 can represent the influence of the V component on RGB.

[0651] The following section explains how to obtain the second visual data based on the instruction set, the ninth matrix data, and the first matrix data corresponding to the first visual data.

[0652] Based on the ninth instruction, the second visual data is obtained according to the ninth matrix data and the first vector data corresponding to each of the multiple pixels.

[0653] The first vector data corresponding to a pixel can be determined based on the first element values ​​of each of the multiple first components corresponding to the pixel in the first matrix data. For example, the first vector data corresponding to a pixel may include the third element values ​​of each of the multiple first components corresponding to the pixel. The third element value corresponding to the first component of the pixel can be determined based on the first element value corresponding to the first component of the pixel. The ninth instruction can be used to instruct the execution of a matrix-vector multiplication between the ninth matrix data and the first vector data.

[0654] The following explanation focuses on the first vector data.

[0655] As one implementation, when the first color space is YUV and the second color space is RGB, the first vector data corresponding to a pixel can be determined based on the first element value (i.e., first component value) corresponding to the pixel's Y component, U component, and V component. As described above, the first component value corresponding to the Y component can be called the Y component value. The first component value corresponding to the U component can be called the U component value. The first component value corresponding to the V component can be called the V component value. For example, the first vector data corresponding to a pixel can include the third element value corresponding to the pixel's Y component, U component, and V component. The third element value corresponding to the Y component can be called the Y' component value. The third element value corresponding to the U component can be called the U' component value. The third element value corresponding to the V component can be called the V' component value.

[0656] The first vector data can satisfy the following formulas (19)-(22).

[0657]

[0658] Y'=Y (20)

[0659] U'=U-128 (21)

[0660] V'=V-128 (22)

[0661] Where yuv can represent the first vector data. Y' can represent the Y' component value. U' can represent the U' component value. V' can represent the V' component value. Y can represent the Y component value. U can represent the U component value. V can represent the V component value.

[0662] The following explanation focuses on obtaining the second visual data based on the ninth instruction, the ninth matrix data, and the first vector data corresponding to each of the multiple pixels.

[0663] The second visual data can be obtained by accumulating the matrix outside each of the multiple pixels. The accumulating matrix outside each pixel can be the accumulating matrix outside the ninth matrix data and the first vector data corresponding to the pixel. The second visual data can satisfy the following formula (23).

[0664]

[0665] in, It can represent the matrix summation between the data of the ninth matrix and the first vector data corresponding to the pixels.

[0666] When the first visual data corresponding to the first color space is converted into the second visual data corresponding to the second color space, where the second color space is the RGB color space, the second visual data can be displayed.

[0667] Figure 20 A flowchart illustrating a data processing method provided in an embodiment of this application. This method can be applied to a processor.

[0668] like Figure 20 As shown, the method includes S2010-S2020.

[0669] In S2010, based on the instruction set, the first matrix data corresponding to the first visual data is processed.

[0670] In S2020, the processing results are output.

[0671] According to embodiments of this application, the instruction set may be determined based on an Extensible Single Instruction Multiple Data (SMILD) instruction set. An SMILD instruction set may include one of the following: an SME instruction set or an SVE instruction set. Therefore, the instruction set may be determined based on an SME instruction set and / or an SVE instruction set, i.e., the instruction set may be an instruction set related to an SME instruction set and / or an instruction set related to an SVE instruction set. For example, the instruction set related to an SME instruction set may be the SME instruction set. Optionally, the instruction set related to an SME instruction set may be obtained by adjusting the SME instruction set. Optionally, the instruction set related to an SME instruction set may include the SME instruction set and a first newly added instruction set. The instruction set related to an SVE instruction set may be the SVE instruction set. Optionally, the instruction set related to an SVE instruction set may be obtained by adjusting the SVE instruction set. Optionally, the instruction set related to an SVE instruction set may include the SVE instruction set and a second newly added instruction set.

[0672] The first matrix data may include at least one first element value corresponding to each of the multiple first components. The first component may be a color component of a first color space.

[0673] The instruction set can be used to control the processor to execute at least one of the following processes: storage format conversion of the first visual data, visual geometric transformation of the first visual data, sampling of the first component, or color space conversion of the first visual data. As one implementation, the storage format conversion of the first visual data can separate at least one first element value corresponding to each of the multiple first components in the first matrix data. Separating at least one first element value corresponding to each of the multiple first components in the first matrix data can be understood as rearranging at least one first element value corresponding to each of the multiple first components in the first matrix data.

[0674] The instruction set may include instructions corresponding to at least one of the following: access, matrix outside accumulation, or matrix multiplication by vector. Access may include at least one of the following: horizontal access (e.g., horizontal loading or horizontal storage), vertical access (vertical loading or vertical storage), or distributed access (distributed loading or distributed storage). For example, instructions corresponding to access may be used for at least one of storage format conversion, visual geometric transformation, sampling of a first component, or color space conversion. For example, instructions corresponding to at least one of horizontal loading, vertical loading, or distributed loading may be used to load first matrix data from memory or other matrix data obtained from the first matrix data. Matrix outside accumulation may be used for sampling of the first component. For example, instructions corresponding to matrix outside accumulation may be used to obtain a matrix multiplication result between two matrix data. The two matrix data may be determined based on the first matrix data. As one implementation, the two matrix data may include first sub-matrix data corresponding to the first component and matrix data for expanding the data dimension of the first sub-matrix data. Instructions corresponding to matrix multiplication by vector may be used for color space conversion. As one implementation, instructions corresponding to matrix multiplication by vector may be used to obtain a matrix multiplication by vector result between ninth matrix...

Claims

1. A data processing method, characterized in that, Applied to a processor, the method includes: Based on the instruction set, process the first matrix data corresponding to the first visual data; Output the processing results; The instruction set is determined according to the Extensible Single Instruction Multiple Data (SMILD) instruction set. The first matrix data includes at least one first element value corresponding to each of a plurality of first components. The first component is a color component of a first color space. The instruction set is used to control the processor to execute at least one of the following processes: storage format conversion of the first visual data, visual geometric transformation of the first visual data, sampling of the first component, or color space conversion of the first visual data. The instruction set includes instructions corresponding to at least one of the following: access, matrix outside accumulation, or matrix multiplication by vector. The access includes at least one of the following: horizontal loading, vertical loading, or scatter loading. The instruction corresponding to the access is used for at least one of the above processes. The instruction corresponding to the matrix outside accumulation is used for sampling of the first component. The instruction corresponding to the matrix multiplication by vector is used for color space conversion.

2. The method according to claim 1, characterized in that, The processor includes a two-dimensional register; Wherein, when the instruction set is used to control the processor to perform storage format conversion of the first visual data, the step of processing the first matrix data corresponding to the first visual data based on the instruction set includes: Based on the instruction set, the first matrix data is loaded from memory into the two-dimensional register to obtain the second matrix data stored in the two-dimensional register. The first matrix data contains alternating first element values ​​corresponding to at least two of the first components, and the second matrix data includes second sub-matrix data corresponding to each of the plurality of first components.

3. The method according to claim 2, characterized in that, The instruction set includes at least one first instruction corresponding to each of the plurality of first components. The plurality of first components includes a first component A and a plurality of first components B, or the plurality of first components are all first components B. The first component A is a first component with an independent storage area, and the first component B is a first component without an independent storage area. The storage area corresponding to the first component A stores at least one first element value, which is the first element value corresponding to the first component A. The storage area corresponding to the first component B stores a plurality of first element values, including at least one first element value corresponding to the first component B and at least one first element value corresponding to other first components B. The step of loading the first matrix data from memory into the two-dimensional register based on the instruction set to obtain the second matrix data stored in the two-dimensional register includes: For any first component among the plurality of first components, based on at least one first instruction corresponding to the first component, at least one first object corresponding to the first component in the first matrix data is loaded from the memory to the two-dimensional register to obtain second sub-matrix data corresponding to the first component stored in the two-dimensional register; Wherein, when the first component is the first component A, the first object corresponding to the first component A includes the first row data corresponding to the first component A, the first column data corresponding to the first component A, or the first matrix block data corresponding to the first component A; When the first component is the first component B, the first object includes the first element value corresponding to the first component B in the second row data, the first element value corresponding to the first component B in the second column data, or the first element value corresponding to the first component B in the second matrix block data. The second row data is the row data corresponding to the plurality of first components B, the second column data is the column data corresponding to the plurality of first components B, and the second matrix block data is the matrix block data corresponding to the plurality of first components B.

4. The method according to claim 3, characterized in that, The two-dimensional register includes at least one first region, and the first instruction includes a first identifier, at least one second identifier, and at least one third identifier. The first identifier is used to indicate a first load operation, the second identifier is used to indicate the first region, and the at least one third identifier is used to indicate a first address corresponding to each of the at least one first object. Wherein, loading at least one first object corresponding to the first component in the first matrix data from the memory to the two-dimensional register based on at least one first instruction corresponding to the first component includes: For any first instruction in at least one first instruction corresponding to the first component, Based on the at least one third identifier, determine the first address corresponding to each of the at least one first object; Based on the first identifier and the first address corresponding to each of the at least one first object, the first object corresponding to each of the at least one first address is obtained from the memory; Based on the first identifier, the first object corresponding to at least one of the first addresses is loaded into the first region corresponding to at least one of the second identifiers; Specifically, at least one first instruction corresponding to the first component is executed in parallel or at least one first instruction corresponding to the first component is executed sequentially.

5. The method according to claim 4, characterized in that, The processor further includes a first register for storing at least one fourth identifier corresponding to the first object, wherein the fourth identifier corresponding to the first component is a first value and the fourth identifiers corresponding to other first components are second values, the first value is used to indicate that a first loading operation is performed for a first element value corresponding to the first component, and the second value is used to indicate that a first loading operation is not performed for a first element value corresponding to the first component. The first instruction also includes a fifth identifier, which is used to indicate the first register; Wherein, loading the first object corresponding to at least one first address into the first region corresponding to at least one second identifier based on the first identifier includes: Obtain at least one fourth identifier corresponding to each of the at least one first object from the first register corresponding to the fifth identifier; Based on the first identifier and at least one fourth identifier corresponding to each of the at least one first object, the first object corresponding to each of the at least one first address is loaded into the first region corresponding to each of the at least one second identifier, wherein the fourth identifier of the first object stored in the first region is the first value.

6. The method according to any one of claims 1-5, characterized in that, The instruction set is also used to control the processor to perform visual geometric transformations of the first visual data. In the case of visual rotation, the step of processing the first matrix data corresponding to the first visual data based on the instruction set further includes: Based on the instruction set, the second sub-matrix data corresponding to each of the plurality of first components obtained from the two-dimensional register are rotated by a first angle to obtain the third sub-matrix data corresponding to each of the plurality of first components stored in memory; or Wherein, the instruction set is further configured to control the processor to perform visual geometric transformations of the first visual data, wherein the visual geometric transformations include visual rotation, the step of processing the first matrix data corresponding to the first visual data based on the instruction set includes: When the first matrix data includes first sub-matrix data corresponding to each of the plurality of first components, based on the instruction set, the first sub-matrix data corresponding to each of the plurality of first components obtained from the two-dimensional register is rotated by a first angle to obtain third sub-matrix data corresponding to each of the plurality of first components stored in the memory.

7. The method according to claim 6, characterized in that, The step of rotating the second sub-matrix data corresponding to each of the plurality of first components obtained from the two-dimensional register by a first angle based on the instruction set to obtain the third sub-matrix data corresponding to each of the plurality of first components stored in the memory includes: For any one of the plurality of first components, When the first angle is 90°, 180°, or 270°, the second sub-matrix data corresponding to the first component is obtained from the two-dimensional register according to a first method, and the second sub-matrix data corresponding to the first component is stored in the memory according to a second method, resulting in the third sub-matrix data corresponding to the first component stored in the memory. The first and second methods are determined based on the instruction set, and the first and second methods are used to make the first angle 90°, 180°, or 270°; and / or When the first angle is 180°, the second sub-matrix data corresponding to the first component is obtained from the two-dimensional register in a third manner, the obtained second sub-matrix data is flipped, and the flipped second sub-matrix data is stored in the memory in a fourth manner to obtain the third sub-matrix data corresponding to the first component stored in the memory. The third and fourth methods are determined based on the instruction set, and the third method, the flipping, and the fourth method are used to make the first angle 180°.

8. The method according to claim 7, characterized in that, The second submatrix data corresponding to the first component includes multiple first element values ​​corresponding to the first component. The second submatrix data includes M rows and N columns, where M and N are integers greater than 1. When the first angle is 90°, 180°, or 270°, the instruction set includes a second instruction and a third instruction. The third instruction indicates that the second method is to store the first element value obtained according to the first method in sequence. When the first angle is 90°, the second instruction indicates that the first method is to read in parallel or sequentially, according to columns 0 to N-1, where each column reads the first element value from row M-1 to row 0, and at least one column is read in parallel or sequentially. When the first angle is 270°, the second instruction indicates that the first method is to read in parallel or sequentially, according to columns N-1 to 0, where each column reads the first element value from row 0 to row M-1, and at least one column is read in parallel or sequentially. When the first angle is 180°, the second instruction indicates that the first method is to read in parallel or sequentially, according to rows M-1 to 0, where each row reads the first element value from column N-1 to column 0, and at least one row is read in parallel or sequentially. When the first angle is 180°, the instruction set includes a fourth instruction, a fifth instruction, and a sixth instruction. The fourth instruction is used to indicate that the third method is to read in parallel or sequentially, from row M-1 to row 0, with each row reading from the first element value of column 0 to the first element value of column N-1. The fifth instruction is used to indicate that the row data input vector of the acquired second submatrix data is flipped. The sixth instruction is used to indicate that the fourth method is to store the flipped second submatrix data sequentially.

9. The method according to claim 7, characterized in that, The step of rotating the second sub-matrix data corresponding to each of the plurality of first components obtained from the two-dimensional register by a first angle based on the instruction set to obtain the third sub-matrix data corresponding to each of the plurality of first components stored in the memory includes: For any one of the plurality of first components, When the first angle is 90° or 270°, based on the instruction set, the second sub-matrix data corresponding to the first component in the two-dimensional register is matrix transposed. When the first angle is 90°, the columns of the transposed second sub-matrix data are reversed to obtain the third sub-matrix data corresponding to the first component stored in the memory. When the first angle is 270°, the rows of the transposed second sub-matrix data are reversed to obtain the third sub-matrix data corresponding to the first component stored in the memory; and / or When the first angle is 180°, based on the instruction set, the second sub-matrix data corresponding to the first component in the two-dimensional register is row-reversed, and the row-reversed second sub-matrix data is column-reversed to obtain the third sub-matrix data corresponding to the first component stored in the memory; or based on the instruction set, the second sub-matrix data corresponding to the first component in the two-dimensional register is column-reversed, and the column-reversed second sub-matrix data is row-reversed to obtain the third sub-matrix data corresponding to the first component stored in the memory.

10. The method according to any one of claims 1-9, characterized in that, The instruction set includes a seventh instruction, and the second matrix data includes second sub-matrix data corresponding to the first component C, or the first matrix data includes first sub-matrix data corresponding to the first component C, wherein the first component C is the first component among the plurality of first components; Wherein, when the instruction set is used to control the processor to perform sampling of the first component, the step of processing the first matrix data corresponding to the first visual data based on the instruction set includes: In the two-dimensional registers included in the processor, based on the seventh instruction, the third matrix data corresponding to the first component C is obtained according to the second sub-matrix data, the fourth matrix data, and the fifth matrix data corresponding to the first component C, or the third matrix data corresponding to the first component C is obtained according to the first sub-matrix data, the fourth matrix data, and the fifth matrix data corresponding to the first component C; Wherein, the fourth matrix data and the fifth matrix data are used to expand the second sub-matrix data corresponding to the first component C by a factor of R in the first direction and / or by a factor of S in the second direction. The fourth matrix data and the fifth matrix data are determined based on the second sub-matrix data corresponding to the first component C. The first direction is a horizontal direction and the second direction is a vertical direction, or the first direction is the vertical direction and the second direction is the horizontal direction, where R and S are greater than or equal to 1; or The fourth matrix data and the fifth matrix data are used to expand the second sub-matrix data corresponding to the first component C by a factor of R in the first direction and / or by a factor of S in the second direction; The seventh instruction is used to instruct that the matrix outside accumulation between the second sub-matrix data and the fourth matrix data be performed first, or the matrix outside accumulation between the first sub-matrix data and the fourth matrix data be performed first, and then the matrix outside accumulation between the sixth matrix data and the fifth matrix data be performed. The sixth matrix data is the matrix outside accumulation between the second sub-matrix data and the fourth matrix data, or the sixth matrix data is the matrix outside accumulation between the first sub-matrix data and the fourth matrix data.

11. The method according to claim 10, characterized in that, The first component C is a first component among the plurality of first components that satisfies the following conditions: a first number is less than a second number, the first number is the number of first element values ​​in the second matrix data or the first matrix data corresponding to the first component C, the second number is the number of pixels included in the first visual data, and when the first component C is upsampled, a third number is equal to the second number, the third number is the number of element values ​​in the third matrix data corresponding to the first component C, and the element value is the first element value or the element value obtained by upsampling; or The first component C is any one of the plurality of first components, and the third number is equal to the predetermined number.

12. The method according to claim 10 or 11, characterized in that, The fourth matrix data is the interpolation matrix data of the second sub-matrix data or the first sub-matrix data in the first direction. The fourth matrix data is used to expand the data dimension of the second sub-matrix data or the first sub-matrix data in the first direction by a factor of R. The fifth matrix data is the interpolation matrix data of the second sub-matrix data or the first sub-matrix data in the second direction. The fifth matrix data is used to expand the data dimension of the second sub-matrix data or the first sub-matrix data in the second direction by a factor of S.

13. The method according to any one of claims 10-12, characterized in that, The fourth matrix data and the fifth matrix data are determined using an interpolation method, which includes at least one of the following: linear interpolation, bilinear interpolation, bicubic interpolation, or nearest neighbor interpolation.

14. The method according to any one of claims 1-13, characterized in that, The first visual data includes first pixel values ​​corresponding to each of the plurality of pixels, the first pixel values ​​include at least one first component value corresponding to each of the plurality of first components, the first element value is used to indicate the first component value, the first matrix data includes first sub-matrix data corresponding to each of the plurality of first components, and the instruction set includes at least one ninth instruction; Wherein, when the instruction set is used to control the processor to perform color space conversion of the first visual data, the step of processing the first matrix data corresponding to the first visual data based on the instruction set includes: Based on the at least one ninth instruction, second visual data is obtained according to the ninth matrix data and the first vector data corresponding to each of the plurality of pixels; Wherein, the second visual data includes a second pixel value corresponding to each of the plurality of pixels, the second pixel value includes a second component value corresponding to each of the plurality of second components, the second component is a color component of a second color space, the ninth matrix data is determined according to the conversion relationship between the plurality of first components and the plurality of second components, the first vector data corresponding to the pixel is determined according to the first element value corresponding to each of the plurality of first components in the first matrix data corresponding to the pixel, and the ninth instruction is used to instruct the execution of matrix-vector multiplication between the ninth matrix data and the first vector data to convert the first pixel value corresponding to the pixel into the second pixel value corresponding to the pixel.

15. The method according to claim 14, characterized in that, When the first matrix data corresponding to the first visual data is stored in memory, the method further includes: A second loading operation for at least one second vector data is executed in parallel until the first matrix data is loaded from the memory into the two-dimensional register; The first matrix data corresponding to the first visual data includes at least one second vector data corresponding to each of the plurality of first components, and the second vector data corresponding to the first component includes at least one first element value corresponding to the first component.

16. The method according to any one of claims 1-15, characterized in that, The Scalable Single Instruction Multiple Data instruction set includes at least one of the following: Variable Matrix Extended SME instruction set or Variable Vector Extended SVE instruction set.

17. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the electronic device to perform the method as described in any one of claims 1-16.

18. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-16.

19. A chip system, characterized in that, It includes at least one processor and a communication interface, the communication interface and the at least one processor being interconnected via a line, the at least one processor being used to run a computer program or instructions to perform the method as described in any one of claims 1-16.

20. A computer program product, characterized in that, Includes a computer program that, when run, causes a computer to perform the method as described in any one of claims 1-16.

Images