YUV texture generation method, device and electronic equipment

Abstract

The embodiment of the present invention provides a YUV texture generation method, device and electronic device. When it is necessary to generate a YUV texture, a first correspondence between a first target object and a target image coordinate system and a second correspondence between the YUV texture and the target image coordinate system are established using the same image coordinate system as a medium. The RGB value of the first coordinate corresponding to the target pixel point in the first target object is obtained based on the first correspondence, and converted to obtain YUV, and the converted YUV is written to the second coordinate of the YUV texture. In this way, there is no need to generate the Y component of all pixels in the order of texture coordinates, and then generate the U component and V component of all pixels based on the same order of texture coordinates, so that the accuracy of the sharing relationship between the Y component and the U component and the V component can be guaranteed, which effectively improves the generation efficiency of the YUV texture and further improves the encoding efficiency of the encoder.

Description

Technical Field

[0001] The present invention relates to the field of live broadcast technology, and in particular to a YUV texture generation method, device and electronic equipment. Background Art

[0002] In the field of live broadcasting, before each video frame enters the encoder for encoding, each video frame is represented in the form of RGB data, and these RGB data will be stored in RGB textures. However, some encoders only support encoding of YUV textures. In this case, it is necessary to generate YUV textures based on each RGB data so that the encoder can encode each video frame normally.

[0003] Related technologies propose to use fragment shaders to convert RGB data to generate YUV textures. However, texture coordinates only support sequential reading or writing of Y, U, and V components, and fragment shaders only support reading or writing texels within the current fragment. Therefore, when using fragment shaders to convert RGB data, it is necessary to read or write Y, U, and V components strictly in the read and write order supported by texture coordinates.

[0004] Although when displaying YUV texture, the color of each pixel in the video frame is determined by a set of YUV values, that is, 1 Y component, 1 U component and 1 V component corresponding to each pixel in the display screen. However, when storing YUV data, in order to save storage space and the bandwidth occupied during subsequent transmission, the number of U components and V components is usually reduced, and the YUV texture is generated by using multiple Y components to share a pair of UV components, that is, the Y component, U component and V component in the YUV texture are not stored in a 1:1:1 ratio.

[0005] In order to facilitate the encoder to accurately determine which Ys share which pair of UVs, the fragment shader is first used to generate all Y components in the order of texture coordinates, and then the fragment shader is used to generate all U and V components in the same order of texture coordinates. In this way, the encoder can determine the sharing relationship between Y components and UV components based on the same texture coordinate order.

[0006] However, this YUV texture generation method requires running the fragment shader twice to obtain an accurate YUV texture, which makes the YUV texture generation efficiency low and affects the encoding efficiency of the subsequent encoder. Summary of the invention

[0007] The purpose of the embodiments of the present invention is to provide a YUV texture generation method, device and electronic device to improve the efficiency of converting RGB texture to YUV texture, and ultimately improve the encoding efficiency of the encoder. The specific technical solution is as follows:

[0008] In a first aspect of the present invention, a YUV texture generation method is first provided, wherein the method comprises:

[0009] Establishing a first correspondence between a first coordinate system of a first target object and a target image coordinate system, and a second correspondence between a texture coordinate system of a second texture and the target image coordinate system; wherein the first target object is an RGB texture or an RGB video frame, and the second texture is a YUV texture;

[0010] Determine, according to the first corresponding relationship, a first coordinate corresponding to the target image coordinate point in the first coordinate system, and determine, according to the second corresponding relationship, a second coordinate corresponding to the target image coordinate point in the second texture coordinate system;

[0011] Convert the RGB value at the first coordinate into YUV;

[0012] The YUV is written into the YUV texture at the second coordinate.

[0013] In combination with the first aspect, in a second possible embodiment, the method further includes:

[0014] If the first target object is an RGB texture, determining the texture coordinate system of the RGB texture to be the first coordinate system;

[0015] If the first target object is an RGB video frame, the image coordinate system of the RGB video frame is determined to be the first coordinate system.

[0016] In combination with the first possible implementation example or the second possible implementation example of the first aspect, the method further includes:

[0017] According to the width / height of the first target object, the width / height of the second texture is adjusted so that the adjusted width / height of the second texture satisfies the proportional relationship among the target Y component, U component and V component.

[0018] In combination with the first aspect, in a fourth possible embodiment, the method further includes:

[0019] A notification message is sent to the image processor to notify the image processor to read the YUV data and forward the read YUV data to the encoder.

[0020] In a second aspect of the present invention, a YUV texture generating device is provided, the device comprising:

[0021] A correspondence establishing module, used to establish a first correspondence between a first coordinate system of a first target object and a target image coordinate system, and a second correspondence between a texture coordinate system of a second texture and the target image coordinate system; wherein the first target object is an RGB texture or an RGB video frame, and the second texture is a YUV texture;

[0022] a first determining module, configured to determine a first coordinate corresponding to a target image coordinate point in the first coordinate system according to the first corresponding relationship, and to determine a second coordinate corresponding to the target image coordinate point in the second texture coordinate system according to the second corresponding relationship;

[0023] A conversion module, used for converting the RGB value at the first coordinate into YUV;

[0024] A storage module is used to write the YUV into the second coordinate of the YUV texture.

[0025] In combination with the second aspect, in a second possible embodiment, the device further includes:

[0026] The second determination module is used to determine the texture coordinate system of the RGB texture as the first coordinate system if the first target object is an RGB texture; and to determine the image coordinate system of the RGB video frame as the first coordinate system if the first target object is an RGB video frame.

[0027] In combination with the first embodiment or the second embodiment of the second aspect, in a third possible embodiment, the device further includes:

[0028] A size adjustment module is used to adjust the width / height of the second texture according to the width / height of the first target object so that the adjusted width / height of the second texture satisfies the proportional relationship between the target Y component, U component and V component.

[0029] In combination with the first aspect, in a fourth possible embodiment, the device further includes:

[0030] The notification module is used to send a notification message to the image processor to notify the image processor to read the YUV data and forward the read YUV data to the encoder.

[0031] In another aspect of the present invention, an electronic device is provided, the electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0032] Memory, used to store computer programs;

[0033] The processor is used to implement any of the above-mentioned steps of the YUV texture generation method when executing the program stored in the memory.

[0034] In another aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned YUV texture generation methods is implemented.

[0035] In another aspect of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned YUV texture generation methods.

[0036] Beneficial effects of the present invention:

[0037] The embodiment of the present invention provides a YUV texture generation method, device and electronic device. When it is necessary to generate a YUV texture, a first correspondence between a first target object and a target image coordinate system and a second correspondence between the YUV texture and the target image coordinate system are established using the same image coordinate system as a medium. The RGB value of the first coordinate corresponding to the target pixel point in the first target object is obtained based on the first correspondence, and converted to obtain YUV, and the converted YUV is written to the second coordinate of the YUV texture. In this way, there is no need to generate the Y component of all pixels in the order of texture coordinates, and then generate the U component and V component of all pixels based on the same order of texture coordinates, so that the accuracy of the sharing relationship between the Y component and the U component and the V component can be guaranteed, which effectively improves the generation efficiency of the YUV texture and further improves the encoding efficiency of the encoder. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0039] Figure 1a A schematic diagram of a video frame pixel arrangement provided by an embodiment of the present invention;

[0040] Figure 1b A schematic diagram of storing RGB three components of an RGB texture provided by an embodiment of the present invention;

[0041] Figure 1c , Figure 1d , Figure 1e A schematic diagram of YUV texture YUV three-component storage provided by an embodiment of the present invention;

[0042] Figure 2 A possible process diagram of a YUV texture generation method provided by an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of a possible storage location of a target pixel and a corresponding YUV texture is provided for an embodiment of the present invention;

[0044] Figure 4 A schematic diagram of another possible target pixel point and the storage position of the corresponding YUV texture provided by the YUV texture generation method of the embodiment of the present invention;

[0045] Figure 5 A schematic diagram of a logical structure of a YUV texture generating device provided by an embodiment of the present invention;

[0046] Figure 6 A schematic diagram of a logical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0048] In the field of live broadcasting, video frames are usually represented in the form of RGB data. For example, assuming that there is an arrangement of pixels in the first video frame as follows Figure 1a As shown, the color of each pixel can be determined by the RGB data of each pixel. Figure 1a A set of RGB data of the first pixel point in the upper left corner (ie, the enlarged pixel point) can determine the color of the first pixel point.

[0049] In order to enrich the display effect of the live screen, the texture is usually used as a carrier to extract the RGB data of each pixel in each video frame and store it in the RGB texture. The subsequent encoder obtains the R component, G component, and B component of the corresponding coordinate point according to the texture coordinate system of the RGB texture, and encodes a set of RGB data, so that the display can display each video frame according to the RGB encoded by the encoder.

[0050] For ease of understanding, a possible application scenario is used to illustrate:

[0051] If the current live broadcast screen is an online teaching screen, the screen contains the screens of the three modules of "teacher", "course PPT" and "student A", among which the video source of the "teacher" module is the first camera, the video source of the "student A" module is the second camera, and the "course PPT" is a local file. In order to display the screens of the above three modules in the same display interface, texture is used as a carrier, and the video frames taken by the first camera, the video frames taken by the second camera and the local files are drawn into the same texture. When displayed on the live broadcast client, the effect of the drawn texture is directly displayed. The whole process can be vividly understood as: the texture is equivalent to a picture frame, and the video frames taken by the first camera, the video frames taken by the second camera and the local files are continuously superimposed in the same picture frame to obtain an overall picture.

[0052] For example, Figure 1a The first video frame is stored in Figure 1b Taking the RGB texture in as an example, the storage of video frames to RGB texture is explained:

[0053] According to the coordinate axis direction of the image coordinate system (i.e. from left to right, from top to bottom), obtain the RGB value of the first pixel in the upper left corner of the first video frame (see Figure 1a Then the R component, G component and B component of the pixel point are calculated according to Figure 1b The RGB channels shown are stored at the corresponding texture coordinates according to the coordinate axis direction of the texture coordinate system (i.e., from left to right, from top to bottom). Figure 1b As shown, the three R components, G component and B component of the gray part correspond to the RGB value of the first pixel point at the upper left corner of the first video frame. Similarly, the RGB data of each pixel point in the first video frame can be stored in the RGB texture.

[0054] During the live screen display stage, the encoder directly generates groups of RGB arrays based on the R component, G component, and B component pairs of each pixel of each video frame stored in the RGB texture to restore the color of each pixel of each video frame.

[0055] For example, when the display or display device needs to display the first video frame, the encoder can obtain the R component, G component and B component corresponding to the target texel coordinate from the RGB texture according to the coordinate axis direction of the texture coordinate system (also from left to right and from top to bottom), and then determine Figure 1b The R component, G component and B component stored in the three texels of medium gray belong to Figure 1a The first pixel point in the first video frame is extracted from the data stored in the three texels. Figure 1aThe coordinate position relationship of the first video frame shown is restored to obtain the RGB data of the first pixel. Similarly, the RGB value of each pixel point of the first video frame can be obtained, and the display device can display a colorful picture according to the RGB data of each pixel point encoded by the encoder.

[0056] However, there are some special encoders that only support encoding of YUV textures, but not RGB textures. The textures input to such encoders must be in YUV format. If the RGB texture is forced to be input, the image will not be displayed normally. For such special situations, it is necessary to convert the original RGB texture into a YUV texture, or convert the RGB data of each video frame into YUV data, and then store the converted YUV data in the YUV texture. Only then can the encoder encode normally and the display display normally.

[0057] The principle of converting RGB data into YUV data is as follows: extract the brightness information of the three components of RGB of the pixel point to obtain the Y component, extract the hue of the three components of RGB to obtain the U component, and extract the color saturation of the three components of RGB to obtain the V component.

[0058] Since the complete image can still be displayed without the UV component, and the human eye does not have high requirements for color. Therefore, when storing the three components of YUV, in order to save storage space and subsequent transmission bandwidth, the number of U components and V components is usually reduced, and multiple Y components are used to share a pair of UV components for storage, which can effectively reduce the amount of data required for storage and transmission. For example, the ratio of Y component: U component: V component is 4:2:2 or 4:2:0; or the ratio of Y component: U component: V component is 4:1:1.

[0059] For example, Figure 1c Take the YUV storage format shown as an example (in the figure, Y component: U component: V component = 4:2:1), where the four Y components Y0 to Y3 share U0V0 and U1V1, and the four Y components Y4 to Y7 share U2V2 and U3V3.

[0060] Because multiple Y components share a pair of UV components, the ratio between the Y component and the U component and the V component is not 1:1:1. Figure 1cFor example, if the four components Y0 to Y3 should theoretically share U0V0 and U1V1, the picture can be displayed normally. However, if the encoder obtains that the UV components corresponding to the four components Y0 to Y3 are U2V2 and U3V3, it does not meet the YUV data requirements of the original picture, causing the restored picture to be distorted. In this way, how to accurately determine the UV components corresponding to the four Ys becomes the key to ensuring the normal display of the picture.

[0061] In order to facilitate the encoder to accurately determine the sharing relationship between the Y component and the UV component when obtaining the Y component, U component and V component stored in the YUV texture, the relevant technology proposes to use a fragment shader to generate a YUV texture, which usually adopts the following two methods to store the Y component, U component and V component:

[0062] Method 1: Figure 1d For example, using a fragment shader, first move the sampling fragment according to the image coordinate order and the relative position relationship of each pixel point, obtain the Y component of all pixels in turn, and store the obtained Y component in the Y channel corresponding to the texture in the coordinate order of the texture coordinate system. Then, according to the same image coordinate order and the relative position relationship of each pixel point, move the sampling fragment to downsample the pixel points to obtain the corresponding U component and V component, and store the obtained U component in the U channel corresponding to the texture in the coordinate order of the texture coordinate system, and store the obtained V component in the V channel corresponding to the texture. In this way, the encoder can directly obtain the Y component, U component and V component from the corresponding channel according to the coordinate order of the texture coordinate system, and then determine the number of components obtained to determine the sharing relationship between the Y component, the U component and the V component.

[0063] Method 2: Figure 1e For example, using a fragment shader, first obtain the Y components of all pixels in sequence according to the image coordinate order and the relative position relationship of each pixel, and then store the obtained Y components in the first texture. Then, according to the same image coordinate order, downsample according to the relative position relationship of each pixel, obtain the corresponding U component and V component, and store the obtained U component and V component in the second texture, that is, store the Y component and UV component separately. In this way, when the encoder needs to obtain the Y component, U component and V component of the YUV texture, it can determine from the first texture and the second texture in sequence according to the texture coordinate order that Y0~Y3 share U0V0 and U1V1.

[0064] It can be seen that in the above two methods, when using the fragment shader to generate the YUV texture, the fragment shader needs to be run twice, once to generate the Y component and once to generate the UV component to obtain an accurate YUV texture, which makes the YUV texture generation efficiency low.

[0065] In addition, the subsequent encoder needs to wait until the Y component, U component and V component of all pixels are generated, and then accurately determine the sharing relationship between the Y component, U component and V component according to the same texture coordinate order. It can be seen that the method of generating YUV texture using fragment shader in related technology is easy to affect the subsequent encoding efficiency of the encoder.

[0066] Based on this, an embodiment of the present invention provides a YUV texture generation method, which is dedicated to improving the YUV texture generation efficiency, thereby improving the encoding efficiency of subsequent encoders. The YUV texture generation method provided by the embodiment of the present invention can be applied to any electronic device or processor with texture format conversion. The electronic device includes but is not limited to a personal mobile terminal, a personal computer, and a server. The processor includes but is not limited to an image processor.

[0067] like Figure 2 As shown, the YUV texture generation method provided by the present invention includes the following steps:

[0068] S11: establishing a first corresponding relationship between a first coordinate system of a first target object and a target image coordinate system, and a second corresponding relationship between a second texture coordinate system and the target image coordinate system; wherein the first target object is an RGB texture or an RGB video frame, and the second texture coordinate system is a YUV texture coordinate system;

[0069] S12: determining a first coordinate corresponding to the target pixel point in the first coordinate system according to the first corresponding relationship, and determining a second coordinate corresponding to the target pixel point in the second texture coordinate system according to the second corresponding relationship;

[0070] S13: Convert the RGB value at the first coordinate into YUV;

[0071] S14: Write the converted YUV into the second coordinate of the YUV texture.

[0072] When an embodiment of the present invention is selected and a YUV texture needs to be generated, a first correspondence between the original RGB video frame or the original RGB texture and the target image coordinate system and a second correspondence between the YUV texture and the same target image coordinate system are established with the target image coordinate system as a medium. Through the first correspondence, the RGB value at the corresponding first coordinate is obtained from the original first target object, and then data conversion is performed to obtain the corresponding YUV value, and then the obtained YUV value is written to the second coordinate according to the second correspondence.

[0073] In this way, it is different from the prior art that the Y component of all pixels is first generated according to the coordinate axis order, and then the U component and the V component are generated according to the same coordinate axis order. By selecting the embodiment of the present invention, the RGB value can be quickly read based on the first coordinate corresponding to the acquired target pixel in the original first target object, and the corresponding Y component, U component and V component can be converted. Then, according to the second coordinate corresponding to the target pixel in the YUV texture, the converted Y component, U component and V component can be quickly written into the coordinate corresponding to the YUV texture, which can effectively improve the generation efficiency of the YUV texture.

[0074] Moreover, by selecting the embodiment of the present invention, the encoder does not need to wait for the Y component, U component and V component of all pixels to be generated before obtaining the Y component, U component and V component in the order of texture coordinates. The encoder can obtain the Y component, U component and V component corresponding to the second coordinate corresponding to the target pixel in parallel, thereby effectively improving the encoding efficiency of the encoder for YUV data.

[0075] The above steps S11-S14 will be described in detail below:

[0076] In the embodiment of the present invention, the first target object may be an RGB video frame or an RGB texture. In a possible embodiment, after the execution subject acquires the first target object, the first coordinate system of the first target object is determined according to the type of the first target object.

[0077] For example, if the first target object is an RGB video frame, the first coordinate system may be determined to be the original image coordinate system of the RGB video frame. If the first target object is an RGB texture, the first coordinate system may be determined to be the texture coordinate system of the RGB texture.

[0078] In a possible embodiment, if the first target object is an RGB texture, when executing step S11, a corresponding relationship between the first coordinate system and the target image coordinate system is established, which can be specifically implemented in the following manner:

[0079] According to the storage format of the target texture and the size of the target image object, the correspondence between the texture coordinate system of the target texture and the image coordinate system of the target image object is determined. Wherein, the target texture is an RGB texture, the storage format of the target texture includes: the storage order of the R channel, the G channel and the B channel and the number of the R channel, the G channel and the B channel, and the size of the target image object includes the width and height of the target image object, and the specific width and height units can be: pixels.

[0080] For example, in a possible application scenario, Figure 1bAs shown, if the storage format of the RGB texture is: along the texture coordinate axis direction (i.e., from left to right, from top to bottom), then the number of R channels, G channels, and B channels in this RGB texture is: the width of the RGB texture / 3. At this time, it can be determined that the three texture coordinates in the RGB texture store the values ​​of the RGB three channels corresponding to a target pixel point, so it can be determined that the first corresponding relationship between the texture coordinate system of the RGB texture and the target image object is: according to the data storage direction specified in the storage format of the RGB texture, the values ​​of the RGB three channels stored in the three texture coordinates correspond to the RGB value of a target pixel point.

[0081] At this time, when executing step S12, the positions of the three corresponding texture coordinates in the corresponding texture coordinate system can be determined according to the image coordinate position of the target pixel point in the target image coordinate system. Figure 4 For example, it can be determined that the texture coordinate position corresponding to the texture coordinate system of the pixel point 4 in the RGB texture is: the three texture coordinate positions occupied by the three data R4, G4 and B4.

[0082] In a possible embodiment, if the first target object is an RGB video frame, then when executing step S11, the correspondence between the image coordinate system of the RGB video frame and the target image coordinate system can be directly established, and specifically, the correspondence between the image coordinate system of the RGB video frame and the image coordinate system of the target image object can be determined by the size of the RGB video frame and the size of the target image object. The units of the size of the RGB video frame and the size of the target image object can both be: pixels.

[0083] Based on the first corresponding relationship and the second corresponding relationship established in S11, the first coordinate of the target pixel in the first coordinate system and the second coordinate in the second texture coordinate system of the YUV texture can be determined. According to actual calculation needs, the first coordinates and the second coordinates corresponding to multiple target pixels can be determined at one time.

[0084] For example, if the RGB values ​​of four target pixels can be obtained at one time and the four Y components can be quickly converted, the embodiment of the present invention can determine the four first coordinates corresponding to the four target pixels in the first coordinate system at one time when executing step S12, and then convert the RGB values ​​of the four target pixels into corresponding Y components, U components and V components at one time when executing step S13.

[0085] In a possible embodiment, when executing step S13, the RGB value at the first coordinate is converted into a YUV value, and based on the RGB value at the first coordinate corresponding to each target pixel point and the target YUV format, the Y component, U component and V component at the first coordinate corresponding to each target pixel point are determined using a preset conversion relationship. The preset conversion relationship may be a preset RGB to YUV conversion matrix, or a preset RGB to YUV conversion relationship formula.

[0086] Exemplarily, taking the target YUV format of 4:2:2 as an example, that is, 4 Y components share a pair of UV components, 4 Y components are normally sampled based on 4 RGB pixels, and then downsampled to generate a pair of UV components. In order to facilitate fast calculation, in an embodiment of the present invention, when executing step S12, the RGB values ​​at the 4 first coordinates corresponding to the 4 target pixels can be obtained at one time, and then the 4 Y components corresponding to the 4 first coordinates can be calculated at one time with the help of a preset conversion matrix. Further, downsampling is performed based on the RGB values ​​of the 4 target pixels to calculate 2 U components and 2 V components.

[0087] When executing step S14, based on the second coordinate determined in step S12 and the Y component, U component and V component calculated in step S13, the calculated YUV three components can be directly stored at the second coordinate in the YUV texture to obtain a texture in YUV format.

[0088] Exemplarily, in a possible embodiment, when the first target object is an RGB video frame, the YUV texture generation method provided in the embodiment of the present invention can refer to Figure 3 As shown:

[0089] Through step S11, the corresponding relationship between the image coordinate system of the RGB video frame and the texture coordinate system of the YUV texture is determined.

[0090] Through step S12, the target pixel points that need to be converted are determined: pixel point 4, pixel point 5, pixel point 6 and pixel point 7, and then the coordinate position of each target pixel point in the image coordinate system of the RGB video frame (i.e., the position of the light gray rectangular box on the left) is determined, and according to the correspondence between the image coordinate system of the RGB video frame and the texture coordinate system of the YUV texture, the second coordinate of the Y component corresponding to pixel point 4, pixel point 5, pixel point 6 and pixel point 7 (i.e., the position of the light gray rectangular box on the right), the second coordinate of the U component and the second coordinate of the V component (i.e., the position of the dark gray rectangular box on the right) are determined.

[0091] Through step S13, based on the RGB values ​​of pixel 4, pixel 5, pixel 6 and pixel 7, the values ​​of the Y component, U component and V component corresponding to each pixel are calculated respectively using a preset conversion relationship or a preset conversion matrix. Figure 3 The YUV storage format shown is 4:2:0, so by fully sampling pixel 4, pixel 5, pixel 6 and pixel 7, four Y components are calculated, and then pixel 4, pixel 5, pixel 6 and pixel 7 are downsampled, and a corresponding pair of U components and V components are calculated with the help of pixel 5 and pixel 7.

[0092] Through step S14, the calculated value of the Y component, the value of the U component and the value of the V component are respectively stored in the second coordinate of the YUV texture determined in step S12.

[0093] In another possible embodiment, when the first target object is an RGB texture, the YUV texture generation method provided in the embodiment of the present invention can refer to Figure 4 As shown, specifically:

[0094] Through step S11, a first corresponding relationship between the target image coordinate system and the texture coordinate system of the RGB texture, and a second corresponding relationship between the target image coordinate system and the YUV texture are established.

[0095] Through step S12, according to the first corresponding relationship, the first coordinates of the target pixel points (i.e., pixel point 4, pixel point 5, pixel point 6, and pixel point 7) in the target image coordinate system in the RGB texture are determined, that is, Figure 4 The first coordinate point of the gray frame area is shown (ie, R4-B7).

[0096] Through step S13, the RGB value of the first coordinate point is converted into YUV, that is, using the preset conversion relationship, R4, G4, B4...R7, G7, B7 are converted into the Y component, U component and V component corresponding to the target YUV storage format. Exemplarily, if the storage format of YUV is: 4:2:0, it is necessary to fully sample R4~B7, calculate and obtain 4 Y components (ie, Y4~Y7), and then downsample R4~B7, that is, sample and obtain the values ​​of R5, G5, B5 and R7, G7, B7, and use the sampled values ​​to calculate the corresponding U component and V component.

[0097] Through step S14, the converted YUV data is written into the second coordinate corresponding to the YUV texture.

[0098] In a possible embodiment, the second texture is obtained by:

[0099] A blank texture of the same size is generated based on the RGB texture. The blank texture is used to store the converted Y component, U component, and V component.

[0100] If the ratio of Y component, U component and V component in the YUV storage format is 4:4:4, that is, there is no situation where multiple Y components share a pair of UV components. At this time, the storage space occupied by each Y component, each U component and each V component is the same, which is the same as the storage space occupied by each R component, each G component and each B component in the original RGB texture. In this case, the blank texture is the second texture.

[0101] For some YUV storage formats, the ratio of Y component to UV component is not 1:1:1. For example, the YUV storage format is NV12 (i.e. YUV420), where the ratio of Y component, U component and V component is 4:2:0, that is, there are 4 Y components sharing a pair of UV components. If the size of the blank texture is not adjusted, the blank texture will not be able to fully store the Y components and UV components.

[0102] In order to ensure that the converted YUV three components can be completely and accurately stored in the YUV texture, in a possible embodiment, before executing step S11, the YUV texture generation method provided by the present invention further includes:

[0103] The width / height of the second texture is adjusted according to the width / height of the RGB texture so that the adjusted width / height of the second texture satisfies the proportional relationship between the target Y component, the U component, and the V component.

[0104] Specifically, in a possible embodiment, the width or height of the blank texture is adjusted according to the pre-divided storage space distribution of the Y component, the U component, and the V component.

[0105] For example, Figure 1c For example, if the storage space distribution of the pre-divided Y component, U component and V component is: Y component, V component and U component are distributed from top to bottom, the height of the blank texture needs to be adjusted. If the storage space distribution of the pre-divided Y component, U component and V component is: Y component, V component and U component are distributed from left to right, the width of the blank texture needs to be adjusted.

[0106] Taking the YUV storage format of 4:2:0 as an example, if the storage space distribution of the pre-divided Y component, U component and V component is: Y component, V component and U component are distributed from top to bottom, then the height of the blank texture needs to be adjusted to 3 / 2 times the height of the RGB texture (that is, adjusted to 3 / 2h, h is the height of the RGB texture). In this way, the extra 1 / 2h is used to store the converted U and V components.

[0107] In actual application scenarios, in order to speed up the execution of the above steps S11 to S14, in a possible embodiment, with the aid of the compute shader that can support parallel and arbitrary reading and writing of any coordinates in the image coordinate system, multiple compute shaders are used to execute the above steps S12 to S14 in parallel to improve the efficiency of generating YUV textures. Exemplary:

[0108] Taking the first target object as an RGB video frame as an example, using multiple compute shaders to perform the above steps S12-S14 may specifically include:

[0109] Step 1: Get the width and height of the RGB video frame. Specifically, the width and height of the RGB video frame are represented in the form of a two-dimensional vector. For example, vec2 imagesize = (sz.x, sz.y), where sz.x is the width of the RGB video frame and sz.y is the height of the RGB video frame.

[0110] Step 2: Determine the target pixel point, determine the specific pixel position of the target pixel point in the RGB video frame, and determine the second coordinate corresponding to the target pixel point in the YUV texture.

[0111] Exemplarily, the ID of the target pixel point and the specific coordinate position of the target pixel point are determined = (ID.x, ID.y). Figure 3 For example, assuming that the target pixel is pixel 4, the coordinate position of pixel 4 is determined as: (ID4.x, ID4.y).

[0112] Among them, if the coordinates of the target pixel point exceed the image size range of the RGB video frame, the target pixel point will not be processed, that is, if ID.x>sz.x, ID.y>sz.y, the target pixel point can be determined as an invalid pixel point and no subsequent processing is required.

[0113] Taking the YUV storage format of YUV420 as an example, the method for determining the second coordinate corresponding to the target pixel point in the YUV texture is specifically as follows:

[0114] Since the Y component is obtained based on full sampling of the target pixel points, the number of Y components is the same as the number of target pixels, that is, the number of coordinate points storing the Y component is the same as the number of target pixels, then it can be determined that the second coordinate of the Y component storing the target pixel point = the coordinate of the target pixel point (ID.x, ID.y). For example, the second coordinate of the Y component of pixel point 4 = (ID4.x, ID4.y).

[0115] Since the UV component is obtained based on the downsampling of the target pixel, the number of U components and the number of target pixels are in a ratio of 1:2, and since the embodiment of the present invention stores the U component and the V component together in the same YUV texture, after determining the position of the downsampled target pixel, it is necessary to determine the storage position of the UV component in combination with the storage space distribution of the pre-divided Y component, U component and V component.

[0116] For example, Figure 3 As shown, if the storage space distribution of the pre-divided Y component, U component and V component is: Y component, V component and U component are distributed from top to bottom. Because the U component and V component are downsampled by pixels, pixel 5 and pixel 7 are taken as the target pixel points for sampling. Therefore, it can be determined that the second coordinate of the V component storing the target pixel point = (ID.5, sz.y + (ID.5) / 2), and the second coordinate of the U component storing the target pixel point = ((ID.5) + 1, sz.y + (ID.5) / 2).

[0117] Step 3: Read the RGB value of the target pixel according to the coordinate value (ID.x, ID.y) of the target pixel. The RGB three components of the acquired target pixel are represented in the form of a three-dimensional vector. For example, vec4 color = (r, g, b, id), where id is the ID of the target pixel, and r, g, b are the R component, G component, and B component of the target pixel.

[0118] Step 4: Convert the RGB value obtained in step 3 into a Y component, a U component and a V component according to a preset conversion relationship (or a preset conversion matrix).

[0119] Step 5: Store the Y component, U component and V component converted in step 4 at the corresponding coordinates according to the second coordinates determined in step 2 to obtain a YUV texture.

[0120] In a possible embodiment, after the compute shader executes the above steps 1 to 5, a notification message is sent to the image processor to notify the image processor to obtain the Y component, U component and V component from the YUV texture. The image processor outputs the obtained Y component, U component and V component to the encoder, and the encoder encodes the YUV data. In this way, the image processor can directly read the target YUV data from a texture at one time, and the encoder can encode the YUV data without waiting for the image processor to read the Y component, U component and V component in batches, which effectively improves the encoding efficiency of the encoder, thereby increasing the number of video frames that can be encoded per unit time, which is conducive to improving the frame rate of the live display screen, making the live screen smoother.

[0121] On the other hand, an embodiment of the present invention provides a YUV texture generating device, such as Figure 5 As shown, the device comprises:

[0122] A correspondence establishing module 51; used to establish a first correspondence between a first coordinate system of a first target object and a target image coordinate system, and a second correspondence between a texture coordinate system of a second texture and the target image coordinate system; wherein the first target object is an RGB texture or an RGB video frame, and the second texture is a YUV texture;

[0123] A first determination module 52; used to determine the first coordinates corresponding to the target image coordinate point in the first coordinate system according to the first corresponding relationship, and to determine the second coordinates corresponding to the target image coordinate point in the second texture coordinate system according to the second corresponding relationship;

[0124] Conversion module 53; used for converting the RGB value at the first coordinate into YUV;

[0125] Storage module 54; used for writing the YUV into the second coordinate of the YUV texture.

[0126] In a possible embodiment, the YUV texture generating device provided by the embodiment of the present invention further includes:

[0127] The second determination module 55 is used to determine the texture coordinate system of the RGB texture as the first coordinate system if the first target object is an RGB texture; and to determine the image coordinate system of the RGB video frame as the first coordinate system if the first target object is an RGB video frame.

[0128] In a possible embodiment, the YUV texture generating device provided by the embodiment of the present invention further includes:

[0129] The size adjustment module 56 is used to adjust the width / height of the second texture according to the width / height of the first target object so that the adjusted width / height of the second texture satisfies the proportional relationship between the target Y component, U component and V component.

[0130] In a possible embodiment, the YUV texture generating device provided by the embodiment of the present invention further includes:

[0131] The notification module 57 is used to send a notification message to the image processor to notify the image processor to read the YUV data and forward the read YUV data to the encoder.

[0132] The embodiment of the present invention further provides an electronic device, such as Figure 6As shown, it includes a processor 61, a communication interface 62, a memory 63 and a communication bus 64, wherein the processor 61, the communication interface 62, and the memory 63 communicate with each other through the communication bus 64.

[0133] Memory 63, used for storing computer programs;

[0134] The processor 61 is used to execute the program stored in the memory 63 to implement the following steps:

[0135] Establishing a first correspondence between a first coordinate system of a first target object and a target image coordinate system, and a second correspondence between a second texture coordinate system and the target image coordinate system; wherein the first target object is an RGB texture or an RGB video frame, and the second texture coordinate system is a coordinate system of a YUV texture;

[0136] Determine, according to the first corresponding relationship, a first coordinate corresponding to the target pixel point in the first coordinate system, and determine, according to the second corresponding relationship, a second coordinate corresponding to the target pixel point in the second texture coordinate system;

[0137] Convert the RGB value at the first coordinate to YUV;

[0138] Write the converted YUV into the second coordinate of the YUV texture.

[0139] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0140] The communication interface is used for communication between the above terminal and other devices.

[0141] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0142] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0143] In another embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the YUV texture generation method described in any of the above embodiments is implemented.

[0144] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the YUV texture generation method described in any one of the above embodiments.

[0145] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0146] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0147] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0148] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A YUV texture generation method, characterized in that: The method comprises: Establishing a first correspondence between a first coordinate system of a first target object and a target image coordinate system, and a second correspondence between a texture coordinate system of a second texture and the target image coordinate system; wherein the second texture is a YUV texture, and the first target object is an RGB texture or an RGB video frame, if the first target object is an RGB texture, determining the texture coordinate system of the RGB texture to be the first coordinate system, and determining a first correspondence between the first coordinate system and the target image coordinate system according to a storage format of the RGB texture and a size corresponding to the target image; if the first target object is an RGB video frame, determining the image coordinate system of the RGB video frame to be the first coordinate system, and determining a first correspondence between the first coordinate system and the target image coordinate system according to the size of the RGB video frame and a size corresponding to the target image; Determine, according to the first corresponding relationship, a first coordinate corresponding to the coordinate of the target pixel point in the target image coordinate system in the first coordinate system, and determine, according to the second corresponding relationship, a second coordinate corresponding to the coordinate of the target pixel point in the target image coordinate system in the second texture coordinate system; Convert the RGB value at the first coordinate into YUV; The YUV is written into the YUV texture at the second coordinate.

2. The method according to claim 1, characterized in that The method further comprises: According to the width / height of the first target object, the width / height of the second texture is adjusted so that the adjusted width / height of the second texture satisfies the proportional relationship among the target Y component, U component and V component.

3. The method according to claim 1, characterized in that The method further comprises: A notification message is sent to the image processor to notify the image processor to read the YUV data and forward the read YUV data to the encoder.

4. A YUV texture generating device, characterized in that: The device comprises: a correspondence establishing module, used for establishing a first correspondence between a first coordinate system of a first target object and a target image coordinate system, and a second correspondence between a texture coordinate system of a second texture and the target image coordinate system; wherein the second texture is a YUV texture, and the first target object is an RGB texture or an RGB video frame; if the first target object is an RGB texture, the texture coordinate system of the RGB texture is determined to be the first coordinate system, and the first correspondence between the first coordinate system and the target image coordinate system is determined according to the storage format of the RGB texture and the size corresponding to the target image; if the first target object is an RGB video frame, the image coordinate system of the RGB video frame is determined to be the first coordinate system, and the first correspondence between the first coordinate system and the target image coordinate system is determined according to the size of the RGB video frame and the size corresponding to the target image; a first determining module, configured to determine, according to the first corresponding relationship, a first coordinate corresponding to the coordinate of the target pixel point in the target image coordinate system in the first coordinate system, and to determine, according to the second corresponding relationship, a second coordinate corresponding to the coordinate of the target pixel point in the target image coordinate system in the second texture coordinate system; A conversion module, used for converting the RGB value at the first coordinate into YUV; A storage module is used to write the YUV into the second coordinate of the YUV texture.

5. The device according to claim 4, characterized in that The device also includes: A size adjustment module is used to adjust the width / height of the second texture according to the width / height of the first target object so that the adjusted width / height of the second texture satisfies the proportional relationship between the target Y component, U component and V component.

6. The device according to claim 4, characterized in that The device also includes: The notification module is used to send a notification message to the image processor to notify the image processor to read the YUV data and forward the read YUV data to the encoder.

7. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 3 when executing a program stored in a memory.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 3 are implemented.

Images