Method of generating osd index data

Abstract

A method for generating a plurality of OSD index data for a back-end circuit for processing a plurality of image data to be displayed on a display device is disclosed. The method comprises the steps of: receiving the plurality of image data from an application processor; and retrieving information of a detection layer embedded in the plurality of image data, wherein the information of the detection layer indicates the plurality of OSD index data corresponding to at least one user interface layer in the plurality of image data.

Description

Technical Field

[0001] The present invention relates to a method for a display device, and more particularly to a method for generating OSD (On-Screen Display) index data for a display device. Background Technology

[0002] Back-end circuits (BE circuits) (such as back-end chips, also known as image processing circuits or image post-processing circuits) are typically used in display systems to process the image data to be displayed. After the application processor (AP) generates a frame of image data, it sends that frame to the back-end circuit. The back-end circuit then performs various image post-processing operations on the received image data, such as frame rate conversion, noise reduction, and contrast adjustment, thereby improving visual effects and / or meeting the specifications of the display device. The image data after these processing steps is then transmitted to the panel for display.

[0003] An application processor can combine multiple image layers to generate image data. These image layers are generated by different user interface (UI) applications or image sources. Generally, image content can consist of a video layer and at least one user interface layer. The video layer includes video content from a video source as a background; each user interface layer can be generated by a user interface application and can be embedded in the video layer to blend with the video content. The application processor can then send the combination of all image layers to back-end circuitry for post-processing.

[0004] To facilitate post-processing, the back-end circuitry needs to know whether the image data for each pixel was generated by the video layer or the user interface layer. For example, in a mobile phone's output image, the background wallpaper and push notifications require different processing methods; therefore, the back-end circuitry needs to identify the image type. However, the image data output from the application processor typically does not include this information. In existing technology, the application processor can send OSD (On-Screen Display) index data via an additional transmission interface to indicate whether the image data for each pixel comes from the video layer or the user interface layer. In this way, the back-end circuitry can obtain an image display bitmap indicating the position of the user interface layer and the background video, thus enabling post-processing based on the OSD information.

[0005] However, sending OSD index data to the back-end circuitry via the application processor has several drawbacks. For example, OSD index data requires additional transmission interfaces or bandwidth to reach the back-end circuitry, resulting in additional hardware costs and higher power consumption. Since the application processor needs to be able to determine the OSD index data, computational resources must be allocated to check whether each pixel contains the user interface image after the video layer and user interface layer are mixed. Furthermore, significant storage resources are required to store the OSD index data. Moreover, the back-end circuitry struggles to map the received OSD index data to the correct frame number and position, requiring substantial operations to synchronize the OSD index data and image content. Therefore, improvements to the existing technology are necessary. Summary of the Invention

[0006] Therefore, the main objective of this invention is to provide a novel method for generating OSD (On-Screen Display) index data to solve the above-mentioned problems.

[0007] An embodiment of the present invention discloses a method for generating multiple OSD index data for a back-end circuit (BE circuit) that processes multiple image data to be displayed on a display device. The method includes the following steps: receiving the multiple image data from an application processor (AP); and retrieving information from a detection layer embedded in the multiple image data, wherein the detection layer information indicates the multiple OSD index data corresponding to at least one user interface (UI) layer within the multiple image data.

[0008] Another embodiment of the present invention discloses a method for generating a plurality of OSD index data for an application processor, the application processor being used to generate a plurality of image data to be displayed on a display device. The method includes the following steps: embedding at least one user interface layer and a detection layer into a video layer for display on the display device; and transmitting the plurality of image data, which is a mixture of the at least one user interface layer, the detection layer, and the video layer, to a back-end circuit. The detection layer is used to detect the at least one user interface layer. Attached Figure Description

[0009] Figure 1A This is a schematic diagram of an exemplary image pattern.

[0010] Figure 1B It shows the corresponding Figure 1A OSD index data of the image pattern in the image.

[0011] Figure 2The following is a flowchart illustrating the operation of the system according to Embodiment 1 of the present invention.

[0012] Figure 3 A detailed implementation of embedding a detection layer into an image is shown.

[0013] Figure 4 An image pattern of an exemplary detection layer is shown.

[0014] Figure 5 This is a schematic diagram illustrating how an image is mixed with a detection layer to find OSD index data in an embodiment of the present invention.

[0015] Figure 6 A detailed implementation of the image reconstruction operation is shown.

[0016] Figure 7 This is a flowchart of a first embodiment of the present invention.

[0017] The reference numerals in the attached figures are explained as follows:

[0018] 20 Display System

[0019] 200 application processors

[0020] 210 Back-end circuit

[0021] L1~L3 User Interface Layer

[0022] L i L UI L video Image data

[0023] α i α UI transmittance parameters

[0024] 70 Process

[0025] Steps 700-712 Detailed Implementation

[0026] Please refer to Figure 1A and Figure 1B , Figure 1A This is a schematic diagram of an exemplary image pattern. Figure 1B It shows the corresponding Figure 1AThe image pattern contains OSD (On-Screen Display) index data. In one embodiment, the application processor (AP) can send image data about the image pattern to the back-end circuit (BE circuit). The image pattern includes a background image and a user interface (UI) image. Specifically, the image data output by the application processor consists of a video layer and one or more user interface layers, with the image data of different layers coming from different image sources. For example, the image content of the video layer can be an image generated by decoding video files or network streaming data, while the user interface layer may include menus, push notifications, status bars, time, battery information, real-time information, and / or various information blocks that can be overlaid on the background image / video. After the images of the video layer and the user interface layer are mixed, the application processor can send the mixed image data to the back-end circuit, which then processes the image data and transmits it to the display device for display.

[0027] An image display bitmap maps a bit array to a frame of image data, indicating which pixels display the video layer image and which pixels display the user interface layer image. In one embodiment, the OSD index data can be set to "1" if the corresponding pixel displays the user interface image, and set to "0" if the corresponding pixel displays the background image. Figure 1B As shown. The size of the image display bitmap is exactly the same as the resolution of the displayed image, where each OSD index data can be mapped to one pixel; or, the image display bitmap can have a smaller size, such that image information of some adjacent pixels can be indicated by one OSD index data. In another embodiment, one OSD index data for one pixel or several adjacent pixels can be carried in several bits, which can be used to store information other than the presence of the user interface layer, including light transmittance, image blending ratio, etc. For example, the OSD index data may include several bits to represent values ​​between "0" and "1", where "0" indicates that only the background image exists, and "1" indicates that the user interface image completely occupies the background image, and other values ​​represent the proportion of the user interface image in that pixel in the blending of the user interface and the background image. Generally, user interface images do not usually require excessive image processing in the back-end circuitry; therefore, the back-end circuitry should obtain the relevant OSD index data based on the information carried by the image display bitmap and perform image post-processing accordingly.

[0028] In one embodiment, a detection layer can be deliberately embedded in the hybrid image of the application processor to obtain OSD index data. The detection layer has a predetermined image pattern, and the back-end circuitry is aware of this predetermined image pattern. Therefore, the back-end circuitry can retrieve the OSD index data based on the image data of the detection layer. In this case, the additional steps and resources used for judging, storing, transmitting, and synchronizing the OSD index data can be omitted.

[0029] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the operation of system 20 according to Embodiment 1 of the present invention. Figure 2 As shown, the display system 20 includes an application processor 200 and a back-end circuit 210. The display system 20 may also include a display device, such as a panel or display screen (not shown). The application processor 200 can be used to mix a video layer and a user interface layer. More specifically, the application processor 200 can embed user interface layers L1-L3 into the video layer, wherein each user interface layer L1-L3 may include menus, push notifications, and / or information blocks to be displayed on the display device. The application processor 200 can also embed a detection layer into the video layer, wherein the detection layer can be used to detect user interface layers L1-L3. Therefore, the application processor 200 transmits image data containing a mixture of user interface layers L1-L3, the detection layer, and the video layer to the back-end circuit 210.

[0030] In one embodiment, the application processor 200 may be a system-on-chip (SoC) or any other type of main processing circuitry used to generate image content including video and user interfaces, and is equipped with an operating system (such as Android) for installing various applications, but is not limited thereto. Common examples of SoCs include Qualcomm's Snapdragon series. The back-end circuitry 210 may be a graphics processing unit (GPU), a discrete graphics processing unit (GPU), a discrete display chip, a discrete motion estimation and motion compensation (MEMC) chip, or image processing circuitry in any other electronic device with display capabilities, but is not limited thereto. Common examples of back-end circuitry include Sony's X1 processor. In another embodiment, the application processor 200 may also be a system-on-chip for a television set-top box.

[0031] After receiving image data, the back-end circuit 210 can extract the detection layer information embedded in the image data and obtain the OSD index data corresponding to the image data according to the indication of the detection layer information. The OSD index data includes multiple OSD index data points used to indicate whether the corresponding pixel has a user interface image. Since the back-end circuit 210 knows the image information of the embedded detection layer, it can remove the image of the detection layer based on the known information, thereby reconstructing the image content. It should be noted that the image pattern of the detection layer should not be displayed on the display device; therefore, the image of the detection layer must be removed before the back-end circuit 210 outputs the image data.

[0032] Figure 3 A detailed implementation of embedding a detection layer into an image is illustrated. The image data to be displayed may include a video layer and several user interface layers (in this example, three user interface layers L1 to L3). Each pixel on each embedded user interface layer has image data and an associated parameter α, where the value of α indicates the transmittance of the pixel on that layer. The final image data to be displayed is determined based on the image data of each layer and the transmittance parameter α of the user interface layer. In one embodiment, the transmittance parameter α may be set between "0" and "1", where α = 0 indicates that the image of the pixel on that layer is completely transparent, allowing the image below to be displayed; while α = 1 indicates that the image of the pixel on that layer is completely opaque, completely obscuring the image below.

[0033] To detect user interface layers L1 to L3 and determine the OSD index data corresponding to user interface layers L1 to L3, image data L can be used. i and transmittance parameter α i A detection layer is embedded between the user interface layers L1-L3 and the video layer. The user interface layers L1-L3, the detection layer, and the video layer overlap each other to create an image output by the application processor 200. Figure 4 An image pattern of an exemplary detection layer is shown. For example... Figure 4 As shown, the detection layer can have a completely black image, and the transmittance parameter α of the detection layer... i It presents a checkerboard pattern; in other words, the detection layer has a light-transmitting area and an opaque area, which are arranged alternately to form a checkerboard pattern. Figure 4 Each white or black block in the checkerboard pattern may comprise a single pixel or an array of pixels. In a preferred embodiment, each white or black block in the checkerboard represents one pixel, such that the actual number of blocks included in the checkerboard on the detection layer is much greater than the number of blocks in the checkerboard. Figure 4 As shown. In this case, for every two adjacent pixels, one is assigned to the light-transmitting area and the other is assigned to the opaque area, which can achieve better screen display index data and image reconstruction results.

[0034] In the opaque area, the image information of the video layer is completely obscured, and only the user interface image is displayed (if a user interface image exists). Therefore, the back-end circuit 210 can extract the image information in the opaque area to determine the corresponding OSD index data. More specifically, assuming the detection layer is a completely black image, if the back-end circuit 210 finds that the image of a pixel in the opaque area is black, it can determine that the pixel displays the image of the detection layer and therefore the pixel does not have a user interface layer, thus setting the corresponding OSD index data to "0"; if the back-end circuit 210 finds that the image of a pixel in the opaque area is not black, it can determine that the pixel displays a user interface image and therefore the pixel may have at least one user interface layer (because the user interface layer above is not obscured), thus setting the corresponding OSD index data to "1".

[0035] Please refer to Figure 4 Matching Figure 3 As shown, for a specific pixel, assuming the total image data of the user interface layers L1 to L3 is L UI The transmittance parameter is α. UI That is, the image parameters generated by the combination of user interface layers L1 to L3 are image data L. UI and transmittance parameter α UI The image data for this pixel in the video layer is L. video As described above, the detection layer includes alternately arranged transparent and opaque areas, wherein the transmittance parameter α of the transparent areas... i The transmittance parameter α of the opaque region is equal to "0". i It equals "1". Furthermore, if the detection layer is a completely black image, its image data equals "0". Therefore, if a specific pixel is located in the light-transmitting area (α)... i If (=0), then the output image data for this pixel can be obtained in the following ways:

[0036] Output image data = L video ×(1-α UI )+L UI ×α UI ;

[0037] This is equivalent to the image content composed of a video layer and a user interface layer that is to be displayed on the display device. If a specific pixel is located in the opaque area (α) i If = 1), then the output image data of this pixel can be obtained in the following ways:

[0038] Output image data = L UI ×α UI ;

[0039] In this case, the video layer image is completely obscured, so the user interface layers L1 to L3 located above the detection layer can be easily detected.

[0040] As described above, the image pattern of the detection layer is known information to the back-end circuit 210, so the back-end circuit 210 can obtain OSD index data based on this image information. Since the location of the opaque area of ​​the detection layer can only display the user interface image, the back-end circuit 210 can detect the OSD index data corresponding to the user interface layers L1 to L3 at the location of the opaque area overlapping the detection layer. For pixels located in the transparent area, their corresponding OSD index data cannot be directly detected. Therefore, the back-end circuit 210 can predict the OSD index data of the transparent area by interpolation, for example, by referring to pixels in adjacent opaque areas to calculate the OSD index data of each transparent area. In one embodiment, the back-end circuit 210 can combine the OSD index data detected in the opaque area and the OSD index data calculated in the transparent area to obtain an image display bitmap corresponding to an image frame.

[0041] It is worth noting that the detection layer may cause changes to the image to be output to the display device, especially in opaque areas. Therefore, the back-end circuit 210 is required to reconstruct the original image data where the detection layer image is absent. As mentioned above, the image in the transparent area is not affected by the detection layer. Therefore, based on the image data in the transparent area, a complete frame of image data can be reconstructed, thereby restoring the image to be displayed on the display device. In one embodiment, the image frame can be reconstructed by interpolation. That is, the back-end circuit 210 can refer to pixels in adjacent transparent areas to determine the image data in the opaque area and further transmit the reconstructed image frame to the display device. In one embodiment, the reconstructed image frame includes restored user interface layer information, which can be further used to obtain an image display bitmap with higher accuracy.

[0042] Therefore, preferably, the image data and transmittance parameters of the detection layer can be allocated such that the transparent and opaque areas are arranged alternately (e.g., forming a checkerboard or similar pattern) so that the output image can be reconstructed by interpolation.

[0043] Please refer to Figure 5 Its adoption Figure 1A Taking the image content as an example, a detection layer is mixed in to find the OSD index data. For example... Figure 5 As shown, the video layer displays a background image (containing an apple), and the transmittance parameter α of all pixels on the video layer is equal to "1" (i.e., opaque, where α = 1 is represented by white). A user interface layer overlaid on the video layer displays an information block on the left, with the transmittance parameter α in the information block area. UI The transmittance parameter α at other locations is equal to "1".UI It equals "0" (where α) UI =1 is represented in white and α UI =0 is represented by black). A detection layer is embedded between the user interface layer and the video layer. The detection layer has completely black image data, and its transmittance parameter α i It features a checkerboard pattern. The application processor 200 can mix the image content from the user interface layer, the detection layer, and the video layer, and then send the mixed image data to the back-end circuit 210. Based on the information from the detection layer, the back-end circuit 210 can extract the OSD index data to determine that the OSD index data in the information block area is equal to "1", while the OSD index data in other locations is equal to "0". Based on the image data within the light-transmitting area of ​​the detection layer, the back-end circuit 210 can also reconstruct the output image.

[0044] Figure 6 A detailed implementation of the image reconstruction operation is illustrated. When the transmittance parameters of the detection layer have a checkerboard pattern, image data can be easily reconstructed using interpolation based on four adjacent pixels. However, if the transmittance parameters of the detection layer do not have a checkerboard pattern or if the opaque area is large and covers several adjacent pixels, image data in the opaque area can be reconstructed or restored by referring to more distant pixels.

[0045] In one embodiment, the image pattern of the corresponding detection layer may differ for different image frames. For example, different checkerboard patterns can be used as the detection layer for two consecutive image frames; that is, a transparent pixel in the current frame may be an opaque pixel in the next frame, and / or an opaque pixel in the current frame may be a transparent pixel in the next frame. In this case, the back-end circuit can also reconstruct image data based on the previous image frame and / or the next image frame, thereby achieving optimized reconstruction results.

[0046] Generally, the user interface layer embedded in the video layer is used to generate the image to be displayed on the display device. However, the purpose of the detection layer is to detect the user interface layer, and the image pattern of the detection layer needs to be removed from the image data during the reconstruction process. Therefore, the image of the detection layer is not displayed on the display device, a characteristic of the detection layer that differs from other user interface layers.

[0047] Furthermore, for successful reconstruction of the original image, the embedded detection layer needs to consist of both transparent and opaque regions, and the arrangement of the transparent regions enables smooth reconstruction. In one embodiment, a majority of pixels on an image frame are assigned to transparent regions, while only a small number of pixels are assigned to opaque regions for detecting OSD index data. Alternatively or additionally, the detection layer does not contain a large area (at least larger than a specific area or comprising at least a specific number of pixels) where all pixels are assigned to opaque regions; that is, within a large area of ​​the detection layer, at least one pixel is assigned to a transparent region. In other words, the detection layer does not contain a large number of clustered opaque pixels, thus enabling the correct reconstruction of the original hybrid image without the detection layer.

[0048] In addition, OSD index data can only be detected in opaque areas, but not directly in transparent areas. Therefore, OSD index data in transparent areas can be obtained by referring to adjacent pixels. Furthermore, if a user interface image on a user interface layer only appears in the transparent area of ​​the detection layer, then that user interface layer cannot be detected successfully.

[0049] Furthermore, the light-transmitting and opaque areas can be arranged in any manner, not limited to the checkerboard pattern proposed in this specification. In one embodiment, the arrangement of light-transmitting and opaque pixels can be appropriately adjusted at different locations. For example, in locations where a user interface layer image is more likely to appear (e.g., areas closer to the panel or screen boundary), opaque pixels can be arranged with a higher density to achieve optimized OSD index data detection. Conversely, in locations where user interface layer images are less frequent (e.g., the central display area), opaque pixels can be arranged with a lower density (with a larger light-transmitting area), or even no opaque pixels may be present at that location, making it easier to reconstruct the original image and improving the accuracy of image reconstruction.

[0050] It is worth noting that the purpose of this invention is to provide a method for generating OSD index data, which can be achieved by embedding a detection layer into the original output image. Those skilled in the art can modify or vary this method, but are not limited thereto. For example, in the above embodiment, the transmittance parameter in the transparent area is "0" and the transmittance parameter in the opaque area is "1". However, in another embodiment, the transmittance parameter of the detection layer can be set to any value and / or adjusted in an appropriate manner. For example, the transmittance parameter in the opaque area of ​​the detection layer can have a value close to "1", such as "0.95" or "0.9". In this case, the back-end circuit can still determine the OSD index data based on the image in the opaque area, and since the opaque area also includes video layer image information that helps with image reconstruction, the reconstruction of the output image is more efficient. Furthermore, in the above embodiment, the detection layer has a completely black image; however, in another embodiment, other colors can also be used. As long as the color of the detection layer is different from the main color of the user interface image and the back-end circuit knows its color information, the corresponding user interface layer can be successfully detected. In another alternative embodiment, multiple colors may be used on a detection layer, and / or the detection layers for different image frames may be composed of different colors to achieve different detection effects.

[0051] Furthermore, in the above embodiments, the detection layer is embedded above the video layer and below all user interface layers. In another embodiment, the detection layer may also be embedded between the video layer and one or more target user interface layers, and OSD index data corresponding to the target user interface layer may be obtained. For example, in... Figure 3 In the image layer structure shown, when the detection layer is embedded between user interface layers L1 and L2, it only detects user interface layers L2 and L3 and obtains the corresponding OSD index data (user interface layer L1 is located below the detection layer and is not detected). In fact, depending on the implementation of image layer blending in the application processor, the detection layer can be embedded in any way to detect OSD index data according to system requirements. For example, the back-end circuitry may need to process certain user interface images in different ways and obtain the corresponding OSD index data for those user interface layers.

[0052] The above-described process for generating OSD index data can be summarized as a single procedure 70, such as... Figure 7 As shown. Process 70 can be implemented in a display system having an application processor and a back-end circuit, such as... Figure 2 The display system 20 shown, process 70 includes the following steps:

[0053] Step 700: Begin.

[0054] Step 702: The application processor generates a detection layer for detecting at least one user interface layer.

[0055] Step 704: The application processor embeds at least one user interface layer and a detection layer into the video layer.

[0056] Step 706: The application processor transmits image data, which is a mixture of at least one user interface layer, a detection layer, and a video layer, to the back-end circuitry.

[0057] Step 708: The back-end circuit retrieves information from the detection layer embedded in the image data, wherein the information from the detection layer indicates OSD index data in the image data corresponding to at least one user interface layer.

[0058] Step 710: The back-end circuitry removes the information from the detection layer to reconstruct a frame of image data to be displayed on the display device.

[0059] Step 712: End.

[0060] For details on the operation and changes of process 70, please refer to the explanation in the above paragraphs, which will not be repeated here.

[0061] In summary, this invention provides a method for intentionally embedding a detection layer into a mixed image to generate OSD index data. The detection layer may include translucent and opaque regions with different transmittance parameters, arranged in a checkerboard pattern. The translucent regions can display both the user interface image and the video layer image, while the opaque regions only display the user interface image, with the video layer obscured. Therefore, OSD index data can be detected based on image information in the opaque regions, and OSD index data in the translucent regions can be calculated by referencing pixels in adjacent opaque regions, thereby generating an image display bitmap. Since the translucent regions contain information from the original output image, image data in the opaque regions can be reconstructed by interpolation, referencing pixels in adjacent translucent regions. This allows for more efficient extraction of OSD index data from image information, eliminates the need for additional transmission interfaces or bandwidth for transmitting OSD index data, and makes synchronization between OSD index data and image content easier and more convenient.

[0062] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating multiple OSD index data, used in a back-end circuit for processing multiple image data to be displayed on a display device, the method comprising: Receive the multiple image data from an application processor; Information from a detection layer embedded in the plurality of image data is extracted, wherein the information of the detection layer indicates the plurality of OSD index data corresponding to at least one user interface layer in the plurality of image data, and the detection layer includes a light-transmitting area and a light-blocking area, wherein the image of the light-blocking area indicates whether the corresponding pixel exists in the image of the at least one user interface layer; and The image information of the opaque area is detected to determine the plurality of OSD index data corresponding to the pixel positions on the at least one user interface layer that overlap with the opaque area of ​​the detection layer.

2. The method as described in claim 1, characterized in that, The image of the detection layer is not displayed on the display device.

3. The method as described in claim 1, characterized in that, Also includes: Based on the multiple image data in the light-transmitting area of ​​the detection layer, a frame of image data to be displayed on the display device is reconstructed.

4. The method as described in claim 1, characterized in that, At least one pixel is assigned to the light-transmitting area within a large region of the detection layer.

5. The method as described in claim 1, characterized in that, For pixels on the detection layer, the pixel density of the opaque area at a location where the image of the at least one user interface layer is more likely to appear is higher than the pixel density of the opaque area at another location where the image of the at least one user interface layer is less likely to appear.

6. A method for generating a plurality of OSD index data, for an application processor, the application processor being used to generate a plurality of image data to be displayed on a display device, the method comprising: At least one user interface layer and one detection layer are embedded in a video layer for display on the display device; as well as Transmit the plurality of image data, which is a mixture of the at least one user interface layer, the detection layer, and the video layer, to a back-end circuit; The detection layer is used to detect the at least one user interface layer. The detection layer includes a light-transmitting area and an opaque area. The image of the opaque area indicates whether the corresponding pixel has the image of the at least one user interface layer. The multiple OSD index data are determined by detecting the image information of the pixel positions that overlap with the opaque area.

7. The method as described in claim 6, characterized in that, The image of the detection layer is not displayed on the display device.

8. The method as described in claim 6, characterized in that, Also includes: The detection layer is embedded between the at least one user interface layer and the video layer.

Images