Media data processing method, display device, and storage medium

Abstract

The application discloses a media data processing method, a display device and a storage medium. The method comprises the following steps: receiving media data sent by a projection device, wherein the media data comprises image data; decoding, rendering and displaying the image data on the target display device; when the image data conforms to full color scale data, decoding and rendering the image data in a first mode, so that the image data displayed on the target display device still conforms to the full color scale data; wherein the first mode is to process the image data by a decoding and rendering separation mode; when the image data conforms to limited color scale data, decoding and rendering the image data in a second mode, so that the image data displayed on the target display device still conforms to the limited color scale data. The media data processing method, the display device and the storage medium provided by the application have the advantage of avoiding color loss.

Description

Media data processing method, display device and storage medium Technical Field

[0001] The embodiments of the present application relate to the field of screen projection technology, and in particular to a media data processing method, a display device, and a storage medium. Background Art

[0002] Screen casting technology involves transferring the content displayed on one terminal to another (e.g., a PC and an Android device) via screenshot encoding and other methods, where it is decoded and displayed. Fluency and real-time performance are key factors in evaluating the quality of screen casting. Good fluency means no lag or distortion during the casting process, and a frame rate of at least 20fps.

[0003] However, when the content displayed on an iOS device is projected onto an Android device, the image displayed on the Android device suffers from color loss.

[0004] Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a media data processing method, a display device, and a storage medium, which can solve the problem of color loss in the images displayed by Android devices.

[0006] To achieve the above-mentioned purpose, one aspect of an embodiment of the present application provides a method for processing media data, which is used for a target display device of an Android system, and the target display device is configured with a screen projection application. The method is at least partially implemented based on the screen projection application, and the method includes: receiving media data sent by the screen projection device, the media data including image data; decoding, rendering and displaying the image data on the target display device; when the image data conforms to full-color-scale data, decoding and rendering the image data in a first manner so that the image data displayed on the target display device still conforms to full-color-scale data; wherein the first manner is to process the image data by separating decoding and rendering; when the image data conforms to limited-color-scale data, decoding and rendering the image data in a second manner so that the image data displayed on the target display device still conforms to limited-color-scale data.

[0007] Through the above scheme, the full-color image data is processed in the first way of separating decoding and rendering, so that the component value range of the color space of the full-color image data will not change during the processing process, so that the image data still conforms to the full-color data after rendering, and thus the image finally displayed on the target display device is a full-color image, thereby avoiding color loss and improving the screen projection effect of the target display device.

[0008] Optionally, the Android system includes a first module and a second module; when the image data conforms to full-color-scale data, the image data is decoded and rendered in a first manner, including: when the image data conforms to full-color-scale data, the screen projection application calls the first module to decode the image data, and the screen projection application calls the second module to render the decoded image data.

[0009] Through the above solution, the screen projection application calls the first module to decode the full-color image data and calls the second module to render the full-color image data, so that each step of processing the full-color image data is controlled to be performed in a specific manner, and during this period, there is no change in the component value range of the color space, thereby avoiding color loss.

[0010] Optionally, the first module includes a MediaCodec interface; the screen projection application calls the first module to decode the image data, including: the screen projection application calls the MediaCodec interface to decode the media data; when the screen projection application calls the MediaCodec interface, the surface parameter of the MediaCodec interface is idle.

[0011] Through the above solution, the surface parameter of MediaCodec is idle, indicating that no rendering area information is provided, so as to avoid direct rendering after MediaCodec decoding, which may cause color loss.

[0012] Optionally, the second module includes a conversion interface and a rendering interface; the screen projection application calls the second module to render the decoded image data, including: the screen projection application calls the conversion interface to convert the decoded image data into RGB format data; the screen projection application calls the rendering interface to render the RGB format data to the canvas provided by the target display device.

[0013] Through the above solution, the conversion interface is used to realize data format conversion, and the rendering interface is used to realize rendering.

[0014] Optionally, the media data processing method further includes: determining whether the image data conforms to full-color-scale data or limited-color-scale data.

[0015] Optionally, the determining whether the image data conforms to full-color scale data or limited-color scale data includes: determining whether the full-color scale parameters in the media image data are configured; when the full-color scale parameters in the image data are configured, determining that the image data conforms to full-color scale data; when the full-color scale parameters in the image data are idle, determining that the image data conforms to limited-color scale data.

[0016] Optionally, when the image data conforms to limited color level data, decoding and rendering the image data in a second manner includes: when the image data conforms to limited color level data, the screen projection application generates rendering area information; the screen projection application also calls the MediaCodec interface to decode the media image data, and renders the decoded image data to the canvas provided by the target display device; wherein, when the screen projection application calls the MediaCodec interface, the surface parameters of the MediaCodec interface are configured based on the rendering area information.

[0017] Through the above solution, the MediaCodec interface is used to decode the media image data, and the decoded image data is rendered in the rendering area, so as to improve the decoding and rendering speed.

[0018] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application further provides a media data processing method for a target display device of an Android system, wherein the target display device is configured with a media application, and the method is at least partially implemented based on the media application, and the method includes: receiving media data, wherein the media data includes image data; when the image data conforms to full-color scale data, processing the image data by separating decoding and rendering, and displaying the rendered image data on the target display device, so that the image data displayed on the target display device still conforms to full-color scale data.

[0019] Optionally, the Android system has a first module and a second module; when the image data conforms to full-color-scale data, processing the media image data by separating decoding and rendering includes: when the image data conforms to full-color-scale data, the media application calls the first module to decode the image data, and the media application calls the second module to render the decoded image data.

[0020] Optionally, the first module includes a MediaCodec interface; the media application calling the first module to decode the image data includes: the media application calling the MediaCodec interface to decode the image data; when the media application calls the MediaCodec interface, the surface parameter of the MediaCodec interface is idle.

[0021] Optionally, the second module includes a conversion interface and a rendering interface; the media application calls the second module to render the decoded image data, including: the media application calls the conversion interface to convert the decoded image data into RGB format data; the media application calls the rendering interface to render the RGB format data onto the canvas provided by the target display device.

[0022] Optionally, the media application includes at least one of a screen projection application and an image processing application.

[0023] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application further provides a display device of an Android system, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the media data processing method as described above.

[0024] To achieve the above objectives, another aspect of the embodiments of the present application further provides a computer-readable storage medium having computer instructions stored thereon, which implement the above-mentioned media data processing method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 is a schematic diagram of an application scenario of a screen projection system;

[0027] FIG2 is a screen display of a screen projection device and a receiving device of an Android system in the related art;

[0028] FIG3 is a diagram of the system architecture provided in an embodiment of the present application;

[0029] FIG4 is a flowchart of a method for processing media data provided by an embodiment of the present application;

[0030] FIG5 is a flowchart of another method for processing media data provided by an embodiment of the present application;

[0031] FIG6 is a schematic diagram of a display device of an Android system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] Screen projection refers to the synchronous display of the current screen image of the projection device on the screen of the receiving device. The projection device and the receiving device can be the same type of electronic devices, such as both the projection device and the receiving device can be laptops; the projection device and the receiving device can also be different types of electronic devices. As shown in Figure 1, the projection device can be a mobile phone and the receiving device can be an interactive tablet. It should be noted that the above examples are only used to illustrate the application scenarios that screen projection may be applicable to, and should not be understood as a limitation on the application scenarios of screen projection.

[0033] The inventors of this application have discovered that when projecting the screen of an iPhone device to an interactive receiving device (such as a smart interactive tablet) on an Android system, there is a problem of color loss. For example, in Figure 2, the left picture is the screen of an iPhone phone, and the right picture is the screen of a receiving device on an Android system. It can be clearly seen that the background color of the left picture is different from the color within the border, while the background color of the right picture is integrated with the color within the border; there is a clear boundary line between the upper and lower columns of the left picture, while the boundary line between the upper and lower columns of the right picture is almost invisible.

[0034] To find out the cause of the above situation, the inventors of this application used an Android phone-Android system receiving device as a control group. When the screen of the Android phone was projected onto the Android system receiving device, it was found that the Android system receiving device was able to restore the Android phone's screen very well, that is, the border color was clearly different from the background color, and the boundary line was clearly visible. This set of experiments showed that there may be differences in the data output by iPhone phones and Android phones. Observing the data output by iPhone phones and Android phones, it was found that there were differences in the component value ranges of the color space between the two. For example, in the image data output by the iPhone phone, the component value range of the color space is [0, 255], that is, the iPhone phone outputs full-range data; in the image data output by the Android phone, the Y component value range is [16, 235], and the other components (such as Cr, Cy; U, V) value range is [16, 240], that is, the Android phone outputs limited-range data.

[0035] Since the Android system receiving device processes the data output by the screen projection mobile phone through software and hardware. Therefore, the inventor of the present application cannot determine whether the cause of the color loss problem is the hardware configuration of the Android system receiving device or the screen projection application. In order to find out the reason, the inventor of the present application added another two sets of experiments. One set of experiments is: use an iPhone mobile phone to record the screen to obtain a screen recording video, and use the playback application of the Android system device to play the screen recording video locally. The results of this set of experiments are: there is a color loss problem in the played picture. This shows that the cause of the color loss problem has nothing to do with the application. Another set of experiments is: use an iPhone mobile phone to record the screen, and use other Android system devices to play the screen recording file. The results of this set of experiments are: there is still a color loss problem in the played picture. This shows that the cause of the color loss problem may be related to the Android system.

[0036] Before an Android device can display an iPhone's screen projection, or play a video recorded by an iPhone, it must decode and render the received data. During this decoding and rendering process, existing media applications (screen projection applications or playback applications) call various interfaces provided by the Android system.

[0037] The inventors of this application studied these interfaces and found that the calling process of media applications (such as screen projection applications) is as follows: the media application will first use the SurfaceView (or glSurfaceView) component to create the area to be rendered, and then call the MediaCodec interface to complete the decoding and rendering of the data. Among them, when calling the MediaCodec interface, the information of the area to be rendered will be directly configured to the surface parameter of the MediaCodec interface. At this time, full-scale image data is input to the MediaCodec interface, and the MediaCodec interface outputs the rendered image data, which is limited-scale data.

[0038] The execution process of the MediaCodec interface is equivalent to a "black box" for users. We can only understand that the MediaCodec interface is called to complete decoding and rendering, but we don't know how the MediaCodec interface specifically performs decoding and rendering, let alone at which step the change in the component value range of the color space related to the color loss problem occurs. It is also difficult to understand or change the execution process of the MediaCodec interface through the native code of the Android system.

[0039] The inventors of this application tried to debug the parameter configuration information of the MediaCodec interface to affect the output results. When the surface parameter during decoding of the MediaCodec interface is changed to null (for example: MediaCodec#configure(mediaFormat, null, null, codecFlag)), the full-scale image data is input to the MediaCodec interface, and the MediaCodec interface will output the decoded image data, and the decoded image data is full-scale data. In other words, the decoding operation of the MediaCodec interface will not change the component value range of the color space. After rendering the decoded image data using the rendering interface provided by the Android system, it is found that the rendered data is also full-scale data. In this way, the color loss problem is solved. Therefore, the inventors of this application believe that media applications can call different interfaces to decode full-scale image data and render the decoded full-scale image data (that is, separate the decoding and rendering processes) in order to control the decoding process and the rendering process so that neither involves changes in the component value range of the color space. This avoids the problem of only calling the MediaCodec interface to decode and render full-color image data, which causes color loss, as in related technologies.

[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0041] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0042] The media data processing method provided in the embodiments of the present application can be applied to a display device in an Android system. The display device may include a mobile phone, a wearable device, a tablet computer, a personal digital assistant (PDA), a laptop computer, a mobile computer, an interactive tablet, or other terminal device or portable terminal device with a display screen. In one example, the interactive tablet may be large in size and have a touch function, and can be used in conference or educational scenarios.

[0043] The display device can run an Android system, and applications can be installed and run on the Android system. Figure 3 is a system architecture diagram provided in an embodiment of the present application. The embodiment of the present application will be discussed based on this system architecture. In order to facilitate the explanation of the logic, only the business logic relationship is illustrated with a schematic architecture diagram, without strictly expressing the specific location of each business in the system architecture. In addition, the naming of each module in the system architecture diagram is an illustrative example. The embodiment of the present application does not limit the naming of each module in the system architecture diagram. In actual implementation, the specific naming of the module can be determined according to actual needs.

[0044] The Android system uses a layered architecture, with each layer having distinct roles and responsibilities, and each layer communicating through interfaces. In some embodiments, referring to Figure 3 , the Android system may include three layers: the application layer, the framework layer (which may include the API Framework, Native Libraries, and the Android Runtime), and the driver layer (which may include the Hardware Abstraction Layer and the Kernel).

[0045] The driver layer primarily includes local services and libraries. It interacts with the display device's hardware. For example, the display device can use the display driver in the driver layer to display images; the display device can use the audio driver in the driver layer to play audio from the speakers; and the display device can use the data driver to receive or send data. Data drivers include Bluetooth drivers and Wi-Fi drivers.

[0046] The framework layer can provide application programming interfaces and programming frameworks for the application layer's applications. For example, the framework layer can provide the application layer with modules or interfaces for media data processing (such as the first module, the second module, the MediaCodec interface, the conversion interface, the rendering interface, etc.).

[0047] The application layer may include a series of application packages, such as media applications, conference applications, browser applications, system settings, etc. Among them, media applications may include screen projection applications, image processing applications and other related applications.

[0048] Based on the architecture shown in FIG3 above, an embodiment of the present application provides a method for processing media data. Referring to FIG4 , this method can be used for a target display device of an Android system, and the target display device is configured with a screen projection application, and at least part of the method can be implemented based on the screen projection application. The method may include the following steps:

[0049] S101. Receive media data sent by a screen projection device, where the media data includes image data.

[0050] Specifically, in the screen projection scenario, the target display device can use hardware screen projection devices or screen projection software to realize pre-screen projection preparations such as screen projection device discovery, screen projection device authentication, and screen projection device connection, so as to establish a screen projection channel between the target display device and the screen projection device. The screen projection device can send media data to the screen projection application configured on the target display device through the screen projection channel, and the screen projection application obtains the media data. Among them, when the screen projection is realized between the screen projection device and the target display device through the screen projection software, the screen projection device and the target display device can be paired by scanning the code, entering the screen projection code, Airplay, etc.

[0051] Media data is compressed data that requires decoding by a media application. Media data may include at least one of image data and audio data. The data processing method provided in the embodiments of the present application is primarily targeted at image data. Therefore, after acquiring media data, it is necessary to first determine whether image data exists in the media data. When the media data is a code stream, the presence of image data can be determined by determining whether image parameters in the media data are configured.

[0052] For example, the media data may be encoded using H.264. The Sequence Parameter Set (SPS) in H.264 is a parameter set used to describe the characteristics and configuration information of a video sequence. The SPS parameters of the media data may be parsed to determine whether the value of the vui_parameters_present_flag in the SPS parameters is 1. If the value of the vui_parameters_present_flag is 1, it may be determined that image data is present in the media data.

[0053] It should be noted that the above example uses H.264 encoding to illustrate how to determine whether media data contains image data. This should not be construed as limiting the media data's codestream type and encoding method. Media data can also have other codestream types and encoding methods, and different encoding methods involve different parameters, which will not be discussed here.

[0054] When image data exists in media data, media applications can decode, render, and display the media data on the display screen through the modules or interfaces provided by the framework layer. Among them, image data can be divided into full-scale image data and limited-scale image data according to the value range of the color space. The component value range of the color space of full-scale image data is [0, 255], the value range of the Y component of the color space of limited-scale image data is [16, 235], and the value range of other components (such as Cr, Cy; U, V) is [16, 240]. The decoding and rendering methods of full-scale image data and limited-scale image data can be the same or different. The following first describes the decoding and rendering method of full-scale image data.

[0055] S102: Decode, render and display the image data on a target display device.

[0056] When the image data conforms to full-color gamut data, the image data is decoded and rendered in the first manner so that the image data displayed on the target display device still conforms to full-color gamut data. That is, the color space components of the image data received in step S101 all have a value range of [0, 255], and the color space components of the image data output in step S102 also all have a value range of [0, 255].

[0057] The first method is to process image data by separating decoding and rendering. In other words, in the first method, the screen projection application will call different modules or interfaces of the Android system to handle decoding and rendering respectively.

[0058] Specifically, the Android system may have a first module and a second module. When the image data conforms to full-color data, the screen projection application may call the first module to decode the image data, and the screen projection application may also call the second module to render the decoded image data. In this way, the screen projection application calls the first module to decode the full-color image data and calls the second module to render the full-color image data, so that each step of processing the full-color image data is controlled to be performed in a specific manner, and during this period, the changes in the component value range of the color space are not involved, thereby avoiding color loss.

[0059] Android systems can have two decoding interfaces. One interface is the MediaCodec interface, which can perform hard decoding (or soft decoding) of encoded data; the other interface can be the FFmpeg (x264) interface, which can perform soft decoding of encoded data. Therefore, there are two ways to decode and render full-scale image data:

[0060] In one possible implementation of decoding and rendering full-color image data, the first module may include a MediaCodec interface. The screen projection application may call the MediaCodec interface to decode the image data and call the second module to render the decoded image data. When the screen projection application calls the MediaCodec interface, the surface parameter of the MediaCodec interface is left unused. For example, the surface parameter of the MediaCodec interface may be configured to null.

[0061] It should be noted that in the existing technology, screen casting applications only use the SurfaceView (or glSurfaceView) component to create a rendering area, and when calling the MediaCodec interface for decoding and rendering, the surface parameter of the MediaCodec interface is configured as the rendering area. In other words, in the existing technology, the SurfaceView (or glSurfaceView) component is only used to create the rendering area, and the MediaCodec interface is responsible for decoding and rendering.

[0062] In the embodiment of the application, the surface parameter of MediaCodec is left idle, indicating that no rendering area information is provided, so as to avoid direct rendering after MediaCodec decoding, which causes color loss. In summary, in the image data processing method provided by the embodiment of the present application, the MediaCodec interface only decodes the image data, and the second module performs rendering. In this way, each step of processing the image data is controlled to be performed in a specific manner, and during this period, the change of the component value range of the color space is not involved, thereby avoiding color loss. In addition, the hard decoding method using the MediaCodec interface has the advantages of fast running speed and wide adaptation models. In some application scenarios (such as screen projection scenarios), the MediaCodec interface can be used for hard decoding.

[0063] Media data includes image data and audio data. The MediaCodec interface can decode not only the image data in the media data, but also the audio data in the media data. Therefore, the MediaCodec interface can be used to decode the media data. If the media data in step S101 includes both image data and audio data, then after the media data is decoded using the MediaCodec interface in step S102, the decoded media data will include both the decoded image data and the decoded audio data. The getOutputImage(index) function of the MediaCodec interface can be used to obtain the decoded image data from the decoded media data.

[0064] The decoded image data is in YUV format. Optionally, when calling the MediaCodec interface, you can configure the ColorFormat parameter of the MediaCodec interface to the YUV420 color format so that the decoded image data is in YUV420 color format, which improves rendering speed and quality.

[0065] Optionally, the second module may include a conversion interface and a rendering interface. The screen projection application may call the conversion interface to convert the decoded image data into RGB format data, and the screen projection application may also call the rendering interface to render the RGB format data onto the canvas provided by the target display device.

[0066] The conversion interface may be a YUVImage conversion interface provided by the Android system; the conversion interface may be a RenderScript interface, an OpenGL interface, etc. provided by a third party. In addition, if the media data in step S101 includes both audio data and image data, then in step S102, it may be rendered to the canvas (surface) provided by the target display device according to the audio and video synchronization requirements. Specifically, the waiting time before drawing on the Surface can be controlled by comparing the audio pts (Presentation Time Stamp) and the video pts, and the Canvas can be obtained through the Surface's lockCanvas, and the image can be drawn to the Surface through the Canvas's drawBitmap interface.

[0067] In another possible implementation of decoding and rendering full-color image data, the first module may include an FFmpeg (x264) interface, and the second module may include a penGL interface (or a Vulkan interface). The screen projection application may call the FFmpeg (x264) interface to decode the image data, and then call the penGL interface (or Vulkan interface) to render the decoded image data to the canvas provided by the target display device. In this way, the FFmpeg (x264) interface decodes the image data, and the penGL interface performs rendering. In this way, each step of processing the image data is controlled to be performed in a specific manner, and during this period, the changes in the component value range of the color space are not involved, so as to avoid color loss.

[0068] It should be noted that the FFmpeg (x264) interface can decode not only image data within media data, but also audio data. Therefore, the FFmpeg (x264) interface can be used to decode media data. If the media data in step S101 includes both image data and audio data, then after decoding the media data using the FFmpeg (x264) interface in step S102, the decoded media data will include both decoded image data and decoded audio data. The decoded image data can be retrieved from the decoded media data using the "get" function.

[0069] In step S102, when the image data conforms to limited color scale data, the image data is decoded and rendered in a second manner so that the image data displayed on the target display device still conforms to limited color scale data. That is, the Y component of the color space of the image data received in step S101 has a value range of [16, 235], and the other components (such as Cr, Cy; U, V) have a value range of [16, 240]. The Y component of the color space of the image data output in step S102 has a value range of [16, 235], and the other components (such as Cr, Cy; U, V) have a value range of [16, 240].

[0070] Specifically, limited-color-scale image data can adopt a decoding and rendering separation method similar to full-color-scale image data (i.e., the screen projection application calls the first module to decode the image data, and the screen projection application calls the second module to render the decoded image data.) Alternatively, limited-color-scale image data can adopt a direct decoding and rendering method. When limited-color-scale image data adopts a direct decoding and rendering method, the image data processing method provided in the embodiment of the present application may include, before step S102, determining whether the image data is full-color-scale data or limited-color-scale data.

[0071] Specifically, whether the image data conforms to full-color scale data or limited-color scale data can be determined by determining whether full-color scale parameters are configured in the image data. When the full-color scale parameters are configured in the image data, the image data can be determined to conform to full-color scale data; when the full-color scale parameters are not configured in the image data, the image data can be determined to conform to limited-color scale data.

[0072] For example, the image data is encoded using H.264. When the value of vui_parameters_present_flag is 1, that is, when the image data is present, it can be determined whether video_full_range_flag in vui_parameters is 1. If video_full_range_flag is 1, it can be determined that the image data conforms to full-color gamut data; if video_full_range_flag is not 1, it can be determined that the image data conforms to limited-color gamut data.

[0073] It should be noted that the above example uses H.264 encoding to illustrate how to determine whether image data is full-color data. This should not be construed as limiting the bitstream type and encoding method of image data. Image data can also have other bitstream types and encoding methods. Different encoding methods involve different parameters, which will not be discussed here.

[0074] In addition, the specific steps of the direct decoding and rendering method may be: when the image data conforms to the limited color level data, the MediaCodec interface may be called to decode the image data, and the decoded image data may be rendered to the canvas provided by the target display device. In this way, the decoding and rendering of the limited color level image data can be realized by calling the MediaCodec interface. Among them, when the projection application calls the MediaCodec interface, the surface parameters of the MediaCodec interface may be configured based on the rendering area information. That is to say, the surface parameters of the MediaCodec interface are configured as rendering area information. Among them, the rendering area information can be generated by the projection application through components such as SurfaceView and glSurfaceView.

[0075] It should be noted that in the prior art, after acquiring image data, components such as SurfaceView and glSurfaceView can directly generate rendering area information. Compared to the prior art, the embodiments of the present application need to first determine whether the image data is full-scale data or limited-scale data after acquiring image data, and then enable components such as SurfaceView and glSurfaceView to generate rendering area information. In order to trigger components such as SurfaceView and glSurfaceView to generate rendering area information, a notification instruction can be sent to these components first. This notification instruction can be used to enable components such as SurfaceView and glSurfaceView to generate rendering area information.

[0076] It's worth noting that regardless of the decoding and rendering method used for full-color or limited-color image data, after rendering the decoded image data to the canvas provided by the target display device, SurfaceFlinger can perform on-demand blending of the layers on the canvas and output them to the FrameBuffer. The target display device then presents the data in the FrameBuffer on the display screen. This process of displaying from the canvas to the display screen does not involve any changes in the color space value range.

[0077] 5 , an embodiment of the present application further provides an image data processing method for a target display device of an Android system. The target display device may be configured with a media application. The method is implemented at least in part based on the media application, and the method may include:

[0078] S201: Receive media data.

[0079] In the screen projection scenario, the media application can be a screen projection application. The target display device can use the hardware screen projection device or screen projection software to realize the pre-screen projection preparation work such as the discovery of the screen projection device, the authentication of the screen projection device, and the connection of the screen projection device, so as to establish a screen projection channel between the target display device and the screen projection device. The screen projection device can send the image data to the media application configured on the target display device through the screen projection channel, and the media application obtains the image data. Among them, when the screen projection is realized between the screen projection device and the target display device through the screen projection software, the screen projection device and the target display device can be paired by scanning the code, entering the screen projection code, Airplay, etc.

[0080] In the image processing scenario, media applications are image processing applications that can access media data stored in the memory of the target display device. Image processing applications can include video processing applications and image processing applications. Video processing applications can play and edit videos, while image processing applications can display and edit images.

[0081] Regardless of the scenario, media data is compressed data that requires decoding by a media application. Media data may include at least one of image data and audio data. The data processing method provided in the embodiment of the present application is mainly for image data. Therefore, after obtaining the media data, it is necessary to first determine whether image data exists in the media data. When the media data is a code stream, the presence of image data can be determined by determining whether the image parameters in the media data are configured. The determination method can be referred to above and will not be repeated here.

[0082] S202 : When the image data conforms to full-color gradation data, the image data is processed by separating decoding and rendering, and the rendered image data is displayed on a target display device, so that the image data displayed on the target display device still conforms to full-color gradation data.

[0083] Specifically, the Android system may have a first module and a second module. When the image data conforms to full-color scale data, the media application may call the first module to decode the image data, and the media application may also call the second module to render the decoded image data. Exemplarily, the first module may include a MediaCodec interface. The media application calling the first module to decode the image data may include: the media application calling the MediaCodec interface to decode the image data. When the media application calls the MediaCodec interface, the surface parameter of the MediaCodec interface is idle. Another exemplary embodiment, the first module may include an FFmpeg (x264) interface, and the FFmpeg (x264) interface may be used to decode the media data.

[0084] Optionally, the second module includes a conversion interface and a rendering interface. The media application may call the second module to render the decoded image data, which may include: the media application calling the conversion interface to convert the decoded image data into RGB format data; and the media application calling the rendering interface to render the RGB format data onto a canvas provided by a target display device.

[0085] Exemplarily, when using the MediaCodec interface for decoding, the conversion interface may be the YUVImage conversion interface provided by the Android system; the conversion interface may be a RenderScript interface, an OpenGL interface, etc. provided by a third party. In addition, if the media data in step S101 includes both audio data and image data, then in step S102, it can be rendered to the canvas (surface) provided by the target display device according to the audio and video synchronization requirements. Specifically, the waiting time before drawing on the Surface can be controlled by comparing the audio pts (Presentation Time Stamp) and the video pts, and the Canvas can be obtained through the lockCanvas of the Surface, and the image can be drawn to the Surface through the drawBitmap interface of the Canvas. Another exemplary example, when using the FFmpeg (x264) interface for decoding, the second module may be a penGL interface or a Vulkan interface.

[0086] Limited color scale image data may adopt a decoding and rendering separation method like full color scale image data (i.e., the media application calls the first module to decode the image data, and the media application calls the second module to render the decoded image data.) Alternatively, limited color scale image data may adopt a direct decoding and rendering method. When the limited color scale image data adopts the direct decoding and rendering method, the method may further include S203, determining whether the image data is full color scale data or limited color scale data. S204, when the image data conforms to limited color scale data, processing the image data by direct decoding and rendering, and displaying the rendered image data on the target display device, so that the image data displayed on the target display device still conforms to limited color scale data.

[0087] Specifically, whether the image data conforms to full-color scale data or limited-color scale data can be determined by determining whether full-color scale parameters are configured in the image data. When the full-color scale parameters are configured in the image data, the image data can be determined to conform to full-color scale data; when the full-color scale parameters are not configured in the image data, the image data can be determined to conform to limited-color scale data.

[0088] For example, the image data is encoded using H.264. When the value of vui_parameters_present_flag is 1, that is, when the image data is present, it can be determined whether video_full_range_flag in vui_parameters is 1. If video_full_range_flag is 1, it can be determined that the image data conforms to full-color gamut data; if video_full_range_flag is not 1, it can be determined that the image data conforms to limited-color gamut data.

[0089] It should be noted that the above examples use H.264 encoding to illustrate how to determine whether image data exists and whether the image data is full-color data. This should not be construed as limiting the bitstream type and encoding method of image data. Image data can also have other bitstream types and encoding methods. Different encoding methods involve different parameters, which will not be discussed here.

[0090] Step S204 may include: when the image data conforms to limited color level data, the media application may generate rendering area information; the media application may also call the MediaCodec interface to decode the image data, and render the decoded image data to the canvas provided by the target display device. In this way, calling the MediaCodec interface can realize the decoding and rendering of limited color level image data. Among them, when the media application calls the MediaCodec interface, the surface parameters of the MediaCodec interface can be configured based on the rendering area information. That is, the surface parameters of the MediaCodec interface are configured as rendering area information. Among them, the rendering area information can be generated by the media application through components such as SurfaceView and glSurfaceView.

[0091] It should be noted that in the prior art, after acquiring image data, components such as SurfaceView and glSurfaceView can directly generate rendering area information. However, compared to the prior art, the embodiments of the present application need to first determine whether the image data is full-scale data or limited-scale data after acquiring image data, and then enable components such as SurfaceView and glSurfaceView to generate rendering area information. In order to trigger components such as SurfaceView and glSurfaceView to generate rendering area information, a notification instruction can be sent to these components first. This notification instruction can be used to enable components such as SurfaceView and glSurfaceView to generate rendering area information.

[0092] With reference to Figure 6, an electronic device 300 is provided in an embodiment of the present application. The components shown herein, their connections and relationships, and their functions are merely examples and do not limit the implementation of the present application described and / or required herein. As shown in Figure 6, the electronic device 300 may include: a processor 310 and a memory 320, each component being interconnected using different buses and being mounted on a common mainboard or in other ways as needed. The processor 310 may process instructions executed within the electronic device, including instructions for graphic information stored in or on the memory for display on an external input / output device (such as a receiving device coupled to an interface). In other embodiments, if necessary, multiple processors and / or multiple buses may be used together with multiple memories and multiple memories. Figure 5 takes a processor 310 as an example.

[0093] The memory 320, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the electronic device method in the embodiments of the present application. The processor 310 executes the non-transitory software programs, instructions, and modules stored in the memory 320 to execute various functional applications and electronic processing methods, that is, to implement the electronic device method in the above-mentioned method embodiments.

[0094] The electronic device 300 may further include an input device and an output device. The processor 310, the memory 320, the input device, and the output device may be connected via a bus or other means. FIG5 shows an example of a bus connection.

[0095] The input device can receive input digital or character information and generate key signal input related to user settings and function control of the electronic device, such as an input device such as a touch screen, a keypad, a mouse, or multiple mouse buttons, a trackball, a joystick, etc. The output device can be an output device such as a receiving device of the electronic device. The receiving device can include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the receiving device can be a touch screen.

[0096] The electronic device of the embodiment of the present application can be used to execute the technical solutions in the above-mentioned method embodiments of the present application. Its implementation principles and technical effects are similar and will not be repeated here.

[0097] The present application also provides a computer storage medium that can store multiple instructions suitable for being loaded by a processor and executing the above-mentioned media data processing method, which will not be described in detail here. The device where the storage medium is located can be a smart interactive tablet.

[0098] An embodiment of the present application further provides a computer program product, including a computer program, which is used to implement the above-mentioned media data processing method when executed by a processor.

[0099] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0100] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0101] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0102] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0103] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0104] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0105] Computer-readable media include permanent and non-permanent, removable and non-removable media that can store information using any method or technology. The information can be computer-readable instructions, data structures, program modules or other data.

[0106] Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0107] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0108] The above are merely embodiments of the present application and are not intended to limit the present application. It will be apparent to those skilled in the art that various modifications and variations may be made to the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included within the scope of the claims of the present application.

Claims

1. A media data processing method for a target display device of an Android system, wherein the target display device is configured with a screen projection application, characterized in that: At least part of the method is implemented based on the screen projection application, and the method includes: Receive media data sent by the screen projection device, wherein the media data includes image data; Decoding, rendering and displaying the image data on the target display device; When the image data conforms to full-color gradation data, the image data is decoded and rendered in a first manner so that the image data displayed on the target display device still conforms to full-color gradation data; the first manner is to process the image data by separating decoding and rendering; When the image data conforms to the limited color scale data, the image data is decoded and rendered in a second manner, so that the image data displayed on the target display device still conforms to the limited color scale data.

2. The method according to claim 1, characterized in that The Android system comprises a first module and a second module; When the image data conforms to full-color scale data, decoding and rendering the image data in a first manner includes: When the image data conforms to full-color scale data, the screen projection application calls the first module to decode the image data, and the screen projection application calls the second module to render the decoded image data.

3. The method according to claim 2, characterized in that The first module includes a MediaCodec interface; The screen projection application calls the first module to decode the image data, including: the screen projection application calls the MediaCodec interface to decode the image data; and when the screen projection application calls the MediaCodec interface, the surface parameter of the MediaCodec interface is idle.

4. The method according to claim 2, characterized in that The second module includes a conversion interface and a rendering interface; The screen projection application calling the second module to render the decoded image data includes: The screen projection application calls the conversion interface to convert the decoded image data into RGB format data; the screen projection application calls the rendering interface to render the RGB format data onto the canvas provided by the target display device.

5. The method according to any one of claims 1 to 4, characterized in that Also includes: It is determined whether the image data conforms to full-scale data or limited-scale data.

6. The method according to claim 5, characterized in that The determining whether the image data is full-scale data or limited-scale data includes: Determining whether full-scale parameters in the image data are configured; When the full-color-scale parameters in the image data are configured, it is determined that the image data conforms to the full-color-scale data; when the full-color-scale parameters in the image data are idle, it is determined that the image data conforms to the limited-color-scale data.

7. The method according to claim 5 or 6, characterized in that When the image data conforms to limited color scale data, decoding and rendering the image data in a second manner includes: When the image data conforms to limited color scale data, the screen projection application generates rendering area information; the screen projection application also calls the MediaCodec interface to decode the image data, and renders the decoded image data onto the canvas provided by the target display device; Among them, when the screen projection application calls the MediaCodec interface, the surface parameters of the MediaCodec interface are configured based on the rendering area information.

8. A media data processing method for a target display device of an Android system, wherein the target display device is configured with a media application, characterized in that: The method is implemented at least in part based on the media application, and the method includes: receiving media data, wherein the media data includes image data; When the image data conforms to full-color scale data, the image data is processed by separating decoding and rendering, and the rendered image data is displayed on the target display device, so that the image data displayed on the target display device still conforms to full-color scale data.

9. The method according to claim 8, characterized in that The Android system comprises a first module and a second module; When the image data conforms to full-color scale data, processing the media image data by separating decoding and rendering includes: When the image data conforms to full-color scale data, the media application calls the first module to decode the image data, and the media application calls the second module to render the decoded image data.

10. The method according to claim 9, characterized in that The first module includes a MediaCodec interface; The media application calling the first module to decode the image data includes: The media application calls the MediaCodec interface to decode the image data; When the media application calls the MediaCodec interface, the surface parameter of the MediaCodec interface is idle.

11. The method according to claim 9 or 10, characterized in that The second module includes a conversion interface and a rendering interface; The media application calling the second module to render the decoded image data includes: The media application calls the conversion interface to convert the decoded image data into RGB format data; the media application calls the rendering interface to render the RGB format data onto a canvas provided by the target display device.

12. The method according to any one of claims 8 to 11, characterized in that The media application includes at least one of a screen projection application and an image processing application.

13. A display device of an Android system, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the media data processing method according to any one of claims 1-7 or 8-12.

14. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the media data processing method according to any one of claims 1 to 7 or 8 to 12 is implemented.

Images