Image data processing method and apparatus

Abstract

The application provides an image data processing method and device. After a terminal device as a data sending end obtains a first data stream containing graphic data of multiple frames of graphs, if a corresponding data receiving end needs to display corresponding images, the terminal device can directly process the graphic data of each frame of image based on a target data coding mode to obtain a second data stream containing coding data of multiple frames of images.

Description

Technical Field

[0001] This application relates primarily to the field of image processing technology, and more specifically to an image data processing method and apparatus. Background Technology

[0002] Currently, in end-to-end application scenarios such as wireless screen projection and video conferencing, the sending device typically encodes the image data of each frame of the rendered image before transmitting it to another terminal device for decoding. The terminal device will then directly display and output the corresponding frame image, meeting the image display needs of multiple devices.

[0003] However, the image encoding and decoding process for each frame of the rendered image is time-consuming, resulting in a large end-to-end image display delay, especially when the device performance is low, which greatly reduces the image display efficiency of the receiving device. Summary of the Invention

[0004] In view of this, this application proposes an image data processing method, the method comprising:

[0005] A first data stream is obtained, which includes graphic data of multiple frames of images; the graphic data of each frame of images is processed by an image renderer to obtain a frame of image for display output;

[0006] Based on the target data encoding method, the graphic data of each frame of the image is processed to obtain a second data stream, which includes the encoded data of multiple frames of images.

[0007] The second data stream is transmitted, and the second data stream is used at least for the terminal device receiving the second data stream to decode and render.

[0008] Optionally, the amount of graphic data in the same frame of an image is greater than or equal to the amount of encoded data.

[0009] Optionally, the graphics data of each frame includes instruction data and image texture data, and the method further includes:

[0010] Remove image texture data that meets preset conditions from at least one frame of the first data stream;

[0011] The second data stream is obtained by processing the graphic data of each frame of the image based on the target data encoding method, including:

[0012] Based on the target data encoding method, the graphic data of each frame of the image after the image texture data removal process is processed to obtain the second data stream.

[0013] Optionally, the step of removing image texture data that meets preset conditions from at least one frame of the first data stream includes:

[0014] The image texture data of each frame in the first data stream is deduplicated; and / or,

[0015] Remove the preset texture data contained in each frame of the first data stream; the preset texture data is determined and stored by the terminal device that will receive the second data stream based on the data source of the first data stream or the second data stream that has already been received.

[0016] Optionally, the step of processing the graphic data of each frame of the image based on the target data encoding method to obtain the second data stream includes:

[0017] Based on the target data encoding method, the instruction data and image texture data from different channels in the graphic data of each frame image are encoded to obtain the second data stream of the corresponding channel;

[0018] The transmission of the second data stream includes:

[0019] The second data stream from different channels is transmitted synchronously.

[0020] Furthermore, this application also proposes an image data processing method, the method comprising:

[0021] A second data stream is obtained, which includes encoded data of multiple frames of images; the encoded data of each frame of image is obtained by the terminal device sending the second data stream by processing the graphic data of each frame of image in the obtained first data stream based on the target data encoding method.

[0022] Based on the target data decoding method, the encoded data of each frame of image is processed to obtain a third data stream; the third data stream includes the decoded data of multiple frames of images;

[0023] The decoded data of each frame of the image is processed and the corresponding frame of the image is displayed.

[0024] Optionally, the amount of encoded data for the same frame of image is less than or equal to the amount of graphic data used to display the frame of image.

[0025] Optionally, the process of processing the decoded data of each frame of image and displaying the corresponding frame image includes:

[0026] Based on the instruction data contained in the decoded data, the corresponding frame image is reconstructed to obtain a fourth data stream; the fourth data stream includes graphic data of multiple frames, and the amount of graphic data from the fourth data stream for the same frame image is greater than the decoded data.

[0027] Render the graphic data of each frame in the fourth data stream and display the corresponding frame image;

[0028] The process of reconstructing the corresponding frame image based on the instruction data contained in the decoded data to obtain the fourth data stream includes:

[0029] Based on the instruction data contained in the decoded data, image texture data that meets preset conditions is obtained; the image texture data that meets preset conditions includes image texture data of the corresponding frame image in the first data stream after deduplication, and / or image texture data pre-stored based on the data source of the first data stream or the received second data stream and removed from the first data stream.

[0030] The fourth data stream is constructed using the decoded data of the same frame image and the image texture data that meets the preset conditions.

[0031] Furthermore, this application also proposes an image data processing apparatus, the apparatus comprising:

[0032] The first data stream acquisition module is used to acquire a first data stream, which includes graphic data of multiple frames of images; the graphic data of each frame of images is processed by an image renderer to obtain a frame of image for display output;

[0033] The graphics data processing module is used to process the graphics data of each frame of the image based on the target data encoding method to obtain a second data stream, which includes the encoded data of multiple frames of the image.

[0034] The second data stream transmission module is used to transmit the second data stream, which is at least used by the terminal device receiving the second data stream to decode and render it.

[0035] Furthermore, this application also proposes an image data processing apparatus, the apparatus comprising:

[0036] The second data stream acquisition module is used to acquire a second data stream, which includes encoded data of multiple frames of images. The encoded data of each frame of image is obtained by the terminal device that sends the second data stream, based on the target data encoding method, by processing the graphic data of each frame of image in the acquired first data stream.

[0037] The encoded data processing module is used to process the encoded data of each frame of image based on the target data decoding method to obtain a third data stream; the third data stream includes the decoded data of multiple frames of images;

[0038] The decoding data processing module is used to process the decoded data of each frame of the image and display the corresponding frame image obtained from the processing.

[0039] In another aspect, this application also proposes a computer-readable storage medium storing a computer program that is loaded and executed by a processor to implement an image data processing method on the corresponding side.

[0040] Therefore, this application provides an image data processing method and apparatus. After a terminal device, acting as a data sending end, obtains a first data stream containing graphic data of multiple frames, if the corresponding data receiving end needs to display the corresponding image, the terminal device can directly process the graphic data of each frame of the image based on the target data encoding method to obtain a second data stream containing encoded data of multiple frames of the image. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of an optional system architecture suitable for the image data processing method proposed in this application environment;

[0043] Figure 2 This is a schematic diagram of another optional system architecture suitable for the application environment of the image data processing method proposed in this application;

[0044] Figure 3 A schematic diagram of the hardware structure of an optional example of a terminal device suitable for the image data processing method proposed in this application;

[0045] Figure 4 A schematic diagram of the structure of the data sending end and the data receiving end for an optional application scenario applicable to the image data processing method proposed in this application;

[0046] Figure 5 This is a schematic diagram of the structure of the data sending end and the data receiving end for another optional application scenario applicable to the image data processing method proposed in this application;

[0047] Figure 6 This is a flowchart illustrating another optional example of the image data processing method proposed in this application;

[0048] Figure 7 This is a flowchart illustrating another optional example of the image data processing method proposed in this application;

[0049] Figure 8This is a flowchart illustrating another optional example of the image data processing method proposed in this application;

[0050] Figure 9 This is a flowchart illustrating another optional example of the image data processing method proposed in this application;

[0051] Figure 10 This is a schematic diagram of an optional example of the image data processing apparatus proposed in this application;

[0052] Figure 11 A schematic diagram of another optional example of the image data processing apparatus proposed in this application;

[0053] Figure 12 A schematic diagram of another optional example of the image data processing apparatus proposed in this application;

[0054] Figure 13 This is a schematic diagram of another alternative example of the image data processing apparatus proposed in this application. Detailed Implementation

[0055] To address the technical problems described in the background section, and in order to reduce end-to-end image display latency and improve image display efficiency for low-performance terminal devices in scenarios such as wireless screen projection and video conferencing, this application proposes to utilize the image rendering capabilities of both end devices to change the data transmission format from the sending end device to the receiving end device. Compared to the sending end device rendering the graphic data of multiple frames of images, obtaining the image data of a single frame, and then compressing and encoding that image data before transmission, this application directly encodes the graphic data (such as instruction data, image texture data, etc., used by the image renderer to obtain a single frame for display output) of each frame of image and transmits the resulting data stream. This simplifies the encoding and decoding steps, reduces the time spent on data encoding by the sending end device and data decoding by the receiving end device, improves the image display efficiency of the receiving end device, and optimizes end-to-end image display latency.

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] Reference Figure 1 This is a schematic diagram of a system architecture applicable to an optional application environment of the image processing method proposed in this application. This application environment may include, but is not limited to, end-to-end application scenarios such as wireless screen sharing and video conferencing. Figure 1 As shown, the system architecture may include at least multiple terminal devices 110, wherein:

[0058] Terminal device 110 may include, but is not limited to, smartphones, tablets, wearable devices, smartwatches, augmented reality (AR) devices, virtual reality (VR) devices, in-vehicle devices, robots, smart medical devices, smart transportation devices, desktop computers, etc., and can be determined according to the actual application scenario. Figure 1 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0059] In this embodiment, the terminal device 110 is used in end-to-end application scenarios such as wireless screen projection and video conferencing. It can act as a data sender or a data receiver, depending on the situation. It is understood that, in a wireless screen projection application scenario, the terminal device 110, acting as the screen projection device, and the terminal device 110, acting as the screen being projected, can communicate and connect via wired or wireless means to achieve data interaction between the two.

[0060] In application scenarios such as video conferencing and online teaching, terminal device 110 typically needs to be configured with a corresponding communication client. This is to support data interaction between multiple terminal devices 110 in such application scenarios. Figure 2 As shown, the system architecture may also include a communication server 120 that supports communication between different terminal devices 110, such as a server that matches each communication client configured on the terminal device 110, to provide corresponding communication services for the terminal device 110 in order to realize data interaction with other terminal devices. This application does not describe in detail the communication principle between each communication client and the matching communication server.

[0061] The aforementioned server can be an independent physical server, a server cluster consisting of multiple physical servers, or a cloud server capable of cloud computing. This application does not limit the type of server or its composition structure, and it can be determined as appropriate.

[0062] In some other embodiments, different terminal devices 110 can also interact with each other using network devices such as base stations. Therefore, as Figure 2 As shown, in certain application scenarios, the system architecture may also include at least one network device 130, and multiple terminal devices 110 may access the network device 130 for data interaction. This application does not elaborate on the implementation principle of such end-to-end data interaction.

[0063] As can be seen, the communication methods between multiple terminal devices 110 can be flexibly adjusted in different application scenarios. Figure 1 and Figure 2 The system architecture shown does not constitute a limitation on the system of the embodiments of this application. In practical applications, the system architecture may include more advanced architectures. Figure 1 and Figure 2 Other devices, such as databases, are not listed here.

[0064] Reference Figure 3 The above is a schematic diagram of the hardware structure of an optional example of a terminal device suitable for the image data processing method proposed in this application, as shown below. Figure 3 As shown, the terminal device may include, but is not limited to, a communication module 310, a storage module 320, a processing module 330, and a display module 340, wherein:

[0065] The communication module 310 may include a communication module capable of data interaction using a wireless communication network, such as a WIFI module, a 5G / 6G (fifth-generation mobile communication network / sixth-generation mobile communication network) module, a GPRS module, a GMS module, a radio frequency communication module, an antenna, etc., to achieve communication with other terminal devices or servers. Of course, as described above in the system architecture description, different terminal devices can also communicate with each other through wired communication. Therefore, the above-mentioned communication module 310 may also include communication ports that support communication with other terminal devices, such as USB interfaces, audio and video connection ports, expansion ports, etc. This application does not limit the type of communication port and can be determined as appropriate.

[0066] In this embodiment, the communication module 310 may also include a communication interface that supports data interaction between various components within the terminal device, such as a USB interface, serial / parallel port, I / O interface, etc. This application does not limit the type of communication module 310 in the terminal device or its communication operation mode.

[0067] The storage module 320 can be used to store a program that implements the image data processing method proposed in the embodiments of this application; the processing module 330 can load and execute the program stored in the memory to implement the various steps of the image data processing method executed by the terminal device as a data receiver or data sender. The specific implementation process can be referred to the description of the corresponding part of the corresponding embodiment below.

[0068] The storage device 320 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device. The processing module 330 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices.

[0069] Display module 340, i.e., a monitor, can be used to display each frame of the rendered image. This application does not elaborate on the image display principle. It should be understood that... Figure 3 The structure of the terminal device shown does not constitute a limitation on the terminal device in the embodiments of this application. In practical applications, the terminal device may include more than Figure 3 The application may include more components, or combinations of certain components, such as at least one input module such as a keyboard, mouse, camera, microphone, etc., at least one output module such as a speaker, vibration mechanism, lamp, etc., and various sensor modules, etc., which are not listed here.

[0070] In any of the application scenarios described above, the same terminal device can act as both a data sender and a data receiver. To implement the image data processing method executed by the terminal device as a data receiver, such as... Figure 4 As shown, the processing module 330 in the terminal device may include an image generator, and an image renderer and an image encoder respectively connected to the image generator, wherein:

[0071] The image generator can obtain a first data stream, which includes graphic data of multiple frames of images. The graphic data of each frame typically includes instruction data and image texture data, used to describe the image content of that frame, so that subsequent image rendering can be performed accordingly. In practical applications, the first data stream can be a binary stream, which can come from a communication server supporting the current application scenario, or from image files stored in the terminal device itself, or from data collected by an image acquisition device. This application does not limit the generation method or content of the graphic data.

[0072] The image renderer can process the graphic data of each frame of image to obtain a frame of image for display output, and send the frame of image to the display module 340 for display. This application does not describe in detail how to use the instruction data and image texture data of each frame of image for image rendering; this can be determined based on the working principle of the appropriate image rendering method.

[0073] Optionally, the aforementioned image renderer can be a functional module within a processing module such as a CPU or GPU (Graphics Processing Unit) that performs image rendering tasks. In this way, after obtaining the graphic data for each frame of an image, the unified interface of this functional module (such as Open Graphics Library, OpenGL, a cross-language, cross-platform application programming interface API for rendering 2D and 3D vector graphics) can be called to render the graphic data and obtain the corresponding frame image. In some embodiments, the image renderer can also be a standalone hardware module that connects to the processor in the processing module that executes the image data processing method. Through this connection, the complete graphic data for each frame of an image is sent to the image renderer for processing. This application does not limit the type of image renderer; it can be determined as appropriate.

[0074] The image encoder can receive graphic data of multiple frames of images sent by the image generator. Based on the target data encoding method (such as various compression algorithms), it processes the graphic data of each frame of image to obtain a second data stream. This second data stream may include the encoded data of multiple frames of images. This application does not describe the data encoding process of different data encoding methods in detail. The image encoder can send the obtained second data stream to the communication module 310 for transmission, so that the communication module 310 currently connected to the terminal device can be connected. The communication module of another terminal device, acting as the data receiving end, receives the second data stream, decodes it, renders it, and displays the corresponding frame image.

[0075] Based on this, such as Figure 4 As shown, the terminal device may also include an image decoder. When it acts as a data receiver, after receiving the second data stream transmitted by the data sender through the communication module, it sends the data stream to the image decoder for decoding processing to obtain a third data stream containing decoded data of multiple frames of images. That is, it restores the instruction data and image texture data of each frame of image obtained by the data sender's encoding processing, and then sends it to the image renderer for rendering processing. The obtained frame of image is then sent to the display module for display output.

[0076] Optionally, if the data sending end performs deduplication and removes preset texture data (such as image texture data already recorded by the corresponding data receiving end) on the image texture data in the complete image data of each frame before encoding the graphic data of each frame, in order to reduce the workload of data encoding and the amount of data transmitted, the image texture data of each frame decoded and restored by the image decoder of the data receiving end will not be complete. In this case, such as Figure 5As shown, the processing module of the data receiving end can also be configured with an image reconstructor. The image decoder can send the instruction data and image texture data of each frame of the decoded image to the image reconstructor for image reconstruction, obtain the complete graphic data of each frame of the image, and then transmit it to the image renderer for rendering processing to ensure the quality of the output image displayed by the data receiving end. The implementation process can be referred to the description of the corresponding part of the method embodiment below. The embodiments of this application will not be described in detail here.

[0077] It should be noted that in practical application scenarios, the processing devices used by the terminal device as a data sender or data receiver to perform image data processing methods can be different, including but not limited to... Figure 4 and Figure 5 The terminal device structure shown can be flexibly adjusted as needed, and this application will not provide detailed examples of each component. Furthermore, regarding the transmission of each data stream within the terminal device, such as... Figure 4 and Figure 5 As shown, instruction data and image texture data can be transmitted through different channels respectively; of course, instruction data and image texture data for each frame of image can also be transmitted through a single channel. This application does not limit this and can be determined as appropriate.

[0078] Based on the system architecture of the image data processing method in the application scenario described in the above embodiments, as well as the relevant description of the composition structure of the terminal device as the data receiver and data sender, the image data processing method executed when the terminal device acts as the data sender and data receiver will be described below, but it is not limited to the execution steps described in the method embodiments below.

[0079] Reference Figure 6 This is a flowchart illustrating an optional example of the image data processing method proposed in this application. This method can be executed by a terminal device. This embodiment describes the implementation process of the image processing method when the terminal device acts as a data sender. Figure 6 As shown, the method may include:

[0080] Step S61: Obtain a first data stream; the first data stream includes graphic data of multiple frames of images, and the graphic data of each frame of images is processed by an image renderer to obtain a frame of image for display output;

[0081] Based on the above description of the technical solution of this application, in end-to-end application scenarios such as video conferencing and wireless screen projection, the terminal device, as the data sending end, obtains a first data stream of graphic data, such as a binary stream, including multiple frames of images. If the terminal device needs to display multiple frames of images, and the terminal device communicating with the terminal device, as the data receiving end, also needs to be able to display the multiple frames of images, then, for the first data stream obtained by the terminal device as the data sending end, on the one hand, it can be processed by an image renderer to obtain a frame of image for display output, which is then displayed by the terminal device. This application does not describe in detail the process of the image renderer processing the graphic data.

[0082] It should be understood that the way a terminal device obtains the first data stream can vary depending on the application scenario. In scenarios such as video conferencing, the terminal device can capture images of participants and generate graphic data including the participants' images (such as instruction data and image texture data for each frame of the video conferencing window). In scenarios such as wireless screen projection, the terminal device can generate corresponding graphic data for each frame of the display interface (such as the application interface) to be displayed. This application does not limit the method of obtaining the first data stream and can be determined as appropriate.

[0083] Step S62: Based on the target data encoding method, the graphic data of each frame of the image is processed to obtain a second data stream, which includes the encoded data of multiple frames of images.

[0084] Following the above analysis, after the terminal device obtains the first data stream, if it needs its corresponding data receiver to display the corresponding image, in order to reduce the image display latency of the data receiver, the terminal device, as the data sender, can directly encode the graphic data of each frame of the image according to the target data encoding method. Compared to using graphic data to complete image rendering and then encoding the image data contained in the resulting image based on video compression algorithms such as H.264 and HEVC, this method reduces the data encoding steps, shortens the time spent on data encoding, helps reduce the image display latency of the data receiver, and is more suitable for terminal devices with poor performance.

[0085] In this application embodiment, the target data encoding method can be any lossless compression algorithm or lossy compression algorithm that supports the data encoding method, so that the amount of graphic data of the same frame image can be greater than or equal to the amount of encoded data. It can be flexibly selected according to actual application needs. This application does not limit the target data encoding method.

[0086] To further improve the efficiency of image data encoding, reduce the amount of data in the second data stream, and thus further reduce latency, the terminal device, acting as the data sender, can select necessary image data (such as complete instruction data and necessary partial image texture data) for each frame of image encoding. This allows the terminal device to deduplicate the image texture data before encoding the image data of each frame, avoiding repeated encoding of image texture data with identical content. Alternatively, if the corresponding data receiver pre-stores the image texture data that the data sender will transmit for the current application scenario, the terminal device, acting as the data sender, does not need to receive that type of image texture data (which can be referred to as preset texture data) from the data receiver. These methods also allow the amount of image data (i.e., the complete image data in the first data stream) in the same frame of image to be greater than or equal to the amount of encoded data (i.e., the output data of the image encoder), achieving the technical effect of reducing the amount of encoded and transmitted data and lowering latency.

[0087] Step S63: Transmit the second data stream, which is used at least for the terminal device receiving the second data stream to decode and render.

[0088] As described above in the embodiments regarding the communication methods between multiple terminal devices, after a terminal device efficiently obtains a second data stream containing encoded data of multiple frames of images, it can use a corresponding communication method to transmit the second data stream to the corresponding data receiving end, i.e., the terminal device receiving the second data stream. This allows the data receiving end to decode and render the received second data stream, displaying the corresponding frame images. The implementation process can be referred to but is not limited to the above. Figure 4 and Figure 5 Functional description of the structure of the data receiving end shown.

[0089] In summary, in the embodiments of this application, after the terminal device, as the data sending end, obtains the first data stream containing graphic data of multiple frames, if the corresponding data receiving end needs to display the corresponding image, the terminal device can directly process the graphic data of each frame of the image based on the target data encoding method to obtain the second data stream containing encoded data of multiple frames of the image. Compared with the method of encoding the image data contained in the rendered image, this greatly reduces the data encoding steps, shortens the time consumed by data encoding processing, reduces the end-to-end image display latency, and is also better suited for terminal devices with poor performance.

[0090] Reference Figure 7 This is a flowchart illustrating another optional example of the image data processing method proposed in this application. This embodiment describes an optional refined implementation of the image processing method described above from the data receiving end, such as... Figure 7As shown, the method may include:

[0091] Step S71: Obtain a first data stream; the first data stream includes graphic data of multiple frames of images, and the graphic data of each frame of images is processed by an image renderer to obtain a frame of image for display output;

[0092] The implementation process of step S71 can be referred to the description of the corresponding part of the above embodiment, and will not be described in detail here.

[0093] Step S72: Remove image texture data that meets preset conditions from at least one frame of the first data stream to obtain graphic data of multiple frames of images after image texture data removal processing;

[0094] As described above in the technical solution of this application, in order to further reduce the amount of encoded data and transmitted data while ensuring the quality of the displayed image at the data receiving end, so that the amount of encoded data of the same frame image is less than the amount of graphic data in the first data stream, after obtaining the first data stream, image texture data that meets the preset conditions can be identified, i.e., unnecessary image texture data. The preset conditions can be determined or dynamically adjusted according to the actual situation, and this application does not limit its content.

[0095] Optionally, the aforementioned preset conditions include content identical to the already encoded image texture data (i.e., necessary image texture data), meaning there are multiple image texture data with duplicate content, and / or content identical to the image texture data already stored by the terminal device receiving the second data stream (which can be referred to as preset texture data for convenience). Based on this, after obtaining the first data stream, the terminal device can perform deduplication processing on the image texture data of each frame in the first data stream; and / or, remove the preset texture data contained in each frame in the first data stream; the preset texture data can be determined and stored by the terminal device receiving the second data stream (i.e., the corresponding data receiving end) based on the data source of the first data stream or the already received second data stream.

[0096] As can be seen, after the terminal device obtains the first data stream, it can identify the image texture data with repeated content, and then perform deduplication processing on the image texture data of each frame of the image. Specifically, it can perform deduplication processing on the image texture data contained in each frame of the image itself, or it can perform deduplication processing on the image texture data of multiple consecutive frames of the image, so as to greatly reduce the amount of image texture data in the data stream to be encoded. The specific implementation method of data deduplication processing will not be described in detail in this embodiment of the application.

[0097] For the preset texture data stored by the aforementioned data receiving end, the data sending end can negotiate / communicate with one or more corresponding data receiving ends to determine the preset texture data stored by the data receiving end, depending on the current application type (i.e., the data source type of the first data stream). For example, in a game scenario, the data receiving end can obtain and store the texture data of the game scene of that type of game, so that the data sending end can record the preset texture data stored by each data receiving end.

[0098] Of course, after the data receiving end decodes the second data stream transmitted by the data sending end, it can dynamically update the stored preset texture data based on the obtained image texture data. At the same time, the data sending end can also synchronously update the recorded preset texture data for the data receiving end. Based on the updated preset texture data, it can continue to identify the image texture data contained in the subsequently obtained first data stream, and remove the identified image texture data that is the same as the updated preset texture data. This avoids compressing and encoding unnecessary image texture data, minimizes the amount of encoded data and transmitted data, and reduces end-to-end latency.

[0099] In practical applications, step S74 above can be performed by... Figure 4 and Figure 5 The image generator shown is implemented in such a way that the image generator can pre-record the above-mentioned preset conditions. In this way, after obtaining the first data stream, the image texture data in the complete graphic data of each frame image can be directly filtered according to the above method, so that the image texture data sent to the image encoder no longer contains image texture data that meets the preset conditions. This application does not limit the method of the image generator to identify and remove image texture data that meets the preset conditions.

[0100] In some other embodiments, for the image texture data deduplication processing method included in the above-mentioned preset conditions, a repetition threshold for the number of repetitions of image texture data can be further configured. The preset conditions may also include the number of repetitions of image texture data with the same content as the already determined encoded image texture data (i.e., necessary image texture data) reaching the repetition threshold. In other words, whether there is a large number of repetitive image texture data in the first data stream. Based on this, after identifying image texture data with repetitive content, the terminal device can further count whether the number of repetitive image texture data reaches the repetition threshold. If so, it is determined to be image texture data that meets the preset conditions and deduplication processing is performed on it; otherwise, it can be determined to be image texture data that does not meet the preset conditions and deduplication processing is not required.

[0101] Based on these embodiments, optionally, when determining the preset texture data stored at the data receiving end, it is also possible to determine, for a preset application type (such as games, online speeches, etc.), the number of image texture data with repeated content in one frame of image or multiple consecutive frames of image reaches a preset threshold, and the texture data does not exist in the image texture data already stored at the data receiving end, the texture data can be determined as a preset texture data of the data receiving end device, that is, the preset texture data of the data receiving end is updated, and the update result is fed back to the corresponding data sending end.

[0102] Step S73: Based on the target data encoding method, the instruction data and image texture data from different channels in the graphic data of each frame image are encoded to obtain the second data stream of the corresponding channel.

[0103] Step S74: Synchronously transmit second data streams from different channels. The second data streams are used at least for the terminal device receiving the second data streams to decode and render.

[0104] In practical applications of this application, after the removal of image texture data that meets the preset conditions is completed according to the method described above, the corresponding target data encoding method can be retrieved according to the application requirements. Based on the target data encoding method, the graphic data of each frame of the image after the removal of image texture data is processed to obtain the second data stream.

[0105] Optionally, in the above data encoding process, for the two main categories of data—instruction data and image texture data—in the graphic data of each frame of an image, corresponding channels (i.e., communication links) can be constructed. The corresponding instruction data and image texture data are transmitted to the image encoder through different channels. In this way, the image encoder can use the same or different target data encoding methods to encode the data transmitted through each channel (instruction data for each frame of the image transmitted through the instruction channel, and at least a portion of the image texture data for each frame of the image transmitted through the texture channel), obtaining corresponding encoded data, such as instruction encoded data for each frame of the image (which can constitute a second data stream for the instruction channel) and texture encoded data (which can constitute a second data stream for the texture channel), establishing the association between the instruction encoded data and texture encoded data of the same frame of the image. Then, the resulting second data stream can be transmitted through one or more communication channels of the communication network.

[0106] In some other embodiments, the graphic data of each frame of an image can be transmitted to an image encoder through one channel. Based on the target data encoding method, the graphic data of each frame of an image transmitted through this one channel is encoded, and then the resulting second data stream is transmitted through another channel. It can be seen that in the image data processing method proposed in this application, different types of data can be transmitted through one channel or multiple corresponding channels. During data encoding processing, the data transmitted through each channel can be encoded independently, or multiple types of data from each frame of an image transmitted through multiple channels can be fused and encoded, etc. The required processing method can be flexibly selected according to the actual situation, and this application will not provide detailed examples of each method.

[0107] In summary, in the embodiments of this application, after the terminal device obtains the first data stream containing complete graphic data of multiple frames, it can remove image texture data that meets preset conditions, such as image texture data with a large amount of repetitive content, preset texture data already stored at the data receiver, etc., which greatly reduces the amount of data for subsequent compression encoding, thereby improving data encoding efficiency. At the same time, by reducing the amount of data transmitted to the second data stream obtained by encoding, the transmission network latency is reduced, so that the data receiver can quickly obtain the encoded data of the required multiple frames of images, reducing the amount of data to decode the encoded data, and further reducing its image display efficiency.

[0108] Reference Figure 8 This is a flowchart illustrating another optional example of the image data processing method proposed in this application. This embodiment can be executed by a terminal device acting as a data receiving end. Based on the above description of the image data processing method executed by the corresponding data sending end, this embodiment can describe the execution steps of the image data processing method from the data receiving end side, such as... Figure 8 As shown, the method may include:

[0109] Step S81: Obtain the second data stream; the second data stream includes encoded data of multiple frames of images;

[0110] As can be seen from the image data processing method described above from the data sending end side, the encoded data of each frame of the second data stream can be obtained by the terminal device (i.e., the data sending end) that sends the second data stream, based on the target data encoding method, by processing the graphic data of each frame of the first data stream. The graphic data of each frame of the image can be processed by the image renderer of the terminal device (i.e., the data sending end) to obtain a frame of image for display output. Regarding the implementation process of how to process the first data stream to obtain the second data stream, please refer to the description of the corresponding part of the above embodiment. This embodiment will not be described in detail here.

[0111] It is understandable that the encoded data of each frame of image can be obtained by encoding the complete graphic data of the frame of image, so that the amount of encoded data of the same frame of image is equal to the amount of graphic data used by the data receiving end to display and output the frame of image; of course, the encoded data of each frame of image can also be obtained by removing the image texture data that meets the preset conditions contained in the graphic data before encoding and processing, so that the amount of encoded data of the same frame of image is less than the amount of graphic data used by the data receiving end to display and output the frame of image, thereby reducing end-to-end latency. For the processing of the first data stream, please refer to the description in the corresponding part above.

[0112] Step S82: Based on the target data decoding method, the encoded data of each frame of the image is processed to obtain a third data stream; the third data stream includes the decoded data of multiple frames of images;

[0113] It should be understood that the target data decoding method matches the target data encoding method on which the corresponding data sending end obtains the encoded data. This application does not describe in detail the decoding implementation process of the encoded data (i.e., the decompression processing method of the received compressed data), which can be determined based on the data encoding and decoding working principle of the corresponding compression algorithm.

[0114] After receiving the encoded data of each frame of image, the terminal device receiving the second data stream, i.e., the data receiving end, can directly decode it to obtain the decoded data of the corresponding frame of image. During this decoding process, if the data receiving end receives the instruction encoded data and texture encoded data of each frame of image synchronously through different channels, the image decoder can also decode the encoded data transmitted through the corresponding channels based on their respective target data decoding methods to obtain the corresponding instruction decoded data (i.e., instruction data) and texture decoded data (i.e., image texture data). If the data receiving end receives the encoded data of each frame of image through one channel, it can directly decode it. Therefore, the processing method of the image decoder can be determined based on the processing method of the image encoder at the corresponding data sending end, and this application does not impose any restrictions on this.

[0115] Step S83: Process the decoded data of each frame image and display the corresponding frame image.

[0116] Following the analysis above, if the decoded data of each frame of an image obtained by the image decoder contains the complete graphic data of that frame, it can be transmitted to the image renderer for rendering processing to obtain the corresponding frame image, which is then sent to the display module of the data receiving end for display output. If the decoded data of each frame of an image obtained by the image decoder contains image texture data that is only a part of the image texture data of that frame (i.e., the necessary image texture data), it can restore the complete image texture data or graphic data of each frame of an image according to the preset conditions on which the image texture data extraction operation executed by the data sending end is based, thereby achieving lossless image reconstruction, avoiding the image quality loss caused by lossy compression, and improving the quality of the displayed image.

[0117] Optional, combined Figure 4 and Figure 5 The schematic diagram shows that the image decoder decodes the input encoded data. It can transmit the decoded instruction data and image texture data to the image renderer for rendering, depending on the situation, through different channels or a single channel. Similarly, if the decoded image texture data is incomplete, the instruction data and image texture data of each frame can be transmitted simultaneously to the image reconstructor through different channels or a single channel. This allows the image reconstructor to restore the corresponding image texture data based on the instruction data of each frame, thus reconstructing the complete graphic data contained in that frame before sending it to the image renderer for rendering, ensuring the quality of the image displayed at the data receiving end.

[0118] In summary, in this embodiment of the application, the data sending end directly encodes the graphic data of each frame of the image before transmitting it, which reduces the data encoding steps and the amount of data transmitted. Correspondingly, it reduces the amount of data received by the data receiving end and the decoding steps of the encoded data of each frame of the image, thereby improving the data decoding efficiency and reducing the end-to-end latency.

[0119] Reference Figure 9 This is a flowchart illustrating another optional example of the image data processing method proposed in this application. This embodiment describes an optional refined implementation of the image data processing method executed by the data receiving end. This refined implementation can correspond to the refined implementation of the image data processing method executed by the data sending end, such as... Figure 9 As shown, the method may include:

[0120] Step S91: Obtain a second data stream; the second data stream includes encoded data of multiple frames of images;

[0121] Step S92: Based on the target data decoding method, the encoded data of each frame of image is processed to obtain a third data stream; the third data stream includes the decoded data of multiple frames of images;

[0122] The implementation process of steps S91 and S92 can be referred to the description of the corresponding parts of the above embodiments, and will not be described in detail here.

[0123] Step S93: Based on the instruction data contained in the decoded data of each frame image, the corresponding frame image is reconstructed to obtain the fourth data stream;

[0124] After the terminal device sending the second data stream receives the first data stream, in order to reduce the amount of encoded and transmitted data, improve encoding efficiency, and reduce latency, it can, as described above, remove the image texture data that meets preset conditions from each frame of the first data stream before inputting it into the image encoder for encoding processing, and then transmit the resulting second data stream. Therefore, in this scenario, the decoded data obtained by any corresponding data receiving end after decoding the encoded data of each frame of the image is incomplete. If it is directly rendered, the displayed image quality will be very poor. Therefore, the data receiving end needs to first restore the complete graphic data of each frame of the image.

[0125] Since the instruction data in the graphic data of each frame of an image can specify the texture at which location within that frame, the image reconstructor can reconstruct the image texture data at each location within that frame based on the instruction data, thus reconstructing the frame and obtaining the fourth data stream. This fourth data stream can include graphic data from multiple frames, and the amount of graphic data from the fourth data stream for the same frame is greater than the decoded data. This fourth data stream is essentially the same as the first data stream, achieving lossless image reconstruction and avoiding the image quality loss caused by lossy compression (i.e., the aforementioned encoding process).

[0126] Based on the above analysis, the implementation method of step S93 may include, but is not limited to: obtaining image texture data that meets preset conditions based on the instruction data contained in the decoded data of each frame image. In conjunction with the relevant description of the preset conditions above, the image texture data that meets the preset conditions may include the image texture data of the corresponding frame image in the first data stream that has been deduplicated, and / or the image texture data that is pre-stored based on the data source of the first data stream or the received second data stream and has been removed from the first data stream, i.e., the aforementioned preset texture data, etc. The image reconstructor can reconstruct the corresponding frame image according to the preset conditions and adopt the corresponding restoration method to obtain relatively complete graphic data. In this way, the decoded data of the same frame image (which includes instruction data and necessary image texture data) and the image texture data that meets the preset conditions can be used to construct the graphic data of the corresponding frame image, thereby forming the fourth data stream.

[0127] Step S94: Render the graphic data of each frame in the fourth data stream and display the corresponding frame image.

[0128] Following the above analysis, in a scenario where the data sending end removes unnecessary image texture data (i.e., image elements) from each frame of image and only encodes the instruction data and necessary image texture data of that frame before transmission, the data receiving end quickly decodes the encoded data of each frame of image received, first restores the instruction data and image texture data of that frame of image, and then performs rendering processing by the image renderer to obtain a high-quality frame of image, while also greatly reducing end-to-end latency.

[0129] Reference Figure 10 This is a schematic diagram of an optional example of the image data processing apparatus proposed in this application. This embodiment describes the composition structure of the image data processing apparatus implementing the image data processing method executed on the data sending end side, such as... Figure 10 As shown, the device may include:

[0130] The first data stream acquisition module 101 is used to acquire a first data stream, which includes graphic data of multiple frames of images; the graphic data of each frame of images is processed by an image renderer to obtain a frame of image for display output.

[0131] The graphics data processing module 102 is used to process the graphics data of each frame of the image based on the target data encoding method to obtain a second data stream, the second data stream including the encoded data of multiple frames of the image;

[0132] The second data stream transmission module 103 is used to transmit the second data stream, which is at least used for the terminal device receiving the second data stream to decode and render it.

[0133] Optionally, the amount of data in the graphic data of the same frame image is greater than or equal to the amount of data in the encoded data.

[0134] In some embodiments, since the graphic data of each frame of an image in the first data stream typically includes various instruction data and image texture data, in order to reduce the amount of encoded and transmitted data, improve encoding efficiency, and further reduce end-to-end latency, such as... Figure 11 As shown, the above-mentioned image data processing apparatus may further include:

[0135] The data removal module 104 is used to remove image texture data that meets preset conditions from at least one frame of the first data stream;

[0136] Based on this, the aforementioned graphics data processing module 102 may include:

[0137] The first encoding unit is used to process the graphic data of each frame of the image after the image texture data removal process, based on the target data encoding method, to obtain the second data stream.

[0138] Optionally, the data removal module 104 mentioned above may include:

[0139] The deduplication processing unit is used to perform deduplication processing on the image texture data of each frame in the first data stream; and / or,

[0140] A preset texture data removal unit is used to remove preset texture data contained in each frame of the first data stream; the preset texture data is determined and stored by the terminal device that is to receive the second data stream based on the data source of the first data stream or the second data stream that has been received.

[0141] In some other embodiments, the above-described graphics data processing module 102 may also include:

[0142] The second encoding unit is used to encode the instruction data and image texture data from different channels in the graphic data of each frame of the image based on the target data encoding method, so as to obtain the second data stream of the corresponding channel.

[0143] Accordingly, the second data stream transmission module 103 may include:

[0144] A synchronous transmission unit is used to synchronously transmit the second data stream from different channels.

[0145] Optionally, the above-mentioned graphics data processing module 102 may also include:

[0146] The third encoding unit is used to encode the graphic data of each frame of an image transmitted through one channel based on the target data encoding method, so as to obtain the second data stream.

[0147] Reference Figure 12 This is a schematic diagram of another optional example of the image data processing apparatus proposed in this application. This embodiment describes the composition structure of the image data processing apparatus implementing the image data processing method executed on the data receiving end side, such as... Figure 12 As shown, the device may include:

[0148] The second data stream acquisition module 121 is used to acquire a second data stream, which includes encoded data of multiple frames of images. The encoded data of each frame of image is obtained by the terminal device that sends the second data stream processing the graphic data of each frame of image in the acquired first data stream based on the target data encoding method. The graphic data of each frame of image is processed by the image renderer of the terminal device to obtain a frame of image for display output.

[0149] The encoded data processing module 122 is used to process the encoded data of each frame of image based on the target data decoding method to obtain a third data stream; the third data stream includes the decoded data of multiple frames of images;

[0150] The decoding data processing module 123 is used to process the decoding data of each frame of image and display the corresponding frame image obtained by the processing.

[0151] In this case, the amount of encoded data for the same frame of image is less than or equal to the amount of graphic data used to display the frame of image.

[0152] In some embodiments, such as Figure 13 As shown, the above-mentioned decoding data processing module 123 may include:

[0153] The image reconstruction unit (such as the image reconstructor described above) 1231 reconstructs the corresponding frame image based on the instruction data contained in the decoded data to obtain a fourth data stream; the fourth data stream includes graphic data of multiple frames of images, and the amount of graphic data from the fourth data stream of the same frame image is greater than the amount of decoded data.

[0154] The image rendering unit (such as the image renderer described above) 1232 is used to render the graphic data of each frame of the image in the fourth data stream and display the corresponding frame image.

[0155] Optionally, the image reconstruction unit 1231 described above may include:

[0156] The data restoration unit is used to obtain image texture data that meets preset conditions based on the instruction data contained in the decoded data; the image texture data that meets preset conditions includes image texture data of the corresponding frame image in the first data stream after deduplication, and / or image texture data pre-stored based on the data source of the first data stream or the received second data stream and removed from the first data stream.

[0157] The fourth data stream construction unit is used to construct a fourth data stream using the decoded data of the same frame image and the image texture data that meets the preset conditions.

[0158] It should be noted that the various modules and units in the above-mentioned device embodiments can all be stored in the memory as program modules. The processor executes the above-mentioned program modules stored in the memory to realize the corresponding functions. The functions realized by each program module and its combination, as well as the technical effects achieved, can be referred to the description of the corresponding part of the above-mentioned method embodiments. This embodiment will not repeat them here.

[0159] This application also provides a computer-readable storage medium on which a computer program can be stored, which can be called and loaded by a processor to implement the various steps of the image data processing method executed by the data sending end or data receiving end of the above embodiments.

[0160] Finally, it should be noted that, regarding the above embodiments, unless the context explicitly indicates an exception, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, and these steps and elements do not constitute an exclusive list; the method or apparatus may also include other steps or elements. An element defined by the phrase "comprising an..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes the element.

[0161] In the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more.

[0162] The terms used in this application, such as "first" and "second," are for descriptive purposes only, used to distinguish one operation, unit, or module from another, and do not necessarily require or imply any such actual relationship or order between these units, operations, or modules. Furthermore, they should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0163] Furthermore, the various embodiments in this specification are described in a progressive or parallel manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatuses, systems, terminal devices, and storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant details can be found in the method section.

[0164] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image data processing method, the method comprising: Obtain a first data stream, the first data stream comprising graphic data of multiple frames of images; The graphic data of each frame of an image is processed by an image renderer to obtain a frame of image for display output; Based on the target data encoding method, the graphic data of each frame of the image is processed to obtain a second data stream. The second data stream includes the encoded data of multiple frames of the image. The necessary graphic data of each frame of the image is processed to avoid processing unnecessary graphic data. The unnecessary graphic data includes duplicate graphic data and / or graphic data already stored in the terminal device receiving the second data stream. The second data stream is transmitted, and the second data stream is used at least for the terminal device receiving the second data stream to decode and render.

2. The method according to claim 1, wherein the amount of data of the graphic data in the same frame image is greater than or equal to the amount of data of the encoded data.

3. The method according to claim 2, wherein the graphic data of each frame image includes instruction data and image texture data, and the method further includes: Remove image texture data that meets preset conditions from at least one frame of the first data stream; The second data stream is obtained by processing the graphic data of each frame of the image based on the target data encoding method, including: Based on the target data encoding method, the graphic data of each frame of the image after the image texture data removal process is processed to obtain the second data stream.

4. The method according to claim 3, wherein removing image texture data that meets preset conditions from at least one frame of the first data stream comprises: Deduplication is performed on the image texture data of each frame in the first data stream; And / or, Remove the preset texture data contained in each frame of the first data stream; The preset texture data is determined and stored by the terminal device that will receive the second data stream based on the data source of the first data stream or the second data stream that has already been received.

5. The method according to any one of claims 1-4, wherein processing the graphic data of each frame of an image based on the target data encoding method to obtain a second data stream comprises: Based on the target data encoding method, the instruction data and image texture data from different channels in the graphic data of each frame image are encoded to obtain the second data stream of the corresponding channel; The transmission of the second data stream includes: The second data stream from different channels is transmitted synchronously.

6. An image data processing method, the method comprising: Obtain a second data stream, which includes encoded data of multiple frames of images; The encoded data of each frame of the image is obtained by the terminal device sending the second data stream processing the graphic data of each frame of the image in the first data stream based on the target data encoding method; wherein, the necessary graphic data of each frame of the image is processed, and the unnecessary graphic data is avoided from being processed. The unnecessary graphic data includes duplicate graphic data and / or graphic data already stored in the terminal device receiving the second data stream. Based on the target data decoding method, the encoded data of each frame of image is processed to obtain a third data stream; the third data stream includes the decoded data of multiple frames of images; The decoded data of each frame of the image is processed and the corresponding frame of the image is displayed.

7. The method according to claim 6, wherein the amount of encoded data of the same frame image is less than or equal to the amount of graphic data used to display the frame image.

8. The method according to claim 7, wherein processing the decoded data of each frame image and displaying the corresponding frame image comprises: Based on the instruction data contained in the decoded data, the corresponding frame image is reconstructed to obtain the fourth data stream; The fourth data stream includes graphic data from multiple frames of images, and the amount of graphic data from the fourth data stream for the same frame of image is greater than the amount of decoded data. Render the graphic data of each frame in the fourth data stream and display the corresponding frame image; The process of reconstructing the corresponding frame image based on the instruction data contained in the decoded data to obtain the fourth data stream includes: Based on the instruction data contained in the decoded data, image texture data that meets preset conditions is obtained; the image texture data that meets preset conditions includes image texture data of the corresponding frame image in the first data stream after deduplication, and / or image texture data pre-stored based on the data source of the first data stream or the received second data stream and removed from the first data stream. The fourth data stream is constructed using the decoded data of the same frame image and the image texture data that meets the preset conditions.

9. An image data processing apparatus, the apparatus comprising: The first data stream acquisition module is used to acquire a first data stream, which includes graphic data of multiple frames of images; The graphic data of each frame of an image is processed by an image renderer to obtain a frame of image for display output; The image data processing module is used to process the image data of each frame of an image based on the target data encoding method to obtain a second data stream. The second data stream includes the encoded data of multiple frames of images. The necessary image data of each frame of an image is processed to avoid processing unnecessary image data. The unnecessary image data includes duplicate image data and / or image data already stored in the terminal device receiving the second data stream. The second data stream transmission module is used to transmit the second data stream, which is at least used by the terminal device receiving the second data stream to decode and render it.

10. An image data processing apparatus, the apparatus comprising: The second data stream acquisition module is used to acquire a second data stream, which includes encoded data of multiple frames of images; The encoded data of each frame of the image is obtained by the terminal device sending the second data stream processing the graphic data of each frame of the image in the first data stream based on the target data encoding method; wherein, the necessary graphic data of each frame of the image is processed, and the unnecessary graphic data is avoided from being processed. The unnecessary graphic data includes duplicate graphic data and / or graphic data already stored in the terminal device receiving the second data stream. The encoded data processing module is used to process the encoded data of each frame of image based on the target data decoding method to obtain a third data stream; the third data stream includes the decoded data of multiple frames of images; The decoding data processing module is used to process the decoded data of each frame of the image and display the corresponding frame image obtained from the processing.

Images