Methods, electronic devices, and computer program products for data processing

Abstract

Embodiments of this disclosure relate to methods, electronic devices, and computer program products for data processing. The data processing method includes generating target tags for target data content segments based on a category of the target data content segment; obtaining a set of reference tags, wherein reference tags in the set are generated based on categories of historical data content segments; and determining the redundancy of the target data content segment based on a comparison between the target tags and the set of reference tags. In this manner, the distribution of large amounts of redundant data from edge servers to terminal devices can be avoided, thereby saving bandwidth and improving latency, thus providing a better user experience for AR / VR content experiences.

Description

Technical Field

[0001] Embodiments of this disclosure generally relate to data processing systems, and more particularly to a method for data processing, electronic devices, and computer program products. Background Technology

[0002] It is already known that augmented reality (AR) and virtual reality (VR) technologies are among the key use cases in fifth-generation mobile communication technology (5G) and mobile edge computing (MEC). With the development of 5G / MEC, the above goals can be achieved by leveraging 5G networks and edge servers to achieve high bandwidth and low latency.

[0003] The large amounts of data transmitted from edge servers to end devices (such as AR glasses or VR headsets), including 3D models, animations, and interactive content, typically consume significant bandwidth and often introduce latency. Furthermore, during content distribution, redundant content—content that has already been transmitted to the end device—is repeatedly transmitted from the edge server back to the end device. Summary of the Invention

[0004] Embodiments of this disclosure relate to an improvement for data processing.

[0005] In a first aspect of this disclosure, a method for data processing is provided. The method includes generating target tags for the target data content segment based on a category of the target data content segment; obtaining a set of reference tags, wherein reference tags in the set are generated based on categories of historical data content segments; and determining the redundancy of the target data content segment based on a comparison between the target tags and the set of reference tags.

[0006] In a second aspect of this disclosure, an electronic device is provided. The electronic device includes at least one processor; and at least one memory storing computer program instructions, the at least one memory and the computer program instructions being configured to, together with the at least one processor, cause the electronic device to perform actions. The actions include generating a target tag for the target data content segment based on a category of the target data content segment; obtaining a set of reference tags, the reference tags in the set being generated based on categories of historical data content segments; and determining the redundancy of the target data content segment based on a comparison of the target tag with the set of reference tags.

[0007] In a third aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-volatile computer-readable medium and includes machine-executable instructions. When executed, the machine-executable instructions cause a device to perform the method of the first aspect.

[0008] It should be understood that the description in the Summary of the Invention section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other objects, features, and advantages of embodiments of the present disclosure will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the present disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0010] Figure 1 A block diagram of an example system in which embodiments of the present disclosure may be implemented is shown;

[0011] Figure 2 A flowchart of a method 200 for data processing according to an embodiment of the present disclosure is shown;

[0012] Figure 3 A schematic diagram illustrating an example of processing a target data content segment according to an embodiment of the present disclosure is shown;

[0013] Figure 4 A schematic diagram of a processing history content segment according to an embodiment of the present disclosure is shown;

[0014] Figure 5 A schematic block diagram of an apparatus that can be used to implement embodiments of the present disclosure is shown.

[0015] Throughout all the accompanying drawings, the same or similar reference numerals are used to denote the same or similar components. Detailed Implementation

[0016] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments illustrated in the accompanying drawings. It should be understood that these specific embodiments are described only to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way.

[0017] Figure 1 A schematic diagram of an exemplary system 100 in which embodiments of the present disclosure may be implemented is shown.

[0018] like Figure 1 As shown, system 100 may include remote device 110. Hereinafter, remote device 110 may also be referred to as edge server 110. Remote device 110 can acquire images and audio content captured by image acquisition devices or audio acquisition devices. This image and audio content can be considered as image and audio content applied to AR / VR technology.

[0019] System 100 may further include terminal device 120. Terminal device 120 can receive data information from remote device 110. For example, terminal device 120 can communicate with remote device 110 via wired or wireless communication. In embodiments of this disclosure, terminal device 120 can be considered as a wearable device based on AR / VR technology. For example, terminal device 120 can be an AR head-mounted display device, a VR head-mounted display device, and headphones, etc.

[0020] System 100 may further include computing device 102. Computing device 102 can process image and audio content acquired by remote device 110. In some embodiments, computing device 102 may be integrated into remote device 110. In some embodiments, computing device 102 may also be considered as an external device independent of remote device 110.

[0021] Most current AR / VR content management solutions are still based on local storage, meaning that content is pre-stored on the terminal device and then processed on the device. This approach significantly increases the burden on the terminal device to store and process large amounts of data, which affects the lightweight design of the terminal device and is detrimental to saving battery life.

[0022] With the development of 5G and edge computing, data and content can be offloaded to edge servers, and data transmission from edge servers to terminal devices can be achieved through high-bandwidth 5G networks. For example, data can be processed first using edge servers and then distributed to terminal devices. This approach enables lightweight design of terminal devices and saves battery life.

[0023] However, the above solutions still have certain limitations. For example, as mentioned above, the large amounts of data transmitted from edge servers to terminal devices, such as 3D models, animations, and interactive content, typically consume significant bandwidth and often introduce latency. Furthermore, during content distribution, some redundant content—content that has already been transmitted to the terminal device—is repeatedly transmitted from the edge server back to the terminal device.

[0024] Because the data transmission based on AR / VR technology has extremely low latency requirements, bandwidth and latency limitations will affect content delivery and reduce user satisfaction with the experience.

[0025] Generally, with the growth of data and the demand for 24 / 7 service availability, data storage and processing equipment face challenges in managing and protecting data. Often, much redundant data is distributed across Internet of Things (IoT) or 5G or edge / cloud servers, which increases the size of the data management and backup window and leads to unnecessary resource consumption, such as data storage, processing, and backup storage space, as well as network bandwidth. Storing large amounts of duplicate data at remote edge sites or in the cloud for data management and processing also requires significant bandwidth.

[0026] Therefore, deduplication techniques have been introduced to process image and audio data. A typical deduplication process involves three steps: chunking the dataset, identifying duplicate chunks, and eliminating redundant chunks. To identify redundant chunks, the deduplication system creates a hash value or digital signature, such as a fingerprint, for each data chunk. Simultaneously, a signed index can be created for a given repository. The index provides a list of references to determine if the data chunk exists in the repository. However, hash-based deduplication processes are computationally intensive and lack the required level of intelligence.

[0027] Therefore, embodiments of this disclosure propose a data processing method. This method generates semantic tags for a current data content segment based on its category. The semantic tags are compared with a previously obtained set of reference tags to determine whether the current data content segment is a redundant data content segment.

[0028] Figure 2 A flowchart of a method 200 for data processing according to some embodiments of the present disclosure is shown. Method 200 can be implemented in... Figure 1 In the system 100 shown. Method 200 can, for example, be set by... Figure 1 The system 100 shown is implemented using a remote device 110 (with computing device 102 integrated within it), or it can be implemented using computing device 102 within system 100. It can be combined with... Figure 1 Let's describe method 200.

[0029] like Figure 2 As shown in box 210, if a target data content segment is obtained, the computing device 102 can generate a target label for the target data content segment based on the category of the target data content segment.

[0030] In some embodiments, feature parameters can be extracted from a target data content segment. If the target data content segment includes image data, the feature parameters may include visual features of the image, such as color, shape, size, texture, etc. Furthermore, the feature parameters may also be associated with objects included in the image. For example, the feature parameters may be object type or object behavior, etc.

[0031] In some embodiments, if the target data content segment includes audio data, the feature parameter may also be an audio element within that audio data. For example, the audio element may be a repeating audio segment within the audio data.

[0032] After extracting the feature parameters, the computing device 102 can determine the category of the target data content segment based on at least one of the aforementioned feature parameters and generate a target label corresponding to the acquired target data content segment based on the category.

[0033] In some embodiments, the extraction of feature parameters of target data content segments, the classification of target content segments, and the generation of labels for target content segments can be achieved using computer vision and machine learning algorithms. These methods may include, for example, YoLo, SimpleCV, and TensorFlow. These algorithms can identify and classify the objects, scenes, and behaviors within the data content segments. The classified objects, scenes, and behaviors can then be defined using semantic tags.

[0034] In some embodiments, if the target data content segment includes image data, background subtraction can be used to separate the background layer from the foreground layer of the image in order to identify objects, scenes, and behaviors in the image. This background subtraction can be, for example, a Gaussian mixture model-based background subtraction. Furthermore, motion detection can be used to separate the foreground layer from the background layer of an image. This method enables the detection of moving objects in the foreground using optical flow-based motion detection methods.

[0035] Figure 3 A schematic diagram illustrating an example of processing a target data content segment according to an embodiment of the present disclosure is shown. Figure 3 The target data content segment 310 is shown. After performing background subtraction on the target data content segment 310, the processed target data content segment 320 can be obtained. In the processed target data content segment 320, it can be seen that the foreground object 301 in the target data content segment 310 has been extracted.

[0036] A trained machine learning model, such as YoLo, can be used to process the foreground, foreground object, or background layers of the extracted target data content segment. For example, the foreground layer, foreground object, or background layer can be used as input to the machine learning model. The model can then generate semantic labels describing the foreground layer, foreground object, or background layer and output these labels.

[0037] In some embodiments, the target labels generated by the machine learning model corresponding to the target data content segment may be provided, for example, in the following form:

[0038] [Foreground layer, visual features (such as color), object type (such as horse), object ID;]

[0039] [Background layer, visual features (such as color), object type (such as grassland), object ID].

[0040] See again Figure 2 At box 220, after generating target tags corresponding to target data content segments, computing device 102 can refer to a tag set in which reference tags are generated based on the categories of historical data content segments.

[0041] In some embodiments, the reference tag set can be generated based on multiple historical data content segments. Feature parameters can be extracted from the historical data content segments. If the historical data content segment includes image data, the feature parameters can include visual features of the image, such as color, shape, size, texture, etc. Furthermore, the feature parameters can also be associated with objects included in the image. For example, the feature parameters can be object type or object behavior, etc.

[0042] In some embodiments, if the historical data content segment includes audio data, the feature parameter may also be an audio element within that audio data. For example, the audio element may be a repeatedly played audio segment within the audio data.

[0043] After extracting the feature parameters, the computing device 102 can determine the category of the historical data content segment based on at least one of the aforementioned feature parameters and generate a reference label corresponding to the acquired historical data content segment based on the category.

[0044] The computing device 102 generates a set of reference tags based on one or more historical data content segments. Figure 4 A schematic diagram of a processing history content segment according to an embodiment of the present disclosure is shown.

[0045] like Figure 4 As shown, computing device 102 can acquire multiple historical data groups 410, 420, and 430. Each of the multiple historical data groups 410, 420, and 430 includes multiple historical data content segments. For example, historical data group 410 includes historical data content segments 411 to 416, historical data group 420 includes historical data content segments 421 to 425, and historical data group 430 includes historical data content segments 431 to 435.

[0046] After identifying feature parameters of multiple historical data content segments, classifying the historical data content segments, and generating tags to describe the historical data content segments, historical data content segments with duplicate tags are deleted, thereby obtaining a set 440 of historical data content segments to be sent to the terminal device 120. This set 440 of historical data content segments may include historical data content segments 441 to 443.

[0047] In some embodiments, the extraction of feature parameters of historical data content segments, the classification of historical content segments, and the generation of a reference label set based on the classification can be performed using computer vision and machine learning algorithms. These methods may include, for example, YoLo, SimpleCV, and TensorFlow. This process is similar to the process and methods for processing target data content segments already described above, and therefore will not be repeated here.

[0048] In some embodiments, the set of tags associated with the processed and retained historical data content segments 441 to 443 can be regarded as a reference tag set for processing subsequent data content segments. This reference tag set can be stored as metadata at a remote device 110.

[0049] In some embodiments, the reference tag set may be sent to the terminal device 120 as metadata along with processed and retained historical data content segments 441 to 443.

[0050] As can be seen, the number of historical data segments retained after processing is significantly reduced compared to the number of unprocessed historical data segments. This improves latency in data transmission and saves bandwidth. It also benefits power saving in terminal devices and improves system performance.

[0051] Back to Figure 2 At box 230, computing device 102 can determine the redundancy of target data content segments based on a comparison of the target tag and the reference tag set.

[0052] In some embodiments, the computing device 102 can determine whether there exists a historical data content segment that matches the target data content segment in a set of historical data content segments corresponding to the set of reference tags by comparing the target tag and the set of reference tags.

[0053] If the computing device 102 determines that no matching historical data content segment exists, it determines the target data content segment as a non-redundant data content segment. If the computing device 102 determines that a matching historical data content segment exists, it determines the target data content segment as a redundant data content segment.

[0054] In some embodiments, if the target data content segment is determined to be a non-redundant data content segment, the target data content segment can be sent to the terminal device.

[0055] In some embodiments, target tags generated based on target data content segments can be added to a set of reference tags generated based on historical data content segments.

[0056] In some embodiments, if the target data content segment is determined to be a non-redundant data content segment, the target data content segment will be deleted.

[0057] The embodiments of this disclosure implement a framework for managing AR / VR content using an MEC / 5G environment to achieve efficient data processing, content management, and distribution. This framework can reduce the computational burden on terminal devices and improve the customer experience in terms of mobility, efficiency, and immersive experiences.

[0058] Furthermore, embodiments of this disclosure introduce the concept of data deduplication. Instead of distributing all data to all terminal devices, the deduplication concept from data storage and production is used to check whether data blocks have already been distributed. If they have been distributed, reallocation is avoided to reduce redundancy. This deduplication process can be implemented using computer vision processing and machine learning algorithms. For example, machine learning models can be used to understand and classify content in AR / VR, and then tags can be added as deduplicated metadata. Unlike traditional hash library deduplication, the methods of embodiments of this disclosure can fully understand the meaning of the content, providing more efficient data processing for AR / VR content distribution.

[0059] Figure 5 A block diagram schematically illustrates a device 500 that can be used to implement embodiments of the present disclosure. Device 500 may be implemented as or included in... Figure 1 The remote device 110 or computing device 102.

[0060] like Figure 5 As shown, device 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to computer program instructions stored in read-only storage device (ROM) 502 or loaded from storage unit 508 into random access storage device (RAM) 503. The RAM 503 may also store various programs and data required for the operation of device 500. CPU 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0061] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 505, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0062] The various processes and procedures described above, such as process 200, may be executed by processing unit 501. For example, in some embodiments, process 200 and / or process 600 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by CPU 501, one or more steps of process 200 described above may be performed.

[0063] As used herein, the term "comprising" and similar expressions should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "an embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. This document may also include other explicit and implicit definitions.

[0064] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, investigation, searching (e.g., looking in a table, database, or other data structure), ascertainment, etc. Furthermore, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Additionally, "determine" can include parsing, selecting, choosing, building, etc.

[0065] It should be noted that the embodiments of this disclosure can be implemented using hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, code provided on programmable memory or a data carrier such as an optical or electronic signal carrier.

[0066] Furthermore, although the operation of the methods of this disclosure is described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps. It should also be noted that the features and functions of two or more devices according to this disclosure may be embodied in one device. Conversely, the features and functions of one device described above may be further divided and embodied by multiple devices.

[0067] While this disclosure has been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the specific embodiments disclosed. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for data processing, comprising: Target tags are generated for the target data content segment based on the category of the target data content segment requested by the terminal device; A reference tag set is obtained, which is generated at least in part by processing multiple different historical data groups, each containing a different set of historical data content segments. The processing includes: identifying historical data content segments in the respective sets of historical data content segments that have duplicate content tags in the multiple different historical data groups; and deduplicating the historical data content segments in the respective sets of historical data content segments by including multiple reference tags in the reference tag set for each selected instance of the historical data content segments identified as having duplicate content tags in the multiple different historical data groups. The reference tags in the reference tag set are generated based on the categories of historical data content segments previously provided to the terminal device. The redundancy of the target data content segment is determined based on a comparison between the target tag and the reference tag set. The generation, acquisition, and determination steps are performed in at least one computing device associated with the edge server; In this embodiment, at least one computing device associated with the edge server implements a trained machine learning model, which is configured to process at least one of a foreground layer, a foreground object, and a background layer of at least one image of a target data content fragment requested by the terminal device, to generate at least a portion of the target label as a semantic label representing at least one of the foreground layer, the foreground object, and the background layer; and wherein the determined result is used to control whether to provide at least a portion of the target data content fragment from the edge server to the terminal device.

2. The method according to claim 1, wherein determining the redundancy of the target data content segment includes: Based on the comparison between the target tag and the reference tag set, it is determined whether there is a historical data content segment that matches the target data content segment in the historical data content segment set corresponding to the reference tag set; as well as If it is determined that no matching historical data content segment exists, the target data content segment is determined as a non-redundant data content segment.

3. The method according to claim 2, further comprising: The target data content segment is sent to the terminal device.

4. The method according to claim 2, further comprising: Add the target label to the reference label set.

5. The method according to claim 2, further comprising: If a matching historical data content segment is determined, the target data content segment is identified as a redundant data content segment.

6. The method of claim 1, wherein generating the target tag comprises: Extract at least one of the following parameters from the target data content segment: Visual features of an image The types of objects in the image, The behavior of objects in the image, and Audio elements; as well as The category of the target data content segment is determined based on at least one of the parameters; as well as Based on the determined category, target tags corresponding to the target data content segment are generated.

7. The method according to claim 1, wherein obtaining the reference tag set comprises: Extract at least one of the following parameters from a historical data segment: Visual features of an image The types of objects in the image, The behavior of objects in the image, and Audio elements; The category of the historical data content segment is determined based on at least one of the parameters; and The reference tags corresponding to the historical data content segments are generated based on the determined categories.

8. The method according to claim 7, further comprising: The historical data content segment and the reference tag set are sent to the terminal device.

9. The method of claim 1, wherein the terminal device comprises at least one of the following: Augmented reality (AR) head-mounted display devices, Virtual reality (VR) head-mounted display devices, or earphone.

10. An electronic device, comprising: processor; as well as A memory coupled to the processor stores instructions to be executed, which, when executed by the processor, cause the electronic device to perform actions, including: Target tags are generated for the target data content segment based on the category of the target data content segment requested by the terminal device; A reference tag set is obtained, which is generated at least in part by processing multiple different historical data groups, each containing a different set of historical data content segments. The processing includes: identifying historical data content segments in the respective sets of historical data content segments that have duplicate content tags in the multiple different historical data groups; and deduplicating the historical data content segments in the respective sets of historical data content segments by including multiple reference tags in the reference tag set for each selected instance of the historical data content segments identified as having duplicate content tags in the multiple different historical data groups. The reference tags in the reference tag set are generated based on the categories of historical data content segments previously provided to the terminal device. The redundancy of the target data content segment is determined based on a comparison between the target tag and the reference tag set. The generation, acquisition, and determination steps are performed in at least one computing device associated with the edge server; In this embodiment, at least one computing device associated with the edge server implements a trained machine learning model, which is configured to process at least one of a foreground layer, a foreground object, and a background layer of at least one image of a target data content fragment requested by the terminal device, to generate at least a portion of the target label as a semantic label representing at least one of the foreground layer, the foreground object, and the background layer; and wherein the determined result is used to control whether to provide at least a portion of the target data content fragment from the edge server to the terminal device.

11. The device of claim 10, wherein determining the redundancy of the target data content segment comprises: Based on the comparison between the target tag and the reference tag set, it is determined whether there is a historical data content segment that matches the target data content segment in the historical data content segment set corresponding to the reference tag set; as well as If it is determined that no matching historical data content segment exists, the target data content segment is determined as a non-redundant data content segment.

12. The device according to claim 11, wherein the action further includes: The target data content segment is sent to the terminal device.

13. The device according to claim 11, wherein the action further includes: Add the target label to the reference label set.

14. The device according to claim 11, wherein the action further includes: If a matching historical data content segment is determined, the target data content segment is identified as a redundant data content segment.

15. The device of claim 10, wherein generating the target tag comprises: Extract at least one of the following parameters from the target data content segment: Visual features of an image The types of objects in the image, The behavior of objects in the image, and Audio elements; as well as The category of the target data content segment is determined based on at least one of the parameters; as well as Based on the determined category, target tags corresponding to the target data content segment are generated.

16. The device of claim 10, wherein obtaining the reference tag set comprises: Extract at least one of the following parameters from a historical data segment: Visual features of an image The types of objects in the image, The behavior of objects in the image, and Audio elements; The category of the historical data content segment is determined based on at least one of the parameters; and The reference tags corresponding to the historical data content segments are generated based on the determined categories.

17. The device according to claim 16, further comprising: The historical data content segment and the reference tag set are sent to the terminal device.

18. The device of claim 10, wherein the terminal device comprises at least one of the following: Augmented reality (AR) head-mounted display devices, Virtual reality (VR) head-mounted display devices, or earphone.

19. A computer program product tangibly stored on a non-transient computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the steps of the method according to any one of claims 1 to 9.

Images