Method, device and computer program product for document transfer

Abstract

Embodiments of the present disclosure relate to a method, device and computer program product for document transmission. The method comprises: selecting, from a document, a portion to be encoded, the document comprising structured portions, each of the structured portions comprising an identifier and data associated with the identifier; obtaining at least one structured portion of the portion to be encoded; performing, for a respective structured portion of the at least one structured portion, the following operations to encode the document: generating a hash value of the data of the respective structured portion; determining whether the hash value repository comprises the hash value; and based on determining that the hash value repository comprises the hash value, replacing the data of the respective structured portion in the document with the hash value; and transmitting the encoded document to a recipient. This approach can reduce bandwidth consumption for transmitting a document comprising structured data while ensuring that the recipient obtains the required information, and potentially saving the required storage space for the recipient to store the received document.

Description

Technical Field

[0001] Embodiments of this disclosure relate to document transfer technology, and more specifically, to methods, apparatus, and computer program products for transferring documents including structured data. Background Technology

[0002] In managing client devices, to obtain accurate status information, it's necessary to send telemetry data reflecting various information about the client device to a server backend (e.g., periodically). In traditional solutions, client devices typically send documents containing structured data (e.g., using JSON format) to the backend, containing all the predefined data types required for subsequent management of the client device. These documents are usually large and need to be sent frequently. Therefore, technicians are still searching for more efficient methods to transmit such telemetry data. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for document transmission is provided. The method includes: selecting a portion to be encoded from a document, the document including structured portions, each structured portion including an identifier and data associated with the identifier; obtaining at least one structured portion of the portion to be encoded; performing the following operations for a corresponding structured portion of the at least one structured portion to encode the document: generating a hash value of the data of the corresponding structured portion; determining whether a hash value repository includes the hash value; and, based on determining that the hash value repository includes the hash value, replacing the data of the corresponding structured portion in the document with the hash value; and sending the encoded document to a recipient.

[0004] In a second aspect of this disclosure, a method for document transmission is provided. The method includes: receiving a transmitted document from a sender, the document including structured portions, each structured portion including an identifier and data associated with the identifier; selecting a portion to be decoded from the document; obtaining at least one structured portion of the portion to be decoded; and performing the following operations for a corresponding structured portion of the at least one structured portion: determining whether the corresponding structured portion includes a valid hash value; and based on determining that the corresponding structured portion includes a valid hash value: determining whether a hash document repository includes a corresponding entry for the valid hash value, the corresponding entry storing data of the document portion and a hash value of the data, the hash value of the data being the same as the valid hash value; and based on determining that the hash document repository includes a corresponding entry, replacing the valid hash value in the document with the data of the document portion.

[0005] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes: a processor; and a memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the device to perform actions including: selecting a portion to be encoded from a document, the document including structured portions, each structured portion including an identifier and data associated with the identifier; obtaining at least one structured portion of the portion to be encoded; performing the following operations for a corresponding structured portion of the at least one structured portion to encode the document: generating a hash value of the data of the corresponding structured portion; determining whether a hash value repository includes the hash value; and, based on determining that the hash value repository includes the hash value, replacing the data of the corresponding structured portion in the document with the hash value; and sending the encoded document to a recipient.

[0006] In a fourth aspect of this disclosure, an electronic device is provided. The electronic device includes: a processor; and a memory coupled to the processor, the memory having instructions stored therein, the instructions causing the device to perform actions when executed by the processor, the actions including: receiving a transmitted document from a sender, the document including structured portions, each structured portion including an identifier and data associated with the identifier; selecting a portion to be decoded from the document; obtaining at least one structured portion of the portion to be decoded; and performing the following operations for a corresponding structured portion of the at least one structured portion: determining whether the corresponding structured portion includes a valid hash value; and based on determining that the corresponding structured portion includes a valid hash value: determining whether a hash document repository includes a corresponding entry for the valid hash value, the corresponding entry storing data of the document portion and a hash value of the data, the hash value of the data being the same as the valid hash value; and based on determining that the hash document repository includes a corresponding entry, replacing the valid hash value in the document with the data of the document portion.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored on a computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to perform a method according to a first aspect or a second aspect of this disclosure.

[0008] Please note that this summary is provided to present a simplified version of the chosen concepts, which will be further described in the detailed embodiments below. This summary is not intended to identify key or principal features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0009] The above and other objects, features, and advantages of this disclosure will become more apparent from a more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein:

[0010] Figure 1 A schematic diagram of an example environment 100 in which various embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 An example method 200 for transmitting documents according to some embodiments of the present disclosure is shown;

[0012] Figure 3 An example method 300 for processing a received transmitted document according to some embodiments of the present disclosure is shown;

[0013] Figure 4 An example architecture 400 for transmitting documents according to some embodiments of this disclosure is shown;

[0014] Figure 5 An example architecture 500 for determining multiple field filters according to some embodiments of the present disclosure is shown, the field filters indicating one or more identifiers of one or more predefined structured portions; and

[0015] Figure 6 A schematic block diagram of a device that can be used to implement embodiments of the present disclosure is shown.

[0016] In all the accompanying drawings, the same or similar reference numerals denote the same or similar elements. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure may be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure and are not intended to limit the scope of protection of this disclosure.

[0018] The term "comprising" and its variations as used herein are open-ended inclusion, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment". Definitions of other terms will be given in the description below.

[0019] Collecting telemetry reports from customer devices is a crucial way for equipment or service providers to monitor and manage customer devices and their applications from the backend. These reports reflect the relevant identifiers and various statuses of customer devices and their applications. As device applications evolve, more and more data needs to be collected. In traditional solutions, customer devices typically send documents containing structured data (e.g., using JSON format) to the backend, containing the types of data required for backend monitoring and management of customer devices. These documents are usually large and need to be sent at short intervals, thus putting significant pressure on network bandwidth, especially the uplink bandwidth of customer devices. For example, if each telemetry report needs to be 10MB in size and the customer device sends a report every hour, then the customer device needs to send 240MB of data per day for status monitoring. For the backend, receiving 1000 reports from customer devices per day would generate 240GB of data.

[0020] While reducing the amount of data sent to the backend might decrease, it could prevent the backend from obtaining comprehensive enough data for effective management of client devices. On the other hand, reducing the frequency of these telemetry reports could reduce the timeliness of the reports received by the backend, hindering its ability to promptly obtain and respond to the latest changes in client devices.

[0021] Since telemetry reports of the same type often have similar structures and contain some of the same content, other methods allow the client equipment to send only the parts of each report that differ from the last one. However, this requires the client-side equipment to "remember" the content of its last sent report and compare each value in the report with the previous report, which consumes a significant amount of storage space and computing resources on the client side.

[0022] To at least partially address the aforementioned problems and other potential issues, embodiments of this disclosure propose a scheme for transmitting documents. This scheme preserves frequently changing structured portions of a document being sent, and processes these portions as one or more structured portions are sent and received. At the sender, the scheme determines whether a portion has been sent to the receiver by determining whether a hash value repository contains the hash value of the data for that portion. If so, the portion in the document is replaced with that hash value before sending the document. If not, the portion is preserved in the sent document, and the hash value is stored in the hash value repository. At the receiver, the scheme retrieves the corresponding one or more structured portions from the received document. If a structured portion does not contain a valid hash value, the hash value of the data for that portion and the data are stored as an entry in a hash document repository. If a structured portion contains a valid hash value, the hash value in the document is replaced with the data from an entry in the hash document repository that has the same hash value, thereby restoring the document. If the hash document repository does not contain a corresponding entry for a valid hash value, the receiver sends a notification to the sender that the document needs to be resent.

[0023] By employing the above method, the sender can deduplicate the transmitted document content and avoid repeatedly sending known portions of the document to the receiver. This significantly reduces the actual size of the transmitted document while ensuring the receiver receives the same data content, thus saving time and bandwidth resources required for document transmission. Since the sender only needs to calculate and store hash values, this solution does not impose a significant additional load on senders with limited capacity and is easily portable to multiple devices. On the other hand, at the receiver, which typically has ample storage and computing resources, portions from a large number of historical documents can be stored, eliminating the need for these portions to be retransmitted by the sender. Compared to methods that only compare limited reports on client devices, this solution achieves a higher deduplication rate. This solution will be described below in the context of client device management. However, it should be understood that this solution is also applicable to other scenarios where documents, including structured portions, need to be frequently transmitted to the receiver.

[0024] Figure 1 A schematic diagram of an example environment 100 in which various embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, example environment 100 includes a sender 110 and a receiver 120 for document 130. For example, sender 110 could be a customer device sending telemetry reports, while receiver 120 could be a service provider backend for managing and monitoring customer devices. Sender 110 and receiver 120 could also be any fixed or mobile device.

[0025] Sender 110 may send multiple documents to receiver 120. For example, a client device may send telemetry reports reflecting its status to a backend at a certain frequency. At some point, sender 110 may need to send document 130 to receiver 120. Document 130 may contain structured data (e.g., JSON formatted data). Sender 110 may process the content of document 130 and send the processed document 130 to receiver 120. After receiver 120 receives the processed document 130, it may perform various operations on the processed document 130 (e.g., parsing, data extraction, and analysis), and / or send the processed document 130 to downstream applications or devices. Furthermore, receiver 120 may also send information directly or indirectly to sender 110, such as the time interval at which telemetry reports should be sent, and aspects that the data in the document should include.

[0026] It should be understood that the architecture and functionality in example environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. Sender 110 and receiver 120 can be a single device, distributed devices, or devices existing in any other suitable form. Furthermore, other devices, systems, or components, etc., not shown, may also be present in example environment 100. For example, receiver 120 may receive documents from multiple different senders. Additionally, embodiments of this disclosure can be applied to other environments with different structures and / or functionalities.

[0027] Figure 2 A flowchart of an example method 200 for document transfer according to some embodiments of the present disclosure is shown. Example method 200 can be, for example, by... Figure 1 The sender 110 shown performs this action. It should be understood that method 200 may also include additional actions not shown, and the scope of this disclosure is not limited in this respect. The following is in conjunction with... Figure 1 Let me describe method 200 in detail.

[0028] At box 210, sender 110 selects a portion to be encoded from document 130, which includes structured portions, each of which includes an identifier and data associated with the identifier. As an example, document 130 may be a structured document in a format such as JSON or XML, or otherwise include one or more identifiable structured portions. In some embodiments, document 130 may be a telemetry report about the status of a customer device, which may be sent by the customer device to a service backend that monitors and manages the customer device.

[0029] In some embodiments, sender 110 may obtain one or more identifiers for one or more predefined structured portions. Then, based on these identifiers, sender 110 selects a portion of document 130 excluding the one or more predefined structured portions as the portion to be encoded. For example, in JSON format, sender 110 may obtain the keys of one or more JSON fields (also called entries) and select the portion of 130 excluding the fields corresponding to those keys as the encoded portion. Such identifiers may be obtained by sender 110 from receiver 120 or a third party when document 130 needs to be sent, may be pre-stored locally by sender 110 and read by sender 110 when needed, or may be obtained in any appropriate manner.

[0030] In some of these embodiments, one or more predefined structured portions may be determined based on the frequency with which the structured portions change in a plurality of documents that include at least partially the same structure as document 130 (e.g., having partially the same fields and / or hierarchical structure).

[0031] At box 220, sender 110 obtains at least one structured portion of the part to be encoded. For example, sender 110 may obtain the at least one structured portion based on an identifier of the structured portion or its relative position within the entire document. In some embodiments, document 130 has a tree-like hierarchical structure (e.g., JSON format). In such embodiments, sender 110 may obtain a predefined encoding / decoding hierarchy and use a structured partition of the encoded portion that is located within the predefined encoding / decoding hierarchy (e.g., a field in a JSON document at a certain level) as at least one structured portion. Sender 110 may use a method similar to that used to obtain the identifier of the predefined structured portion to obtain the encoding / decoding hierarchy.

[0032] At block 230, sender 110 performs the following operations for a corresponding structured portion of at least one structured portion to encode a document: generating a hash value for the data of the corresponding structured portion; determining whether a hash value repository includes the hash value; and, based on the determination that the hash value repository includes the hash value, replacing the data of the corresponding structured portion in the document with the hash value. The hash value repository may, for example, be located locally on the client device. For example, sender 110 may use a predetermined hash algorithm to generate the hash value such that generating hash values ​​for the same data yields the same result. In some embodiments, based on the determination that the hash value repository does not include the hash value, sender 110 stores the hash value in the hash value repository.

[0033] After the structured portion requiring encoding has been encoded as described above, at box 240, sender 110 sends the encoded document 130 to receiver 120. It should be understood that sender 110 may continue to send more documents to receiver 120 using method 200.

[0034] In some embodiments, the sender 110 may also operate based on a notification received from the receiver 120 indicating that the transmitted encoded document 130 includes at least one hash value that the receiver 120 cannot process. In response to receiving the notification, the sender 110 may delete the at least one hash value from the hash value repository. After deleting the at least one hash value from the hash value repository, the sender 110 may also re-encode the document 130 in the same manner as described above, and send the re-encoded document 130 to the receiver. In this way, data loss due to encoding errors or changes in data known to the receiver can be avoided.

[0035] Figure 3 Methods for processing received transmitted documents according to some embodiments of the present disclosure are shown; example method 300 can be, for example, by... Figure 1 The recipient 120 shown performs this action. It should be understood that method 300 may also include additional actions not shown, and the scope of this disclosure is not limited in this respect. The following is in conjunction with... Figure 1 Let me describe method 200 in detail.

[0036] At box 310, receiver 120 receives a transmitted document 130 from sender 110. This document includes structured portions, each of which includes an identifier and associated data. As an example, document 130 may be a structured document in a format such as JSON or XML, or otherwise include one or more identifiable structured portions. For example, document 130 may be an encoded document sent by sender 110 to receiver 120 according to method 200. In some embodiments, document 130 may be a telemetry report regarding the status of a client device. A service backend monitoring and managing the client device can receive the telemetry report from the client device.

[0037] At box 320, receiver 120 selects the portion to be decoded from document 130. In some embodiments, receiver 120 may obtain one or more identifiers (such as keys for JSON fields) for one or more predefined structured portions. Receiver 120 then selects a portion of document 130 other than the one or more predefined structured portions as the portion to be decoded based on these identifiers. These identifiers may be, for example, predetermined and / or stored by receiver 120 and retrieved when needed. In such an example, receiver 120 may provide the identifier to sender 110 for use (e.g., as an identifier for a predefined structured portion in method 200). Receiver 120 may also obtain the identifier in other suitable ways or from a third party.

[0038] In some of these embodiments, similar to the determination of predefined structured portions in method 200, one or more predefined structured portions can be determined based on the frequency with which structured portions change across multiple documents.

[0039] At box 330, receiver 120 obtains at least one structured portion of the part to be decoded. For example, receiver 120 may obtain the at least one structured portion based on an identifier of the structured portion or its relative position within the entire document. In some embodiments, document 130 has a tree-like hierarchical structure (e.g., JSON format). In such embodiments, receiver 120 may obtain a predefined encoding / decoding hierarchy and consider a structured partition (a field in a JSON document at a certain level) of the encoded portion within the predefined encoding / decoding hierarchy as at least one structured portion. Receiver 120 may use a method similar to that used to obtain the identifier of the predefined structured portion to obtain the encoding / decoding hierarchy.

[0040] At box 340, receiver 120 performs the following operations for a corresponding structured portion of the at least one structured portion. First, receiver 120 determines whether the corresponding structured portion includes a valid hash value. Based on the determination that the corresponding structured portion includes a valid hash value, receiver 120 determines whether a hash document repository includes a corresponding entry for the valid hash value. The hash document repository may, for example, be located locally on receiver 120. The corresponding entry stores the data of the document portion and the hash value of that data, and the hash value of the data is the same as the valid hash value. This means that the data represented by the valid hash value is the same as the data in the corresponding entry. Then, based on the determination that the hash document repository includes the corresponding entry, receiver 120 replaces the valid hash value in the document with the data of the document portion.

[0041] In some embodiments, based on the determination that the corresponding structured portion does not contain a valid hash value, the receiver 120 may generate a hash value for the data of the corresponding structured portion and add an entry to a hash document repository that stores the generated hash value and the data of the corresponding structured portion. For example, the receiver 120 may use a predetermined hash algorithm to generate the hash value, such that the same result is obtained when generating hash values ​​for the same data.

[0042] After the receiver 120 performs the above operations on all the corresponding structured parts, the valid hash value in document 130 is restored to the data in the original document, and the receiver 120 thus obtains the complete original data in document 130. It should be understood that the receiver 120 can also continue to use method 300 to process more documents received from the sender 110.

[0043] In some embodiments, based on the determination that the hash document repository does not contain a corresponding entry for at least one determined valid hash value, the receiver 120 may also send a notification to the sender 110 indicating that the document includes the at least one valid hash value that cannot be processed. Through this operation, the receiver 120 can notify the sender of data that should be retransmitted, thereby avoiding data loss due to the inability to recover the original data content.

[0044] In some embodiments, method 300 may be used in conjunction with method 200 and performed separately by the receiver and the sender. In such embodiments, one or more predefined structured portions obtained by the sender 110 in method 200 may be the same as one or more predefined structured portions obtained by the receiver 120 in method 300. Furthermore, at least one structured portion encoded by the sender 110 in method 200 may be the same as at least one predefined structured portion decoded by the receiver 120 in method 300. For example, in an embodiment where document 130 has a hierarchical structure, the predefined encoded hierarchies obtained by the sender 110 and the receiver 120 in methods 200 and 300 respectively may be the same. This ensures that the sender 110 and the receiver 120 perform hash value generation and content deduplication for document 130 with the same granularity. Furthermore, the sender 110 and the receiver 120 may use the same hash algorithm to calculate hash values, ensuring that both parties obtain the same hash value when calculating hash values ​​for the same data. The hash algorithm can be obtained by both parties in any appropriate manner, and the scope of this disclosure is not limited thereto.

[0045] Figure 4Example architecture 400 according to some embodiments of the present disclosure is illustrated, showing an example implementation of sender 110 and an example implementation of receiver 120, namely sender 410 and receiver 420. In example architecture 400, sender 410 may be a client device, while receiver 420 may be a backend of a device or service provider. As shown by arrow 460, sender 410 may send various data to receiver 420, such as documents reflecting the status of the client device, like telemetry reports. It should be understood that the various devices, components, and their distribution in architecture 400 are merely examples, and other suitable devices, components, and their distributions capable of performing the schemes described in this disclosure may also be used.

[0046] Document 430 can be Figure 1 Document 130 in the table can be a telemetry report generated by monitoring, statistics, or related applications on the customer device, such as the example first JSON document shown in Table 1 and the example second JSON document shown in Table 3. The documents in Tables 1 and 3 will be used below as two examples of document 430 to describe architecture 400. For clarity, some parts of these JSON documents are not shown in the tables and are indicated by ellipses.

[0047] Table 1: Example First JSON Document

[0048]

[0049] In this example implementation, the predefined value 440 is shown as separate from the sender 410 and receiver 420. For example, the predefined value 440 may be stored in a third party (not shown) capable of communicating with the sender 410 and / or receiver 420. In some other examples, the encoding engine 415 of the sender 410 may also obtain the predefined value 440 from the receiver 420. The predefined value 440 may include, for example, identifiers of one or more predefined structured portions, and, where the document 430 has a hierarchical structure, codec levels. In the following example description in conjunction with the documents in Tables 1 and 3, for illustrative and not restrictive purposes, it is assumed that the predefined value 440 includes the identifier "applianceInfo.upTimeInDays" of one or more predefined structured portions and the codec level "1", and this remains unchanged throughout the example.

[0050] First, sender 410 sends the first JSON document from Table 1 to receiver 420. Based on the above values, sender 410's encoding engine 415 selects the portion of the first JSON document excluding the field "upTimeInDays" as the part to be encoded. Further, encoding engine 415 obtains the first-level partitions of the part to be encoded, namely "applianceInfo" and "licensing," as the structured portion to be encoded.

[0051] Then, encoding engine 415 generates hash values ​​for the data in the first JSON document excluding "applianceInfo" as follows:

[0052] 5c41329ec6b9328cf8efb5112540f96b

[0053] The hash values ​​generated for the data in "licensing" are as follows:

[0054] b1073d8b27c705f4045dd13b0f3e6eea

[0055] Encoding engine 415 then checks if the generated hash value is included in hash value repository 411. Assuming that hash value repository 411 does not yet store any hash value, encoding engine 415 does not replace any part of the first JSON document. Therefore, the encoded first JSON document remains as shown in Table 1. Sender 410 then sends this document to receiver 420.

[0056] Receiver 420 receives and decodes the first encoded JSON document as shown in Table 1. As previously mentioned, receiver 420's decoding engine 425 can read predefined values ​​440, for example, from local storage or from a third party. In this example, predefined values ​​440 include the identifier "applianceInfo.upTimeInDays" of one or more predefined structured parts and an encoding / decoding level of "1".

[0057] Based on the above values, the decoding engine 425 takes the portion of the first JSON document excluding the "upTimeInDays" field as the part to be decoded, and obtains the first-level partitions "applianceInfo" and "licensing" from the part to be decoded as the structured parts to be decoded. Then, the decoding engine 425 determines whether these structured parts include valid hash values.

[0058] Based on the determination that the corresponding part does not contain a valid hash value, the decoding engine 425 generates the following hash values ​​for the data in the "applianceInfo" section of the first JSON document, excluding "upTimeInDays":

[0059] 5c41329ec6b9328cf8efb5112540f96b

[0060] The hash values ​​generated for the data in "licensing" are as follows:

[0061] b1073d8b27c705f4045dd13b0f3e6eea

[0062] Assuming that no entries are currently stored in the hash document repository 421, the decoding engine 425 stores the aforementioned hash value and its corresponding data in the entries of the hash document repository 421, as shown in Table 2. At this point, the decoding engine 425 has completed processing the encoded first JSON document and sends the processed document to downstream applications or devices such as the document repository 450. The document repository 450 could be, for example, a report data lake storing historical telemetry reports received in the background.

[0063] Table 2: Hash document entries based on the first JSON document storage

[0064]

[0065] In this example, after sender 410 processes and sends the first example JSON document, for example, after a predetermined time interval or in response to some trigger, sender 410 will send the second example JSON document as shown in Table 3 to the receiver. Here, encoding engine 415 generates a hash value for the data in the "applianceInfo" field of the second JSON document, excluding "upTimeInDays", as follows, and determines whether the hash value is already contained in hash value repository 411:

[0066] 5c41329ec6b9328cf8efb5112540f96b

[0067] Table 3: Example Second JSON Document

[0068]

[0069] In this example, the hash value is already included in the hash value repository 411. This indicates that the same data as the corresponding part is already known to the receiver 420. In this example, this is because the same data was previously sent to the receiver 420 in the first JSON document. Therefore, the encoding engine 415 replaces the data of the corresponding part in the second example JSON document (i.e., the data in "applianceInfo" excluding "upTimeInDays") with the hash value without sending the complete data of that part to the receiver 420 again.

[0070] Encoding engine 415 generates the following hash value for the data in the "licensing" section of the second JSON document, and determines whether the hash value is already contained in the hash value repository:

[0071] b0f607e28a886b05ca9ae0dadbbc1cb4

[0072] In this example, the hash value is not included in the hash value repository 411. Therefore, the sender 410 does not use the hash value to replace the corresponding part of the data, and stores the hash value in the hash value repository 411.

[0073] At this point, the encoding engine 415 has completed processing the structured portion of the second JSON document to be encoded and obtained the encoded second JSON document, as shown in Table 4. Then, the sender 410 sends the encoded second JSON document to the receiver 420. It should be understood that the sender 410 can continue to send more documents to the receiver 420 in a similar manner.

[0074] It should be noted that in this example, "upTimeInDays" is excluded from the document's encoding section as a predefined structured part. Therefore, regardless of whether this part's data has been previously sent to recipient 420, it will be retained in the sent document.

[0075] Receiver 420 receives the encoded sample second JSON document from sender 410. Here, receiver 420's decoding engine 425 determines that the document's "applianceInfo" contains a valid hash value, 5c41329ec6b9328cf8efb5112540f96b. Therefore, decoding engine 425 will determine whether this hash value is included in the hash document repository 421.

[0076] Table 4: Encoded Example Second JSON Document

[0077]

[0078] In this example, the hash value has already been stored in the hash document repository 421 when the receiver 420 receives the first JSON document. Therefore, the decoding engine 425 replaces the hash value in the encoded second example JSON document with the data stored in the corresponding entry, thereby recovering the original data of the corresponding part.

[0079] The decoding engine 425 also determines that the “licensing” section of the encoded second JSON document does not contain a valid hash value. Based on this determination, the receiver generates a hash value for this part of the data and stores the hash value and the corresponding data in the hash document repository 421, as shown in Table 5.

[0080] Table 5: Hash document entries based on the second JSON document storage

[0081]

[0082] At this point, the decoding engine 425 has completed processing the structured portion of the encoded second JSON document to be decoded and obtained the decoded second JSON document, as shown in Table 6. This document is consistent with... Figure 2 The second JSON document shown contains the same data. It should be understood that receiver 120 can continue to process more received documents in a similar manner.

[0083] Table 6: Decoded Example Second JSON Document

[0084]

[0085] It should be noted that the file formats and structures, identifiers, encoding / decoding levels, and associated values ​​in the aforementioned examples are merely examples; other suitable file formats and structures, fields, and encoding / decoding levels can also be used.

[0086] By using the above method, the size of the document to be sent can be significantly reduced (more than 70% for some documents of the same type as the JSON document in the previous example) without imposing a significant additional load on the sender and ensuring that the receiver receives the same data, thereby saving network resources.

[0087] Due to reasons such as updates to the hash document repository 421 or encoding errors at the sender 410, the document received by the receiver 420 from the sender 410 may include at least one valid hash value for which there is no corresponding entry in the hash document repository 421. In this case, as part of the error handling mechanism, the receiver 420 may send a notification to the sender, as indicated by arrow 470. This notification indicates that the transmitted encoded document 430 includes the at least one valid hash value that cannot be processed by the receiver. In response to receiving this notification, the receiver 420 may remove the at least one valid value from the hash value repository 411. After this, the sender 410 may re-encode and send the document 430 to the receiver, as indicated by arrow 460.

[0088] In some embodiments, receiver 420 may receive multiple structured documents of different types from multiple senders, and different predefined values ​​440 are defined for different document types. Decoding engine 425 can then retrieve the corresponding predefined value 440 for a specific document type. Similarly, sender 410 can also retrieve the corresponding predefined value 440 for a specific document type. For example, field filters can be defined for each type of telemetry report sent from a client device to the backend, each field filter including identifiers (such as JSON field keys) of one or more predefined structured portions. Field filters can be determined based on analysis of a series of reports of the corresponding type.

[0089] Figure 5 An example architecture 500 for determining multiple field filters according to some embodiments of this disclosure is shown. Architecture 500 may be, for example, part of receiver 120 or receiver 420, and sender 110 or sender 410 may acquire the determined field filters in an appropriate manner. It should be understood that the components for determining the field filters may also be distributed in other ways.

[0090] exist Figure 5 In this context, the field analyzer component 501 can read a series of documents of the same type (such as the latest N telemetry reports of a specific type) from the document repository 550 and analyze these documents. The document repository 550 can be, for example, Figure 4The document repository 450 is used. Based on this analysis, the field analyzer component 501 can define field filters for that type or update corresponding defined field filters. For example, the field analyzer component 501 can read the latest N reports of the same type, count the number of times fields have changed in those N reports, and include fields whose number of changes exceeds a threshold in the field filters. Field filters can also be determined based on more complex analysis, and the scheme of this disclosure is not limited thereto. The multiple defined field identifiers 502-1 to 502-N can then be used by corresponding components 503-1 to 503-N, where each field identifier is for a different document type. For example, components 503-1 to 503-N can be, for example, components for the transfer of documents of the corresponding type according to embodiments of this disclosure, or other document management components.

[0091] Figure 6 A schematic block diagram of a device 600 that can be used to implement embodiments of the present disclosure is shown. Device 600 may be the device or apparatus described in the embodiments of the present disclosure, such as senders 110, 410 and receivers 120, 420, etc. Figure 6 As shown, device 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 602 or loaded from storage unit 608 into random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of device 600. The CPU 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604. Although not shown in... Figure 6 As shown, device 600 may also include a coprocessor.

[0092] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0093] The various methods or processes described above can be executed by processing unit 601. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by CPU 601, one or more steps or actions in the methods or processes described above can be performed.

[0094] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0095] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0096] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media within the respective computing / processing device.

[0097] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may execute entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0098] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0099] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0100] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0101] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for document transfer, comprising: Selecting a portion to be encoded from a document, the document including structured portions, each of the structured portions including an identifier and data associated with the identifier, wherein selecting the portion to be encoded from the document includes: obtaining one or more identifiers of one or more predefined structured portions, and selecting a portion of the document other than the one or more predefined structured portions as the portion to be encoded based on the one or more identifiers; Obtain at least one structured portion of the part to be encoded; Perform the following operations on a corresponding structured portion of the at least one structured portion to encode the document: Generate the hash value of the data for the corresponding structured portion; Determine whether the hash value repository includes the hash value; and Based on the determination that the hash value repository includes the hash value, the data in the corresponding structured portion of the document is replaced with the hash value; and The encoded document is sent to the recipient.

2. The method according to claim 1, wherein the operation further comprises: Based on the determination that the hash value repository does not include the hash value, the hash value is stored in the hash value repository.

3. The method of claim 1, wherein the one or more predefined structured portions are determined based on the frequency of changes in the structured portions of a plurality of documents, wherein the documents in the plurality of documents have at least partially the same structure as the document.

4. The method of claim 1, wherein the document has a tree-like hierarchical structure, and obtaining the at least one structured portion comprises: Obtain the predefined encoding / decoding layer; as well as The structured partition of the part to be encoded, which is located at the predefined encoding / decoding level, is taken as the at least one structured part.

5. The method of claim 1, further comprising performing the following operations in response to receiving a notification from the recipient, the notification indicating that the transmitted encoded document includes at least one hash value that the recipient cannot process: Delete the at least one hash value from the hash value repository; After deleting the at least one hash value from the hash value repository, the document is re-encoded; and The re-encoded document is sent to the recipient.

6. The method of claim 1, wherein the document is a telemetry report on the status of the customer equipment.

7. A method for document transfer, comprising: Receive a transmitted document from the sender, the document including a structured portion, each of the structured portions including an identifier and data associated with the identifier; Select the portion of the document to be decoded; Obtain at least one structured portion of the part to be decoded; Perform the following operations on the corresponding structured portion of the at least one structured portion: Determine whether the corresponding structured portion includes a valid hash value; and Based on the determination that the corresponding structured portion includes a valid hash value: Determine whether the hash document repository includes a corresponding entry for the valid hash value, the corresponding entry storing data of a portion of the document and the hash value of the data, wherein the hash value of the data is the same as the valid hash value; and Based on the determination that the hash document repository includes the corresponding entry, the valid hash value in the document is replaced with the data in the document portion.

8. The method of claim 7, wherein the operation further comprises: Based on the determination that the corresponding structured portion does not include a valid hash value: Generate the hash value of the data for the corresponding structured portion; as well as Add an entry to the hash document repository, the entry storing the generated hash value and the data of the corresponding structured portion.

9. The method of claim 7, wherein selecting the portion to be decoded from the document comprises: Retrieves one or more identifiers of one or more predefined structured parts; as well as Based on the one or more identifiers, a portion of the document other than the one or more predefined structured portions is selected as the portion to be decoded.

10. The method of claim 9, wherein the one or more predefined structured portions are determined based on the frequency of changes in the structured portions of a plurality of documents, the documents of the plurality of documents having at least partially the same structure as the document.

11. The method of claim 7, wherein the document has a tree-like hierarchical structure, and obtaining the at least one structured portion comprises: Obtain the predefined encoding / decoding layer; as well as The structured partition of the part to be decoded, which is located at the predefined encoding / decoding level, is taken as the at least one structured part.

12. The method of claim 7, further comprising: In response to determining that the hash document repository does not contain a corresponding entry for at least one determined valid hash value, a notification is sent to the sender indicating that the document contains at least one valid hash value that cannot be processed.

13. The method of claim 7, wherein the document is a telemetry report on the status of the customer equipment.

14. A method for document transfer, comprising: Select the portion to be encoded from the document, the document including structured portions, each of the structured portions including an identifier and data associated with the identifier; Obtain at least one structured portion of the part to be encoded; Perform the following operations on a corresponding structured portion of the at least one structured portion to encode the document: Generate the hash value of the data for the corresponding structured portion; Determine whether the hash value repository includes the hash value; as well as Based on the determination that the hash value repository includes the hash value, the data in the corresponding structured portion of the document is replaced with the hash value; Send the encoded document to the recipient; and In response to receiving a notification from the recipient, the following actions are performed: the notification indicates that the encoded document sent includes at least one hash value that the recipient cannot process. Delete the at least one hash value from the hash value repository; After deleting the at least one hash value from the hash value repository, the document is re-encoded; and The re-encoded document is sent to the recipient.

15. The method of claim 14, further comprising: Based on the determination that the hash value repository does not include the hash value, the hash value is stored in the hash value repository.

16. The method of claim 14, wherein selecting the portion to be encoded from the document comprises: Retrieves one or more identifiers of one or more predefined structured parts; as well as Based on the one or more identifiers, the portion of the document other than the one or more predefined structured portions is selected as the portion to be encoded.

17. The method of claim 16, wherein the one or more predefined structured portions are determined based on the frequency of changes in the structured portions of a plurality of documents, the documents of the plurality of documents having at least partially the same structure as the document.

18. The method of claim 14, wherein the document has a tree-like hierarchical structure, and obtaining the at least one structured portion comprises: Obtain the predefined encoding / decoding layer; as well as The structured partition of the part to be encoded, which is located at the predefined encoding / decoding level, is taken as the at least one structured part.

19. The method of claim 14, wherein the document is a telemetry report on the status of the customer equipment.

20. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the device to perform the method according to any one of claims 1 to 6.

21. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, the instructions causing the device to perform actions when executed by the processor, the actions including: Receive a transmitted document from the sender, the document including a structured portion, each of the structured portions including an identifier and data associated with the identifier; Select the portion of the document to be decoded; Obtain at least one structured portion of the part to be decoded; Perform the following operations on the corresponding structured portion of the at least one structured portion: Determine whether the corresponding structured portion includes a valid hash value; and Based on the determination that the corresponding structured portion includes a valid hash value: Determine whether the hash document repository includes a corresponding entry for the valid hash value, the corresponding entry storing data of a portion of the document and the hash value of the data, wherein the hash value of the data is the same as the valid hash value; and Based on the determination that the hash document repository includes the corresponding entry, the valid hash value in the document is replaced with the data in the document portion.

22. The apparatus of claim 21, wherein the operation further comprises: Based on the determination that the corresponding structured portion does not include a valid hash value: Generate the hash value of the data for the corresponding structured portion; as well as Add an entry to the hash document repository, the entry storing the generated hash value and the data of the corresponding structured portion.

23. The device of claim 21, wherein selecting the portion to be decoded from the document comprises: Retrieves one or more identifiers of one or more predefined structured parts; as well as Based on the one or more identifiers, a portion of the document other than the one or more predefined structured portions is selected as the portion to be decoded.

24. The device of claim 23, wherein the one or more predefined structured portions are determined based on the frequency of changes in the structured portions of a plurality of documents, the documents of the plurality of documents having at least partially the same structure as the document.

25. The device of claim 21, wherein the document has a tree-like hierarchical structure, and obtaining the at least one structured portion comprises: Obtain the predefined encoding / decoding layer; as well as The structured partition of the part to be decoded, which is located at the predefined encoding / decoding level, is taken as the at least one structured part.

26. The device of claim 21, wherein the action further comprises: In response to determining that the hash document repository does not contain a corresponding entry for at least one determined valid hash value, a notification is sent to the sender indicating that the document contains at least one valid hash value that cannot be processed.

27. The device of claim 21, wherein the document is a telemetry report on the status of the customer device.

28. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the device to perform the method according to any one of claims 14 to 19.

29. A computer program product tangibly stored on a computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of claims 1 to 6, 7 to 13, or 14 to 19.

Images