Data item delimiting method, apparatus and device for data transmission protocol, and storage medium

Abstract

This specification provides a method, apparatus, device, and storage medium for data item delimitation in a data transmission protocol, applied to a data receiving end. It receives data sent by a data sending end, buffers the received data in a first buffer, determines the first byte of the buffered data as the target byte, and iteratively executes the process until the data buffered in the first buffer is fully detected. Starting from the target byte, it detects delimiter strings that divide different data items. If a delimiter string is detected, it determines that the data after the target byte and before the first detected delimiter string is a data item, copies the detected data item to a second buffer, and determines the first byte of the data after the first detected delimiter string as the target byte. The solution provided in this specification can accurately delimit data items and improve data parsing accuracy when data is transmitted in mixed data transmission protocols.

Description

Technical Field

[0001] This specification relates to the field of network communication technology, and in particular to a data item delimitation method, apparatus, device, and storage medium for a data transmission protocol. Background Technology

[0002] Under a data transmission protocol, each data received by the data receiver (such as a host) contains one or more data items, corresponding to one or more data types. When multiple data items of different data types are transmitted together, the transmitted content is a byte stream, and the data receiver parses each data item from the received data stream. In some possible embodiments, since the length of each data item is unpredictable, a separator string is needed as the boundary between different data items.

[0003] For example, when transmitting form data and file data simultaneously via HTTP, form data items, being shorter, can be entirely cached in a memory buffer. However, file data items may be longer, requiring multiple delimiters to receive the complete data. Given limited memory, it's inconvenient to continuously cache the file data being received in memory until completion, especially when the file data length is unpredictable. Therefore, in related technologies, when transmitting form data and file data simultaneously, form data is typically transmitted first, followed by file data. Correspondingly, the receiving end will also default to parsing the form data first, then the file data. That is, if the receiving end parses the file data, it's assumed that the form data has been completely parsed, and the remaining data is file data.

[0004] In this scenario, if file data needs to be transmitted first, followed by form data, the receiving end may mistakenly assume that the entire transmission consists of file data, leading to errors in parsing the form data items. Summary of the Invention

[0005] This application provides a data item delimiting method for a data transmission protocol used to transmit data between a data sender and a data receiver; the method is applied to the data receiver; the data received by the data receiver contains one or more data items; the multiple data items correspond to one or more data types; the data length of each data item in the multiple data items is not fixed and is unpredictable; the data receiver is configured with a first buffer for delimiting each data item contained in the received data, and a second buffer for parsing each data item from the received data; the method includes:

[0006] The system receives data sent by the data sender, caches the received data in the first buffer, determines the first byte of the data cached in the first buffer as the target byte, and iteratively executes the following steps until the data cached in the first buffer has been completely detected:

[0007] Starting from the target byte, detect the separator string used to divide different data items;

[0008] If the separator string is detected, the data that includes the target byte and is located after the target byte and before the first detected separator string is determined to be a data item, and the detected data item is copied to the second buffer;

[0009] The first byte of the data following the first detected delimiter string is determined as the target byte.

[0010] Optionally, the method further includes:

[0011] The step of receiving data sent by the data sending end and buffering it in the first buffer includes:

[0012] The received data is cached in the first buffer, the first cursor is pointed to the memory address corresponding to the last byte of the data cached in the first buffer, and the second cursor is pointed to the memory address corresponding to the next byte of the data cached in the first buffer.

[0013] The first cursor is used to indicate the memory address corresponding to the last byte of the data that has been detected and cached in the first buffer; the second cursor is used to indicate the memory address corresponding to the first byte of the data that has not been detected and cached in the first buffer.

[0014] Optionally, the received data includes request messages or response messages;

[0015] Before determining the first byte of the data cached in the first buffer as the target byte, the method further includes:

[0016] Obtain the separator string contained in the request header of the request message; or, obtain the separator string contained in the response header of the response message.

[0017] Optionally, detecting the separator string used to divide different data items, starting from the target byte, includes:

[0018] Detect whether the data before the memory address corresponding to the byte pointed to by the second cursor in the first buffer contains the separator string; if the separator string is contained, point the first cursor to the memory address corresponding to the first byte of the detected first separator string, and divide the data before the memory address pointed to by the first cursor into data items.

[0019] Optionally, after copying the detected data items to the second buffer, the method further includes:

[0020] The data that has not been detected in the memory space where the first buffer is located is overwritten and written to the memory address corresponding to the first byte of the memory space where the first buffer is located.

[0021] Optionally, starting from the target byte, after detecting the separator string used to divide different data items, the method further includes:

[0022] If the separator string is not detected, the first cursor is pointed to the memory address corresponding to the last byte of the separator string, and it is checked whether the number of bytes between the byte pointed to by the first cursor and the byte pointed to by the second cursor is not less than the number of bytes occupied by the separator string;

[0023] If the number of bytes occupied by the separator string is not less than the number of bytes occupied by the separator string, copy the data between the memory address of the byte pointed to by the first cursor and the memory address pointed to by the second cursor offset forward by a total of the number of bytes occupied by the separator string minus one byte to the second buffer; then overwrite the remaining data in the first buffer that has not been copied to the second buffer to the memory address corresponding to the first byte of the memory space of the first buffer.

[0024] Optionally, detecting whether the number of bytes between the byte pointed to by the first cursor and the byte pointed to by the second cursor is not less than the number of bytes occupied by the delimiter string includes:

[0025] If the number of bytes is less than the number of bytes occupied by the separator string, the data between the memory address corresponding to the byte pointed to by the first cursor and the memory address corresponding to the byte pointed to by the second cursor is directly overwritten and written to the memory address corresponding to the first byte of the memory space where the first buffer is located.

[0026] Optionally, the data transmission protocol includes:

[0027] HTTP, HTTPS, POP3, and SMTP protocols, etc.

[0028] This application also provides a data item delimiting device for a data transmission protocol used to transmit data between a data sender and a data receiver; the device is applied to the data receiver; the data received by the data receiver includes one or more data items; the multiple data items correspond to one or more data types; the data length of each data item in the multiple data items is not fixed and is unpredictable; the data receiver is configured with a first buffer unit for delimiting each data item contained in the received data, and a second buffer unit for parsing each data item from the received data; the device includes:

[0029] The first buffer unit is used to receive and buffer the data sent by the data sender, determine the first byte of the data as the target byte, and iteratively execute the following steps until all the buffered data has been detected: starting from the target byte, detect the separator string used to divide different data items; if the separator string is detected, determine the data that includes the target byte and is located after the target byte and before the first detected separator string as a data item, and copy the detected data item to the second buffer unit; determine the first byte of the data located after the first detected separator string as the target byte;

[0030] The second buffer unit is used to receive the data items to be parsed sent by the first buffer unit, parse the data items, and provide them for use in business processes.

[0031] This application also provides an electronic device, including a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus;

[0032] The memory stores machine-readable instructions, and the processor executes the above method by invoking the machine-readable instructions.

[0033] This application also provides a machine-readable storage medium storing machine-readable instructions, which, when called and executed by a processor, implement the above-described method.

[0034] Through the above methods, on the one hand, by configuring a first buffer for the data receiving end, after pre-parsing the received data for data delimitation, the delimited data items can be copied to a second buffer for data parsing. This allows for simultaneous transmission of multiple data types, where longer data (such as file data) that is difficult to fully receive, delimit, and parse in the second buffer can be delimited and parsed together with shorter data (such as form data) that is easier to fully receive, delimit, and parse in memory, both within the memory space of the first buffer and in the memory space of the second buffer. When transmitting multiple data types simultaneously, regardless of the order in which the data items are transmitted, the receiving end can accurately delimit and parse the data items, improving the flexibility and reliability of data transmission. The technical solution of this application can be achieved as long as the length of the first buffer is greater than the length of the delimiter string, without needing to consider the possible length of individual data items. Therefore, the additional memory used by this application's technical solution can be very small, and the change in memory usage during program execution is negligible.

[0035] On the other hand, copying the data before the delimiter string to the second buffer in the first buffer is equivalent to performing data item delimiting (i.e., pre-parsing) on ​​the data containing one or more data types in the first buffer, and then copying the pre-parsed data items to the second buffer. The second buffer is used by the business process to parse the pre-parsed data items. This scheme can accurately delimit and parse each data item in the data stream received at the data receiving end, even when the length of each data item is unpredictable, thus improving the reliability of data transmission. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is an exemplary embodiment illustrating the content of data received by a data receiver from HTTP protocol transmission.

[0038] Figure 2 This is a schematic diagram illustrating the architecture of an HTTP data transmission system, as shown in an exemplary embodiment.

[0039] Figure 3 This is an exemplary embodiment illustrating a flowchart of a data item delimitation method for a data transmission protocol;

[0040] Figure 4 This is a schematic diagram illustrating a first buffer indicating the location of a target byte, as shown in an exemplary embodiment.

[0041] Figure 5 This is an exemplary embodiment illustrating the position indicated by the cursor when the first buffer of a data receiver has not yet received any data.

[0042] Figure 6 This is an exemplary embodiment illustrating the position indicated by the cursor when the first buffer of a data receiver receives data.

[0043] Figure 7 This is an exemplary embodiment illustrating a method of copying delimited data items from a first buffer to a second buffer;

[0044] Figure 8 This is an exemplary embodiment illustrating a scenario where no delimiter string is detected after the target byte in the first buffer;

[0045] Figure 9 This is an exemplary embodiment illustrating another case where no delimiter string is detected after the target byte in the first buffer;

[0046] Figure 10 This is an exemplary embodiment illustrating the hardware structure of an electronic device containing a data item delimiting device for a data transmission protocol;

[0047] Figure 11 This is a block diagram illustrating a data item delimiting device for a data transmission protocol, as shown in an exemplary embodiment. Detailed Implementation

[0048] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0049] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0050] Under a data transmission protocol, each data received by the data receiver (such as a host) contains one or more data items, corresponding to one or more data types. When multiple data items of different data types are transmitted together, the transmitted content is a byte stream, and the data receiver parses each data item from the received data stream. In some possible embodiments, since the length of each data item is unpredictable, a separator string is needed as the boundary between different data items.

[0051] For example, when transmitting form data and file data simultaneously via the HTTP protocol, form data items, being shorter, can be entirely cached in memory. However, file data items may be longer, requiring multiple delimiters to receive the complete data. Given limited memory space, it's inconvenient to continuously cache the file data being received until completion, especially when the file data length is unpredictable. Therefore, in related technologies, when transmitting form data and file data simultaneously, form data is typically transmitted first, followed by file data. Correspondingly, the receiving end will also default to parsing the form data first, then the file data. That is, if the receiving end parses the file data, it's assumed that the form data has already been parsed, and the remaining data is file data.

[0052] Please see Figure 1 , Figure 1 This is an exemplary embodiment illustrating the content of data received by a data receiver through HTTP protocol transmission. HTTP is an application layer protocol based on TCP / IP. Because TCP is a streaming protocol, meaning its content is a continuous, uninterrupted stream of bytes, and the length of data received in each request is variable, it is necessary to define boundaries for the transmitted content. Therefore, in related technologies, shorter data (such as form parameters) is generally transmitted first, followed by data of variable length that may be longer (such as file data), with delimiters used to separate different data items.

[0053] For example, such as Figure 1 As shown, "-----699156856224027134326940" is the separator string used to separate different data items. For example... Figure 1 As shown, in HTTP data transmission, shorter data (such as form data) is placed first, and data of varying length (such as file data) is placed last, with different data items separated by delimiter strings.

[0054] Among them, Figure 1The HTTP protocol header shown uses Content-Type to represent media type information. When a client requests a service from a server, it only needs to send the request method and path. Common request methods include GET, HEAD, and POST. Each method specifies a different type of communication between the client and the server. The Authorization request header field is mainly used to prove that the client has the right to view a resource. User-Agent is a request header field that allows the client to tell the server its operating system, browser, and other attributes. It should be noted that... Figure 1 This illustration merely demonstrates the interface diagram for data transmission between the data receiver and the data sender via the HTTP protocol and does not imply any special limitation on the transmission interface in this specification.

[0055] In this specification, the delimiter string is typically referred to as the boundary, and it is used to separate different parts of the data by using a randomly generated delimiter string. This is because randomly generated delimiter strings usually have sufficient randomness to ensure their uniqueness within the data content. If the delimiter string is duplicated with the data content, errors may occur when the server parses the request, leading to incorrect request processing. In actual HTTP requests, the delimiter string is usually specified in the boundary parameter of the Content-Type header field. Once the client and server agree on the boundary string, they can use this string to separate different parts, ensuring that each data item can be correctly parsed and processed. The server will use the delimiter string to split and parse different parts of the data in the request body; for example, in multipart / form-data format, the delimiter string is used to separate form fields and files.

[0056] In this scenario, if file data needs to be transmitted first, followed by form data, the receiving end may mistakenly assume that the entire transmission consists of file data, leading to errors in parsing the form data items.

[0057] In view of this, this specification aims to propose a data item delimitation method, which involves buffering the data into a first buffer after the data receiving end receives data containing one or more data items, delimiting the data in the first buffer, and copying the delimited data items to a second buffer for parsing.

[0058] In this specification, a data transmission protocol is used to transmit data between a data sender and a data receiver. The data received by the data receiver contains one or more data items, which correspond to one or more data types. The data length of each data item is variable and unpredictable. The data receiver is configured with a first buffer for delimiting the data items contained in the received data, and a second buffer for parsing the data items from the received data.

[0059] When the data receiving end receives data sent by the data sending end, the received data is cached in the first buffer. The first byte of the data cached in the first buffer is determined as the target byte, and the following steps are iteratively executed until the data cached in the first buffer is completely detected: starting from the target byte, a separator string used to divide different data items is detected; if the separator string is detected, the data located after the target byte and before the first detected separator string, including the target byte, is determined as a data item, and the detected data item is copied to the second buffer; furthermore, the first byte of the data located after the first detected separator string is determined as the target byte.

[0060] Therefore, in the technical solution described in this specification, on the one hand, by configuring a first buffer for the data receiving end, after pre-parsing the received data for data delimitation, the delimited data items can be copied to a second buffer for data parsing. This allows for simultaneous transmission of multiple data types, where longer data (such as file data) that is difficult to fully receive, delimit, and parse in the second buffer can be delimited and parsed together with shorter data (such as form data) that is easier to fully receive, delimit, and parse in memory, both within the memory space of the first buffer and in the memory space of the second buffer. When transmitting multiple data types simultaneously, regardless of the order in which the data items are transmitted, the receiving end can accurately delimit and parse the data items, improving the flexibility and reliability of data transmission.

[0061] On the other hand, copying the data before the delimiter string to the second buffer in the first buffer is equivalent to performing data item delimiting (i.e., pre-parsing) on ​​the data containing one or more data types in the first buffer, and then copying the pre-parsed data items to the second buffer. The second buffer is used by the business process to parse the pre-parsed data items. This scheme can accurately delimit and parse each data item in the data stream received at the data receiving end, even when the length of each data item is unpredictable, thus improving the reliability of data transmission.

[0062] The present application will be described below through specific embodiments, taking the HTTP data transmission protocol as an example, in conjunction with specific application scenarios.

[0063] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating the architecture of an HTTP data transmission system, as shown in an exemplary embodiment. Figure 2 As shown, the data receiving end can be a terminal such as host 20 or mobile phone 21 that has the ability to receive and parse data according to the HTTP protocol, also known as an HTTP client. The data sending end can be server 22. The data receiving end and the data sending end are connected via a network and exchange data via the HTTP protocol.

[0064] In the case where the data sender is the server and the data receiver is the terminal, for example... Figure 2 As shown, when host 20 opens a webpage in a browser or mobile phone 21 uses an application (APP), it can send an HTTP request message to server 22. The HTTP request includes request headers, a request body, and other information. After receiving the HTTP request, server 22 parses the request message and performs corresponding data processing, such as querying data from a database and generating a response message. Once the server has completed data processing, it returns a response message to host 20 or mobile phone 21. Host 20 and mobile phone 21 receive the response message, parse the response message body, and perform application-related processing on the parsed data items.

[0065] The server, a computer providing services, possesses high concurrency processing capabilities and data storage and management capabilities. During data transmission, it primarily parses HTTP request messages, processes HTTP requests, and generates HTTP response messages. The HTTP request message is a request message initiated by the client to the server, consisting of a request header and a request body. The request header describes basic information about the HTTP request, such as the request type, request path, request header fields, and request parameters. The request body describes what data needs to be submitted to the server; for example, in form submission, the form data is stored in the HTTP request body. The response message is the server's response to the client's request message, including a status code, response headers, and a response body. The status code indicates the server's processing result for the request, the response headers contain additional HTTP protocol header information, and the response body contains the actual data returned by the server to the client.

[0066] In some possible embodiments, the host 20 and mobile phone 21 can act as data senders in addition to receiving data, sending user operation information (such as search keywords, order information, etc.) to the server 22. This specification does not limit the specific types of devices included as data receivers and data senders.

[0067] Please see Figure 3 , Figure 3 This is an exemplary embodiment illustrating a data item delimitation method for a data transmission protocol. The method can be applied to a data receiving end. The method may perform the following steps:

[0068] Step 302: Receive data sent by the data sender, cache the received data in the first buffer, and determine the first byte of the data cached in the first buffer as the target byte.

[0069] For example, see Figure 4 , Figure 4 This is a schematic diagram illustrating a first buffer indicating the location of a target byte, as shown in an exemplary embodiment. Figure 4 As shown, the rectangle represents the first buffer 400. When the data “abc123-xyz-def456-xyz-ghi789-xy” is received in the first buffer 400, the first byte of the data is “a”. The target byte 402 is used to indicate the position of the first byte of the data received in the first buffer 400, that is, the target byte 402 indicates the position of the data “a”.

[0070] In this specification, the first byte of the data cached in the first buffer refers to the first byte of the data in the data stream that has been received and stored in the first buffer. The first buffer is a data storage area used to receive data and perform data pre-parsing (i.e., data delimitation). In this specification, the first buffer is used to delimit data items and then copy them to the second buffer for data processing, thus pre-delimiting data whose length is unpredictable and variable, and which is inconvenient to receive and cache entirely in memory. In this specification, the target byte is used to indicate the position of a certain byte and is used for subsequent operations. The target byte is used to divide the data area and detect whether a delimiter string exists in the data area. Setting a target byte helps to more clearly and explicitly indicate the process and steps of data delimitation.

[0071] In one embodiment shown, to more intuitively indicate the detection location, receiving data sent by the data sender and buffering it in the first buffer includes:

[0072] The received data is cached in the first buffer, the first cursor is pointed to the memory address corresponding to the first byte of the data cached in the first buffer, and the second cursor is pointed to the memory address corresponding to the next byte of the data cached in the first buffer after the last byte;

[0073] For example, see Figure 5 , Figure 5 This is an exemplary embodiment illustrating the cursor position when the first buffer of a data receiver has not yet received any data. Figure 5 As shown, the rectangle represents the first buffer 400, which is used to cache data that needs to be decomposed into data items. When the first buffer 400 has not yet received any data, both the first cursor 502 and the second cursor 504 are used to indicate the position of the first byte in the first buffer 400.

[0074] The first cursor is used to indicate the memory address corresponding to the first byte of the uncompleted detection data cached in the first buffer; the second cursor is used to indicate the memory address corresponding to the next byte of the uncompleted detection data cached in the first buffer.

[0075] When data is stored in the first buffer, please refer to Figure 6 , Figure 6 This is an exemplary embodiment illustrating the cursor indicating the position when data is received in the first buffer of a data receiver. Figure 6 As shown, since the data has not yet been parsed, the first cursor 502 is used to indicate the memory address corresponding to the first byte "a" of the uncompleted data cached in the first buffer, and the second cursor 504 is used to indicate the memory address corresponding to the next byte of the last byte "y" of the uncompleted data cached in the first buffer.

[0076] The memory address corresponding to the first byte of the undetected data refers to the memory address in the first buffer where the first byte of the received but undetected data is stored, i.e., the memory address corresponding to the byte following the last byte of the detected data in the first buffer. A cursor is a marker indicating the current or operational position, typically used for browsing, accessing, or modifying data. A cursor can also indicate the current position in a data stream or resource, allowing precise access, reading, or modification of data at a specific location. Furthermore, a cursor can be used to traverse a dataset, processing each element or record individually.

[0077] In one embodiment shown, the received data includes a request message or a response message; before determining the first byte of the data cached in the first buffer as the target byte, the method further includes:

[0078] Obtain the separator string contained in the request header of the request message; or, obtain the separator string contained in the response header of the response message.

[0079] Specifically, when the data receiving end receives an HTTP request message, the separator string is obtained from the request header of the request message. When the data receiving end receives a response message after sending an HTTP request message, the separator string is obtained from the response header of the response message. To obtain the separator string, the data receiving end first parses the request message. This parsing involves using functions provided by a relevant network library or framework to parse the received request message into a processable data structure according to a specified format. Then, the request header portion is obtained from the parsed request message, and specific request header fields are searched for. Finally, the specific field containing the required separator string is found. Finally, according to specific requirements, the value of this field is parsed to obtain the separator string.

[0080] Step 304: Starting from the target byte, detect the separator string used to divide different data items.

[0081] For example, such as Figure 4 As shown, starting from target byte 402, it is checked whether there is a separator string used to divide different data items in the data stored in the first buffer 400.

[0082] The separator string is used to delineate the boundaries of data items. If the separator string is detected in the data stored in the first buffer 400, it means that the data before the separator string and the data after the separator string are two different data items, such as different form data, or one form data and one file data.

[0083] Step 306: Determine if a delimiter string is detected; if so, proceed to step 308.

[0084] Step 308: Determine the data that includes the target byte and is located after the target byte and before the first detected delimiter string as a data item, and copy the detected data item to the second buffer.

[0085] For example, see Figure 7 , Figure 7 This is an exemplary embodiment illustrating a diagram of copying delimited data items from a first buffer to a second buffer. For example... Figure 7As shown, the separator string is the string "-xyz-". Starting from the target byte 402, it is detected whether there is a separator string used to divide different data items in the data stored in the first buffer 400. When the existence of the first separator string "-xyz-" is detected, the data item "abc123" before the detected first separator string "-xyz-" is copied to the second buffer 700.

[0086] The second buffer is where the file data items or parameter data items parsed from the first buffer after delimiter string detection are finally stored. The data stored in the second buffer is used in subsequent application layer business processes.

[0087] Step 310: Determine the first byte of the data following the first detected delimiter string as the target byte.

[0088] For example, such as Figure 7 As shown, the first byte "d" of the data following the first detected separator string "-xyz-" is determined as the target byte 402.

[0089] The target byte indicates the starting position for the next check to see if the separator string exists in the first buffer 400. In this specification, the target byte is moved to the first byte of the data after the separator string after the data item is copied to avoid repeated checks and copying of the same data item. After step 310 is executed, the process returns to step 304 to continue the loop.

[0090] Step 312: Determine whether the data cached in the first buffer has been detected; if not, continue to step 304; if yes, end the iteration.

[0091] In one embodiment shown, to more intuitively indicate the detection location, the detection of the separator string used to divide different data items, starting from the target byte, includes:

[0092] Detect whether the data before the memory address corresponding to the byte pointed to by the second cursor in the first buffer contains the separator string; if the separator string is contained, move the first cursor to the memory address corresponding to the first byte of the detected separator string, and divide the data before the memory address pointed to by the first cursor into data items.

[0093] For example, such as Figure 7As shown, it detects whether the data before the memory address corresponding to the byte pointed to by the second cursor 504 in the first buffer 400 contains the separator string "-xyz-". Since the data cached in the first buffer 400 contains the separator string "-xyz-", the first cursor 502 is pointed to the memory address corresponding to the first byte "-" of the first separator string (i.e., the separator string "-xyz-" after the string "abc123"), and the data "abc123" before the memory address pointed to by the first cursor 502 is divided into data items.

[0094] In this application, the second buffer is equivalent to the memory buffer in the background technology. The first buffer added by the technical solution performs pre-parsing of the received data to define the data boundaries, and copies the defined data items to the second buffer for data parsing.

[0095] In one embodiment shown, to correctly identify the complete delimiter string and improve parameter delimitation accuracy when the delimiter string appears only partially in the first buffer, the method further includes, starting from the target byte, detecting the delimiter string used to divide different data items:

[0096] If the separator string is not detected, the first cursor is pointed to the memory address corresponding to the last byte of the separator string, and it is checked whether the number of bytes between the byte pointed to by the first cursor and the byte pointed to by the second cursor is not less than the number of bytes occupied by the separator string;

[0097] If the number of bytes occupied by the separator string is not less than the number of bytes occupied by the separator string, copy the data between the memory address of the byte pointed to by the first cursor and the memory address pointed to by the second cursor offset forward by a total of the number of bytes occupied by the separator string minus one byte to the second buffer; then overwrite the remaining data in the first buffer that has not been copied to the second buffer to the memory address corresponding to the first byte of the memory space of the first buffer.

[0098] For example, see Figure 8 , Figure 8 This is an exemplary embodiment illustrating a situation where no delimiter string is detected after the target byte in the first buffer. For example... Figure 8As shown, after the target byte 402, only a portion of the delimiter string "-xy" appears in the first buffer 400. At this time, the first cursor 502 points to the byte "g" following the last delimiter string "-xyz-" in the first buffer 400, and the second cursor 504 points to the byte "y" following the last byte of received data in the first buffer 400. It is then checked whether the number of bytes between the byte "g" pointed to by the first cursor 502 and the byte "y" pointed to by the second cursor 504 (i.e., the number of bytes from "g" to "y", excluding the byte following "y") is not less than the number of bytes occupied by the delimiter string "-xyz-" (5 bytes).

[0099] The separator string "-xyz-" occupies 5 bytes, and the number of bytes occupied by the separator string minus one byte is 4 bytes. The number of bytes between byte "g" and the next byte after byte "y" is 9 bytes, which is less than 5 bytes. Therefore, the data "ghi78" between the memory address of byte "g" pointed to by the first cursor 502 and the memory address of byte "9" pointed to by the second cursor 504 offset forward by the number of bytes occupied by the separator string "-xyz-" (5 bytes) minus one byte (i.e., 4 bytes) is copied to the second buffer 700, and the memory address of byte "9" pointed to by the first cursor is updated synchronously.

[0100] In one embodiment shown, detecting whether the number of bytes between the byte pointed to by the first cursor and the byte pointed to by the second cursor is not less than the number of bytes occupied by the separator string includes:

[0101] If the number of bytes is less than the number of bytes occupied by the delimiter string, the data between the memory address corresponding to the byte pointed to by the first cursor and the memory address corresponding to the byte pointed to by the second cursor is directly overwritten and written to the memory address corresponding to the first byte of the memory space where the first buffer is located. At the same time, the first cursor and the second cursor are updated to the address of the first byte of the target position and the address of the next byte after the last byte, respectively.

[0102] For example, see Figure 9 , Figure 9 This is an exemplary embodiment illustrating another case where no delimiter string is detected after the target byte in the first buffer. (See example...) Figure 9As shown, after the target byte 402, only a portion of the separator string "-xy" appears in the first buffer 400. The number of bytes between byte "g" and the next byte of byte "y" is 4 bytes, which is less than the number of bytes occupied by the separator string "-xyz-" (5 bytes). At this time, the data "g-xy" (excluding the next byte of "y") between the memory address of byte "g" pointed to by the first cursor 502 and the memory address corresponding to the next byte of byte "y" pointed to by the second cursor 504 is directly overwritten and written to the memory address corresponding to the first byte of the memory space where the first buffer 400 is located. The first cursor is updated to point to the memory address where byte "g" is located in the data "g-xy", and the second cursor is updated to point to the memory address corresponding to the next byte of byte "y" in the data "g-xy".

[0103] It should be noted that in the embodiments shown above, data that may contain only a portion of the separator string is retained in the first buffer 400. When the first buffer 400 subsequently receives new cached data, it can detect whether the complete separator string exists. The complete separator string includes the hyphens that connect the separator strings.

[0104] In one embodiment shown, the data transmission protocol includes:

[0105] HTTP, HTTPS, POP3, and SMTP protocols, etc.

[0106] Among them, protocols such as HTTP (Hypertext Transfer Protocol), HTTPS (Hypertext Transfer Protocol Secure), POP3 (Post Office Protocol version 3), and SMTP (Simple Mail Transfer Protocol) all require delimiting data items using separator strings to ensure that data is transmitted and parsed in the correct format.

[0107] Corresponding to the above embodiments of the data item delimitation method for data transmission protocols, this specification also provides an embodiment of a data item delimitation device for data transmission protocols.

[0108] Please see Figure 10 , Figure 10This is an exemplary embodiment illustrating the hardware structure of an electronic device containing a data item delimitation device for a data transmission protocol. At the hardware level, the device includes a processor 1002, an internal bus 1004, a network interface 1006, memory 1008, and non-volatile memory 1010, and may also include other necessary hardware. One or more embodiments of this specification can be implemented in software, for example, the processor 1002 reads the corresponding computer program from the non-volatile memory 1010 into memory 1008 and then runs it. Of course, besides software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0109] Please see Figure 11 , Figure 11 This is a block diagram illustrating a data item delimiting device for a data transmission protocol, as shown in an exemplary embodiment. This data item delimiting device for a data transmission protocol can be applied to, for example... Figure 10 The illustrated electronic device is used to implement the technical solution of this specification. The data item delimiting device for the data transmission protocol is applied to a data receiving end; the data received by the data receiving end contains one or more data items; the multiple data items correspond to one or more data types; the data length of each data item in the multiple data items is not fixed and is unpredictable; the data receiving end is configured with a first buffer for delimiting each data item contained in the received data, and a second buffer for parsing each data item from the received data; the device may include:

[0110] The first buffer unit 1102 is used to receive and buffer the data sent by the data sender, determine the first byte of the data as the target byte, and iteratively execute the following steps until all the buffered data has been detected: starting from the target byte, detect the separator string used to divide different data items; if the separator string is detected, determine the data that includes the target byte and is located after the target byte and before the first detected separator string as a data item, and copy the detected data item to the second buffer unit; determine the first byte of the data located after the first detected separator string as the target byte;

[0111] The second buffer unit 1104 is used to receive the data item to be parsed sent by the first buffer unit, parse the data item, and provide it for use in business processes.

[0112] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0113] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0114] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0115] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0116] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0117] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0118] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0119] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0120] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0121] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0122] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0123] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the scope of protection of one or more embodiments of this specification.

Claims

1. A method for delimiting data items in a data transmission protocol, characterized in that, The data transmission protocol is used to transmit data between a data sender and a data receiver; the method is applied to the data receiver; the data received by the data receiver contains one or more data items; the multiple data items correspond to one or more data types; the data length of each data item in the multiple data items is not fixed and is unpredictable; the data receiver is configured with a first buffer for delimiting each data item contained in the received data, and a second buffer for parsing each data item from the received data; the method includes: The system receives data sent by the data sender, caches the received data in the first buffer, determines the first byte of the data cached in the first buffer as the target byte, and iteratively executes the following steps until the data cached in the first buffer has been completely detected: Starting from the target byte, detect the separator string used to divide different data items; If the separator string is detected, the data that includes the target byte and is located after the target byte and before the first detected separator string is determined to be a data item, and the detected data item is copied to the second buffer; The first byte of the data following the first detected separator string is determined as the target byte.

2. The method according to claim 1, characterized in that, The step of caching the received data into the first buffer includes: The received data is cached in the first buffer, the first cursor is pointed to the memory address corresponding to the first byte of the data cached in the first buffer, and the second cursor is pointed to the memory address corresponding to the next byte of the data cached in the first buffer after the last byte; The first cursor is used to indicate the memory address corresponding to the first byte of the uncompleted detection data cached in the first buffer; the second cursor is used to indicate the memory address corresponding to the next byte of the uncompleted detection data cached in the first buffer.

3. The method according to claim 1, characterized in that, The received data includes request messages or response messages; Before determining the first byte of the data cached in the first buffer as the target byte, the method further includes: Obtain the delimiter string contained in the request header of the request message; Alternatively, obtain the delimiter string contained in the response header of the response message.

4. The method according to claim 2, characterized in that, The step of detecting the separator string used to divide different data items, starting from the target byte, includes: Detect whether the data before the memory address corresponding to the byte pointed to by the second cursor in the first buffer contains the separator string; if the separator string is contained, point the first cursor to the memory address corresponding to the first byte of the detected separator string, and divide the data before the memory address pointed to by the first cursor into data items.

5. The method according to claim 2, characterized in that, Starting from the target byte, after detecting the separator string used to divide different data items, the method further includes: If the separator string is not detected, the first cursor is pointed to the memory address corresponding to the last byte of the separator string, and it is checked whether the number of bytes between the byte pointed to by the first cursor and the byte pointed to by the second cursor is not less than the number of bytes occupied by the separator string; If the number of bytes occupied by the separator string is not less than the number of bytes occupied by the separator string, copy the data between the memory address of the byte pointed to by the first cursor and the memory address pointed to by the second cursor offset forward by a total of the number of bytes occupied by the separator string minus one byte to the second buffer; then overwrite the remaining data in the first buffer that has not been copied to the second buffer to the memory address corresponding to the first byte of the memory space of the first buffer.

6. The method according to claim 5, characterized in that, The step of detecting whether the number of bytes between the bytes pointed to by the first cursor and the bytes pointed to by the second cursor is not less than the number of bytes occupied by the delimiter string includes: If the number of bytes is less than the number of bytes occupied by the separator string, the data between the memory address corresponding to the byte pointed to by the first cursor and the memory address corresponding to the byte pointed to by the second cursor is directly overwritten and written to the memory address corresponding to the first byte of the memory space where the first buffer is located.

7. The method according to claim 1, characterized in that, The data transmission protocol includes: HTTP, HTTPS, POP3, and SMTP protocols.

8. A data item delimitation device for a data transmission protocol, characterized in that, The data transmission protocol is used to transmit data between a data sender and a data receiver; the device is applied to the data receiver; the data received by the data receiver contains one or more data items; the multiple data items correspond to one or more data types; the data length of each data item in the multiple data items is not fixed and is unpredictable; the data receiver is configured with a first buffer unit for delimiting each data item contained in the received data, and a second buffer unit for parsing each data item from the received data; the device includes: The first buffer unit is used to receive and buffer the data sent by the data sender, determine the first byte of the data as the target byte, and iteratively execute the following steps until all the buffered data has been detected: starting from the target byte, detect the separator string used to divide different data items; if the separator string is detected, determine the data that includes the target byte and is located after the target byte and before the first detected separator string as a data item, and copy the detected data item to the second buffer unit; determine the first byte of the data located after the first detected separator string as the target byte; The second buffer unit is used to receive the data items to be parsed sent by the first buffer unit, parse the data items, and provide them for use in business processes.

9. An electronic device, comprising a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the method according to any one of claims 1 to 7 by invoking the machine-readable instructions.

10. A machine-readable storage medium storing machine-readable instructions that, when invoked and executed by a processor, implement the method of any one of claims 1 to 7.

Images