Data processing method, device, system, equipment, medium and product
By sharding data into shared memory in a virtualized scenario for reorganization, the data transmission delay and reliability problems between the virtual machine and the host are solved, and an efficient and reliable communication process is achieved.
Patent Information
- Application Number
- CN202411132307.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-12-06
AI Technical Summary
In a virtualized scenario, the data transmission between the host and the virtual machine is affected by the network conditions, resulting in the delay of data transmission or the transmission failure. The existing technology cannot effectively decouple the communication process and cannot batch based on data priority. Quick processing.
By sharding the data to be transmitted, and writing multiple data to be transmitted fragments and fragment information to the shared memory between the sending end device and the receiving end device, the receiving end device can recombine the data based on the shard information to avoid passing network transmission, thereby improving communication efficiency and reliability.
Through shared memory transmission, the delay and failure of network transmission are avoided, the communication efficiency and reliability between the host and the virtual machine are improved, and efficient processing can be carried out according to data priority.
Smart Images

Figure CN119088591A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of communication technology, and in particular, to a data processing method, apparatus, system, equipment, medium and product. Background Art
[0002] In a virtualization scenario, a virtual machine is built on a host machine (also called a physical server host, HOST), and the host machine needs to allocate dedicated memory to the virtual machine to enable it to run normally. Multiple virtual machines can run on one host machine.
[0003] The data transmission rate between the host and the virtual machine will be affected by the network conditions. When the amount of data to be transmitted is large, it may cause a large delay in data transmission or data transmission failure. Summary of the invention
[0004] The present application provides a data processing method, apparatus, system, device, medium and product for ensuring the reliability of data transmission between a host machine and a virtual machine.
[0005] In order to achieve the above objectives, this application adopts the following technical solutions:
[0006] In a first aspect, a data processing method is provided, which is applied to a sending device, and the method includes: fragmenting the data to be transmitted to obtain multiple data fragments to be transmitted and fragment information of each data fragment to be transmitted; the fragment information is used to indicate the position of the data fragment to be transmitted in the data to be transmitted; the multiple data fragments to be transmitted and the fragment information are written into a shared memory between the sending device and the receiving device, so that the receiving device reads the multiple data fragments to be transmitted and the fragment information in the shared memory, and uses the fragment information to reorganize the multiple data fragments to be transmitted into the data to be transmitted.
[0007] Optionally, multiple data slices to be transmitted and slice information are written into a shared memory, including: for each data slice to be transmitted, determining a first target thread for the data slice to be transmitted; the first target thread is a first thread or a second thread in a sending end device; the first thread is used to process the data slice to be transmitted according to the priority of the data to be transmitted; the second thread is used to process the data slice to be transmitted according to the sending time of the data to be transmitted; based on the first target thread of each data slice to be transmitted, the data slice to be transmitted and the slice information are written into the shared memory.
[0008] Optionally, when the first start time is before the second start time, the first target thread is the first thread; when the first start time is after the second start time, the first target thread is the second thread; the first start time is the start time for the first thread to process the data slices to be transmitted; the second start time is the start time for the second thread to process the data slices to be transmitted.
[0009] Optionally, the shared memory includes multiple memory blocks, and the method also includes: determining at least one free memory block address for storing multiple data fragments to be transmitted and fragment information; the size of at least one free memory block is greater than or equal to the memory size required for the multiple data fragments to be transmitted; sending a first indication message to the receiving device; the first indication message is used to instruct the receiving device to read the multiple data fragments to be transmitted and fragment information from the memory block corresponding to the at least one free memory block address.
[0010] Optionally, the data to be transmitted is segmented, including: determining whether the data to be transmitted meets a first preset condition; the first preset condition is that the memory size occupied by the data to be transmitted is larger than the memory size of any free memory block of the shared memory; if the first preset condition is met, the data to be transmitted is segmented.
[0011] Based on the technical solution provided by the present application, the data to be transmitted is fragmented, and multiple data fragments to be transmitted and fragment information are written into a shared memory. Since the fragment information is used to indicate the position of the data fragment to be transmitted in the data to be transmitted. In this way, after reading the shared memory, the receiving device can reorganize the multiple data fragments to be transmitted into the data to be transmitted based on the fragment information. Through shared memory, when transmitting data, there is no need to use network transmission, it will not be affected by network conditions, and avoids large delays or data transmission failures caused by excessive data volume, thereby improving the communication efficiency and communication reliability between the receiving device and the sending device.
[0012] In a second aspect, a data processing method is provided, which is applied to a receiving device, and the method includes: reading multiple data fragments to be transmitted and fragment information from a shared memory between a sending device and a receiving device; the multiple data fragments to be transmitted and the fragment information are obtained after the sending device fragments the data to be transmitted; the fragment information is used to indicate the position of the data fragment to be transmitted in the data to be transmitted; and using the fragment information to reorganize the multiple data fragments to be transmitted into the data to be transmitted.
[0013] Optionally, multiple data slices to be transmitted and slice information in the shared memory are read, including: for each data slice to be transmitted, determining a second target thread for the data slice to be transmitted; the second target thread is the first thread or the second thread in the receiving end device; the first thread is used to process the data slice to be transmitted according to the priority of the data to be transmitted; the second thread is used to process the data slice to be transmitted according to the sending time of the data to be transmitted; based on the second target thread of each data slice to be transmitted, multiple data slices to be transmitted and slice information are read.
[0014] Optionally, when the third start time is before the fourth start time, the second target thread is the first thread; when the third start time is after the fourth start time, the second target thread is the second thread; the third start time is the start time for the first thread to process the data slices to be transmitted; the fourth start time is the start time for the second thread to process the data slices to be transmitted.
[0015] Optionally, reading multiple data slices to be transmitted and slice information based on the second target thread includes: receiving a first indication message; the first indication message is used to indicate at least one free memory block address for storing multiple data slices to be transmitted and slice information; the size of at least one free memory block is greater than or equal to the memory size required by the multiple data slices to be transmitted; based on the second target thread, reading multiple data slices to be transmitted and slice information from the memory block corresponding to the free memory block address.
[0016] In a third aspect, a data processing device is provided, which is applied to a sending end device, and the device includes: a processing unit; a processing unit, which is used to fragment the data to be transmitted, and obtain multiple data fragments to be transmitted and fragment information of each data fragment to be transmitted; the fragment information is used to indicate the position of the data fragment to be transmitted in the data to be transmitted; the processing unit is also used to write the multiple data fragments to be transmitted and the fragment information into a shared memory, so that the receiving end device reads the multiple data fragments to be transmitted and the fragment information in the shared memory, and uses the fragment information to reorganize the multiple data fragments to be transmitted.
[0017] Optionally, the processing unit is specifically used to: for each data slice to be transmitted, determine the first target thread of the data slice to be transmitted; the first target thread is the first thread or the second thread in the sending end device; the first thread is used to process the data slice to be transmitted according to the priority of the data to be transmitted; the second thread is used to process the data slice to be transmitted according to the sending time of the data to be transmitted; based on the first target thread of each data slice to be transmitted, write the data slice to be transmitted and the slice information into the shared memory.
[0018] Optionally, when the first start time is before the second start time, the first target thread is the first thread; when the first start time is after the second start time, the first target thread is the second thread; the first start time is the start time for the first thread to process the data slices to be transmitted; the second start time is the start time for the second thread to process the data slices to be transmitted.
[0019] Optionally, the shared memory includes multiple memory blocks, and the device also includes: a determination unit and a sending unit, the determination unit is used to determine at least one free memory block address for storing multiple data fragments to be transmitted and fragment information; the size of at least one free memory block is greater than or equal to the memory size required for the multiple data fragments to be transmitted; the sending unit is used to send a first indication message to the receiving device; the first indication message is used to instruct the receiving device to read the multiple data fragments to be transmitted and fragment information from the memory block corresponding to the at least one free memory block address.
[0020] Optionally, the processing unit is specifically used to: determine whether the data to be transmitted meets a first preset condition; the first preset condition is that the memory size occupied by the data to be transmitted is larger than the memory size of any free memory block of the shared memory; if the first preset condition is met, the data to be transmitted is fragmented.
[0021] In a fourth aspect, a data processing device is provided, which is applied to a receiving device; a reading unit and a processing unit; the reading unit is used to read multiple data fragments to be transmitted and fragment information from a shared memory between a sending device and a receiving device; the multiple data fragments to be transmitted and the fragment information are obtained after the sending device fragments the data to be transmitted; the fragment information is used to indicate the position of the data fragment to be transmitted in the data to be transmitted; the processing unit is used to reorganize the multiple data fragments to be transmitted using the fragment information.
[0022] Optionally, the reading unit is specifically used to: for each data slice to be transmitted, determine the second target thread of the data slice to be transmitted; the second target thread is the first thread or the second thread in the receiving end device; the first thread is used to process the data slice to be transmitted according to the priority of the data to be transmitted; the second thread is used to process the data slice to be transmitted according to the sending time of the data to be transmitted; based on the second target thread, read multiple data slices to be transmitted and slice information.
[0023] Optionally, when the third start time is before the fourth start time, the second target thread is the first thread; when the third start time is after the fourth start time, the second target thread is the second thread; the third start time is the start time for the first thread to process the data slices to be transmitted; the fourth start time is the start time for the second thread to process the data slices to be transmitted.
[0024] Optionally, the reading unit is specifically used to: receive a first indication message; the first indication message is used to indicate at least one free memory block address for storing multiple data fragments to be transmitted and fragment information; the size of at least one free memory block is greater than or equal to the memory size required for multiple data fragments to be transmitted; based on the second target thread, read multiple data fragments to be transmitted and fragment information from the memory block corresponding to the free memory block address.
[0025] In a fifth aspect, a data processing device is provided, which can implement the functions performed by the data processing device in the above-mentioned aspects or possible designs. The functions can be implemented through hardware, such as: in one possible design, the data processing device may include: a processor and a communication interface, and the processor can be used to support the data processing device to implement the functions involved in the above-mentioned first aspect or any possible design of the first aspect.
[0026] In another possible design, the data processing device may further include a memory, the memory being used to store computer-executable instructions and data necessary for the data processing device. When the data processing device is running, the processor executes the computer-executable instructions stored in the memory, so that the data processing device performs the first aspect or any possible data processing method of the first aspect.
[0027] In the sixth aspect, a computer-readable storage medium is provided, which may be a readable non-volatile storage medium. The computer-readable storage medium stores computer instructions or programs, and when the computer-readable storage medium runs on a computer, the computer can execute the first aspect or any possible data processing method of the above aspects.
[0028] In a seventh aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the data processing method of the first aspect or any possible design of the above aspects.
[0029] In an eighth aspect, an electronic device is provided, the electronic device comprising one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, the one or more memories are used to store computer program codes, the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the data processing method as described in the first aspect or any possible design of the first aspect.
[0030] In a ninth aspect, a chip system is provided, the chip system including a processor and a communication interface, the chip system can be used to implement the functions performed by the data processing device in the first aspect or any possible design of the first aspect. In one possible design, the chip system also includes a memory, the memory is used to store program instructions and / or data. The chip system can be composed of a chip, or it can include a chip and other discrete devices, without limitation. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 A schematic diagram of the structure of a data processing system provided in an embodiment of the present application;
[0032] Figure 2 A structural diagram of another data processing system provided in an embodiment of the present application;
[0033] Figure 3 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;
[0034] Figure 4 A flowchart of a data processing method provided in an embodiment of the present application;
[0035] Figure 5 A flowchart of another data processing method provided in an embodiment of the present application;
[0036] Figure 6 A flowchart of another data processing method provided in an embodiment of the present application;
[0037] Figure 7 A flowchart of another data processing method provided in an embodiment of the present application;
[0038] Figure 8 A structural diagram of another data processing system provided in an embodiment of the present application;
[0039] Fig. 9 A flowchart of another data processing method provided in an embodiment of the present application;
[0040] Fig.10 A structural diagram of another data processing system provided in an embodiment of the present application;
[0041] Fig.11 A flowchart of another data processing method provided in an embodiment of the present application;
[0042] Fig.12 A schematic diagram of the structure of another data processing device provided in an embodiment of the present application;
[0043] Fig.13 A schematic diagram of the structure of another data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0044] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.
[0046] It should also be understood that the term “comprising” indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or components.
[0047] The technical terms involved in this application are explained below.
[0048] 1. Virtual serial port: It is a software-simulated serial communication interface. It can simulate the functions of hardware serial port and allow applications to transmit and communicate serial data through software.
[0049] 2. Shared memory: It is an inter-process communication (IPC) mechanism that allows multiple processes to share the same memory area on the same computer, thereby achieving data exchange and sharing.
[0050] 3. Priority Queue: It is a special type of queue in which each element is associated with a priority value, which determines the order of the element in the queue.
[0051] With the development of virtualization technology, the security of host and virtual machines is becoming more and more important. The communication between security components running between the host and virtual machines needs to support multiple communication scenarios and meet the requirements of high performance, low resource usage, and support data segmentation and data reorganization.
[0052] The current mainstream distributed communication components are based on network message queues and rely on network configuration. The same type of open source components based on shared memory for message transmission have deficiencies in data sharding, data reorganization (also known as large file and big data distribution and reorganization), and parallel sending and receiving. The main manifestations are: 1. The data sharding and data reorganization capabilities are missing, and the business needs to complete this function and cannot completely decouple the communication process. 2. The ability to transmit multiple files in parallel is missing. 3. The message scenario capabilities are missing, and it is impossible to quickly process data in batches according to their priority.
[0053] Figure 1 A structural diagram of a data processing system is shown. Figure 1 As shown, the data processing system includes a host machine and multiple guest virtual machines (GVMs) (such as GVM1, GVM2, GVM3, and GVM4).
[0054] The host machine and the multiple client virtual machines are all configured with different Internet Protocol (IP) addresses.
[0055] After the host machine and multiple client virtual machines are connected to the network, data can be transmitted between the host machine and multiple client virtual machine processes through the IP tunnel protocol (IPC tunnel).
[0056] In this way, the communication efficiency between the host and multiple client virtual machine processes depends on the network. At the same time, the application layer needs to complete its complex logic when sharding and reorganizing data. In the scenario of multiple data parallel transmission and processing, it is impossible to process data according to data priority, etc.
[0057] The underlying layer of the commonly used message middleware in the market depends on the network, and the complex logic of data segmentation and data reorganization needs to be completed by the application. There are the following problems: communication depends on network configuration, depends on network quality in scenarios with high transmission performance and rate requirements, resource bandwidth occupation problems, and applications cannot be decoupled from communication. When the amount of data transmitted is large, it may cause a large delay in data transmission or data transmission failure.
[0058] In view of this, an embodiment of the present application provides a data processing method, which is applied to a transmitting device. The method includes: slicing the data to be transmitted to obtain multiple data slices to be transmitted and slice information of each data slice to be transmitted; the slice information is used to indicate the position of the data slice to be transmitted in the data to be transmitted; the multiple data slices to be transmitted and the slice information are written into a shared memory between the transmitting device and the receiving device, so that the receiving device reads the multiple data slices to be transmitted and the slice information in the shared memory, and uses the slice information to reorganize the multiple data slices to be transmitted into the data to be transmitted.
[0059] The method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.
[0060] It should be noted that the network system described in the embodiment of the present application is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of network systems and the emergence of other network systems, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.
[0061] Figure 2 FIG. 1 is a schematic diagram of a data processing system 10 provided in an embodiment of the present application. Figure 2 As shown, the data processing system 10 may include: a host machine 11 and a plurality of client virtual machines GVM12.
[0062] The host machine 11 and the plurality of GVMs 12 are connected, for example, wirelessly or wiredly.
[0063] In one example, the host machine 11 and the multiple GVMs 12 may be connected via a preset communication channel (vsocket).
[0064] The host machine 11 involved in the embodiment of the present application is used to complete the management and allocation of hardware resources, and present a virtual hardware platform to the virtual machine to achieve the scheduling and isolation of the virtual machine. The virtual hardware platform provides various hardware resources to each virtual machine running on it, such as providing a virtual processor (VCPU), virtual memory, virtual disk, virtual network card, etc.
[0065] For example, the host machine 11 may be configured with multiple virtual serial ports, and one virtual serial port is used to connect to one GVM 12. The virtual serial port is used to synchronize messages between the GVM 12 and the host machine.
[0066] The GVM 12 involved in the embodiments of the present application is used to simulate one or more virtual computers on a physical host through virtual machine software, and these virtual machines work like real computers. Operating systems and applications can be installed on the virtual machines, and the virtual machines can also access network resources. For applications running in the virtual machines, the virtual machines are like working in real computers.
[0067] For example, a GVM 12 is configured with a virtual serial port, which is used to connect to the host machine 11.
[0068] A shared memory is configured between the host machine 11 and each GVM 12. The host machine 11 and the GVM 12 can both write and read the shared memory.
[0069] The host machine 11 and each GVM 12 are configured with a centralized forwarder (IPC Switcher). The centralized forwarder can efficiently support 1 to N, N to 1, and N to N communication scenarios through multi-threading, high priority queues, and low priority queues for centralized transmission and reception. N is a positive integer greater than 1.
[0070] Figure 2 This is just an exemplary framework diagram. Figure 2 The names of the individual devices included in the Figure 2In addition to the functional nodes shown, other nodes may also be included, which is not limited in the embodiments of the present application.
[0071] When it comes to specific implementation, Figure 2 Each device in the Figure 3 The structure shown, or including Figure 3 Parts shown. Figure 3 The data processing device 200 is a schematic diagram of a structure of a data processing device 200 provided in an embodiment of the present application. The data processing device 200 may be a network device, or the data processing device 200 may be a chip or a system on a chip in a network device. Figure 3 As shown, the data processing device 200 includes a processor 201 , a communication interface 202 and a communication line 203 .
[0072] Furthermore, the data processing device 200 may also include a memory 204. The processor 201, the memory 204 and the communication interface 202 may be connected via a communication line 203.
[0073] The processor 201 is a CPU, a general processor, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 201 may also be other devices with processing functions, such as circuits, devices, or software modules, without limitation.
[0074] The communication interface 202 is used to communicate with other devices or other communication networks. The communication interface 202 can be a module, a circuit, a communication interface or any device capable of implementing communication.
[0075] The communication line 203 is used to transmit information between the components included in the data processing device 200.
[0076] The memory 204 is used to store instructions, where the instructions may be computer programs.
[0077] The memory 204 may be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions, or a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, etc., without limitation.
[0078] It should be noted that the memory 204 can exist independently of the processor 201, or can be integrated with the processor 201. The memory 204 can be used to store instructions or program codes or some data, etc. The memory 204 can be located in the data processing device 200, or can be located outside the data processing device 200, without limitation. The processor 201 is used to execute the instructions stored in the memory 204 to implement the data processing method provided in the following embodiments of the present application.
[0079] In one example, the processor 201 may include one or more CPUs, for example, Figure 3 CPU0 and CPU1 in.
[0080] As an optional implementation, the data processing device 200 includes multiple processors, for example, Figure 3 In addition to the processor 201, a processor 205 may also be included.
[0081] It should be pointed out that Figure 3 The composition shown in the Figure 2 The limitations of each device in Figure 3 In addition to the parts shown, Figure 2 The individual devices in the Figure 3 More or fewer components, or combinations of certain components, or different arrangements of components.
[0082] In the embodiment of the present application, the chip system may be composed of a chip, or may include a chip and other discrete devices.
[0083] In addition, the actions, terms, etc. involved in the various embodiments of the present application can refer to each other without limitation. The message name or parameter name in the message exchanged between the various devices in the embodiments of the present application is only an example, and other names can also be used in the specific implementation without limitation.
[0084] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0085] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.
[0086] Combine the following Figure 2 The data processing system shown describes the data processing method provided in the embodiment of the present application.
[0087] Figure 4 A data processing method is provided for the embodiment of the present application, such as Figure 4 As shown, the method includes the following S301-S302:
[0088] S301. The sending end device segments the data to be transmitted to obtain a plurality of data segments to be transmitted and segment information of each data segment to be transmitted.
[0089] The sending device can be Figure 2 The host in can also be Figure 2 The data to be transferred can be files, messages, etc.
[0090] The fragment information is used to indicate the position of the data fragment to be transmitted in the data to be transmitted. For example, it may include a sequence number, a byte position (such as a byte start position) occupied by the data fragment to be transmitted in the data to be transmitted, a size, etc. Different data fragments to be transmitted have different sequence numbers, and different data fragments to be transmitted occupy different byte positions in the data to be transmitted.
[0091] In one example, when the data to be transmitted is a file, the fragmentation information may include the byte position occupied by the fragments of the data to be transmitted in the data to be transmitted. For example, when the size of the data to be transmitted is 2048 kilobytes (kb), there are two fragments of the data to be transmitted, each of which is 1024 kb in size, the fragmentation information of the two fragments of the data to be transmitted may be 0 kb-1024 kb and 1025 kb-2048 kb, respectively.
[0092] When the data to be transmitted is a message, the fragment information may include a sequence number of the transmission data fragment.
[0093] As a possible implementation manner, the transmitting end device may fragment the data to be transmitted according to a preset fragment size to obtain multiple data fragments to be transmitted and fragment information of each data fragment to be transmitted.
[0094] For example, the preset fragment size can be 1 megabyte, 10 megabytes, 20 megabytes, and so on.
[0095] As another possible implementation, the shared memory between the transmitting device and the receiving device is provided with multiple free memory blocks. The transmitting device can fragment the data to be transmitted according to the sizes of the multiple free memory blocks to obtain multiple data fragments to be transmitted and fragment information of each data fragment to be transmitted.
[0096] The sizes of the multiple free memory blocks may be the same or different. For example, when the sizes of the multiple free memory blocks are 2 MB, 20 MB, and 30 MB, the transmitting end device segments the data to be transmitted into pieces of 2 MB, 20 MB, and 30 MB.
[0097] It should be noted that, when the sending end device is a host machine, the receiving end device is a client virtual machine. When the sending end device is a client virtual machine, the receiving end device is a host machine.
[0098] S302, the sending end device writes the multiple data fragments to be transmitted and the fragment information into the shared memory between the sending end device and the receiving end device, so that the receiving end device reads the multiple data fragments to be transmitted and the fragment information in the shared memory, and uses the fragment information to reorganize the multiple data fragments to be transmitted into the data to be transmitted.
[0099] The shared memory refers to the memory that can be written and read by both the sending device and the receiving device.
[0100] As a possible implementation method, the sending device can determine at least one free memory block address for storing multiple data fragments to be transmitted and fragment information, and based on at least one free memory block address, write the multiple data fragments to be transmitted and fragment information into the shared memory between the sending device and the receiving device.
[0101] It should be noted that the size of at least one free memory block is greater than or equal to the memory size required by the multiple data fragments to be transmitted. One free memory block can store one or more data fragments to be transmitted and fragment information.
[0102] In one example, when the sizes of two data fragments to be transmitted are 10 MB and 15 MB respectively, and the size of a free memory block is 25 MB, the transmitting device can put the sizes of the two data fragments to be transmitted into the free memory block.
[0103] Based on the technical solution provided by the present application, the data to be transmitted is fragmented, and multiple data fragments to be transmitted and fragment information are written into a shared memory. Since the fragment information is used to indicate the position of the data fragment to be transmitted in the data to be transmitted. In this way, the receiving device can reorganize multiple data fragments to be transmitted into data to be transmitted based on the fragment information. Through shared memory, when transmitting data, there is no need to use network transmission, and it will not be affected by network conditions, and it avoids large delays or data transmission failures caused by excessive data volume, thereby improving the communication efficiency and communication reliability between the receiving device and the sending device.
[0104] A possible embodiment is as follows Figure 5 As shown, in order to write multiple data fragments to be transmitted and fragment information into a shared memory, S302 of the present application may include the following S401-S402.
[0105] S401: A sending end device determines, for each data slice to be transmitted, a first target thread of the data slice to be transmitted.
[0106] The first target thread is the first thread or the second thread. The first thread is used to process the data slices to be transmitted according to the priority of the data to be transmitted. The second thread is used to process the data slices to be transmitted according to the sending time of the data to be transmitted. The number of the first thread and the second thread can be one or more. The ratio of the first thread or the second thread can be set as needed. For example, it can be 3:1, etc. The first thread and the second thread can process the data slices to be transmitted in parallel.
[0107] It can be understood that by setting multiple threads, parallel processing of multiple data can be achieved, thereby improving the processing efficiency of the data to be transmitted.
[0108] As a possible implementation manner, the sending end device may determine the start time of each thread processing the data slice to be transmitted, and determine the thread with the earliest processing start time among the multiple threads as the first target thread.
[0109] In one example, the transmitting end device may determine a first starting time for the first thread to process the data slice to be transmitted, and a second starting time for the second thread to process the data slice to be transmitted. If the first starting time is before the second starting time, the first thread is determined as the first target thread; if the first starting time is after the second starting time, the second thread is determined as the first target thread.
[0110] It should be noted that the start time of each thread processing the data slice to be transmitted can be determined according to the processing time of the data to be processed in each thread.
[0111] For example, the processing time of the data to be processed in each thread may be determined according to the size of the data to be processed in each thread and the ratio of the processing speed of each thread.
[0112] As another possible implementation method, the sending device can set a first message queue for the first thread and a second message queue for the second thread, and put the data fragments to be transmitted and the corresponding fragment information into the first message queue of the first thread, and put the data fragments to be transmitted and the corresponding fragment information into the second message queue of the second thread, and determine the thread that processes the data fragments to be transmitted and the corresponding fragment information the earliest as the first target thread.
[0113] The first message queue is used to sort the data segments to be transmitted according to the priority of the data to be transmitted, and the second message queue is used to sort the data segments to be transmitted according to the sending time of the second data to be transmitted.
[0114] It should be noted that when a new data segment to be transmitted is added, the first message queue and the second message queue will simultaneously add the data segment to be transmitted. When the first thread processes a data segment to be transmitted in the first message queue, the second thread will delete the data segment to be transmitted in the second message queue.
[0115] It can be understood that by using the first thread and the second thread in different proportions, high-priority data to be transmitted can be given priority while avoiding the problem of low-priority data to be transmitted staying in the message queue for too long, resulting in a long delay for low-priority data to be transmitted.
[0116] S402: The sending end device writes the data slice to be transmitted and the slice information into a shared memory based on the first target thread of each data slice to be transmitted.
[0117] The first target thread of each data slice to be transmitted may be a different thread or the same thread.
[0118] As a possible implementation method, the sending device can determine at least one free memory block address for storing multiple data fragments to be transmitted and fragment information based on the first target thread, and based on the at least one free memory block address, write the multiple data fragments to be transmitted and fragment information into a memory block corresponding to at least one free memory block address in the shared memory between the sending device and the receiving device.
[0119] In one example, when the sizes of free memory blocks are different, the transmitting device may preferentially write the data fragments to be transmitted and the fragment information into the largest free memory block. Furthermore, when the largest free memory block cannot write all the data fragments to be transmitted and the fragment information, the remaining data fragments to be transmitted and the fragment information are written into the remaining largest free memory block until all the data fragments to be transmitted and the fragment information are written into the shared memory.
[0120] A possible embodiment is as follows Figure 6 As shown, the data processing method in the present application may also include the following S501-S502.
[0121] S501. A receiving device determines at least one free memory block address for storing a plurality of data fragments to be transmitted and fragment information.
[0122] The size of at least one free memory block is greater than or equal to the memory size required by the multiple data fragments to be transmitted.
[0123] As a possible implementation method, the sending device reads the size and free memory block address of each free memory block in the shared memory. Furthermore, for each data fragment to be transmitted, the terminal device can determine the free memory block that best matches the data fragment to be transmitted, so as to determine at least one free memory block address for storing multiple data fragments to be transmitted and fragment information.
[0124] It should be noted that the free memory block that best matches the data slice to be transmitted may refer to: among multiple free memory blocks that are larger than the memory required by the data slice to be transmitted, the memory block with the smallest difference in size with the memory required by the data slice to be transmitted.
[0125] In one example, the memory size required for the data fragment a to be transmitted is 10 megabytes. If the sizes of multiple free memory blocks are the first free memory block (9M), the second free memory block (11M), and the third free memory block (15M), respectively, then the free memory block that best matches the data fragment to be transmitted may be the second free memory block.
[0126] S502: The sending end device sends a first indication message to the receiving end device.
[0127] The first indication message is used to instruct the receiving end device to read a plurality of to-be-transmitted data fragments and fragment information from at least one free memory block address. The first indication message includes at least one free memory block address.
[0128] As a possible implementation, the sending device is pre-installed with a virtual serial port, and the sending device can send the first indication message to the receiving device through the virtual serial port. Correspondingly, the receiving device can receive the first indication message based on the receiving end's virtual serial port.
[0129] It should be noted that after a plurality of data fragments to be transmitted and fragment information are written into the shared memory, it will automatically trigger the sending of a first indication message to the receiving device to perform message synchronization through the virtual serial port.
[0130] It can be understood that, through the first indication message, the receiving end device can obtain the data fragments to be transmitted and the fragment information from the shared memory in a timely manner to ensure the data transmission rate.
[0131] A possible embodiment, Figure 7 A data processing method is provided for the embodiment of the present application, such as Figure 7 As shown, the method includes the following S601-S602:
[0132] S601: A receiving device reads a plurality of data fragments to be transmitted and fragment information of each data fragment to be transmitted from a shared memory between a sending device and a receiving device.
[0133] The plurality of data fragments to be transmitted and the fragment information are obtained after the transmitting end device fragments the data to be transmitted. The fragment information is used to indicate the position of the data fragment to be transmitted in the data to be transmitted.
[0134] As a possible implementation manner, the receiving end device may scan the shared memory at a preset frequency to read multiple data fragments to be transmitted and fragment information in the shared memory.
[0135] The preset frequency can be set as required, for example, once per second.
[0136] As another possible implementation method, after writing multiple data fragments and fragment information to be transmitted into the shared memory, the sending device will send a first indication message to the receiving device. The receiving device can read the multiple data fragments and fragment information to be transmitted in response to the first indication message.
[0137] In one example, the first indication message may include a free memory block address for storing each data fragment to be transmitted and fragment information, and the receiving device may read multiple data fragments to be transmitted and fragment information by reading a memory block corresponding to the free memory block address.
[0138] S602: The receiving end device reassembles multiple fragments of data to be transmitted into data to be transmitted using the fragment information.
[0139] As a possible implementation method, when the data to be transmitted is a file, the receiving device can create a file template, and obtain the byte position occupied by each data slice to be transmitted in the data to be transmitted by reading the slice information of each data slice to be transmitted. Furthermore, according to the byte position of each data slice to be transmitted, each data slice to be transmitted is written into the file template to obtain the reorganized data to be transmitted.
[0140] As another possible implementation method, when the data to be transmitted is a message, the receiving device can create a message buffer, and obtain the serial number of each data fragment to be transmitted by reading the fragment information of each data fragment to be transmitted. Further, according to the order of the serial number size of each data fragment to be transmitted, each data fragment to be transmitted is written into the message buffer to obtain the reorganized data to be transmitted.
[0141] It should be noted that the smaller the sequence number of the data segment to be transmitted is, the earlier its position in the message buffer is.
[0142] Furthermore, after the second target thread reassembles a plurality of data fragments to be transmitted into data to be transmitted by using the fragmentation information, it may send the reassembled data to be transmitted to the application layer of the receiving end.
[0143] In a possible embodiment, S601 in the present application may include the following S701-S702:
[0144] S701: For each data slice to be transmitted, a receiving device determines a second target thread of the data slice to be transmitted.
[0145] The second target thread is the first thread or the second thread in the receiving device. The first thread is used to process the data fragments to be transmitted according to the priority of the data to be transmitted; the second thread is used to process the data fragments to be transmitted according to the sending time of the data to be transmitted. The number of the first thread and the second thread in the receiving device can be the same as the number of the first thread and the second thread in the sending device, or different, without limitation.
[0146] As a possible implementation manner, the receiving end device may determine the start time of each thread processing the data slice to be transmitted, and determine the thread with the earliest processing start time among the multiple threads as the second target thread.
[0147] In one example, the receiving end device may determine the third starting time when the first thread processes the data slice to be transmitted, and the fourth starting time when the second thread processes the data slice to be transmitted; if the third starting time is before the fourth starting time, the first thread is determined as the second target thread. If the third starting time is after the fourth starting time, the second thread is determined as the second target thread.
[0148] The specific method for each thread to process the start time of the data segment to be transmitted can refer to the description of S401 above, which will not be described in detail here.
[0149] S702: The receiving device reads multiple data fragments to be transmitted and fragment information based on the second target thread.
[0150] The second target thread of each data slice to be transmitted may be a different thread or the same target thread.
[0151] As a possible implementation manner, the receiving end device may scan the shared memory based on the second target thread to read multiple data fragments to be transmitted and fragment information in the shared memory.
[0152] As another possible implementation method, after writing multiple data fragments and fragment information to be transmitted into the shared memory, the sending device will send a first indication message to the receiving device. The receiving device can respond to the first indication message and read the multiple data fragments and fragment information to be transmitted through the second target thread.
[0153] In one example, the first indication message may include a free memory block address for each data fragment and fragment information to be transmitted, and the second target thread of the receiving device may read multiple data fragments and fragment information to be transmitted by reading a memory block corresponding to the free memory block address.
[0154] It can be understood that based on the second target thread reading multiple data fragments to be transmitted and fragment information, since the second target thread is a thread between the first thread and the second thread of the receiving end device and can process multiple data fragments to be transmitted earlier, the reorganization speed of the data fragments to be transmitted can be improved at the receiving end, further improving the data transmission speed.
[0155] Figure 8 A structural diagram of a data processing system is provided for an embodiment of the present application, such as Figure 8 As shown, the data processing system includes a sending end device and a receiving end device. The data to be transmitted is a file.
[0156] The application layer of the sending device may be used to segment the file, obtain multiple file segments and corresponding segment information, and put the multiple file segments and corresponding segment information into a message queue.
[0157] Multiple threads in the sending device can be used to concurrently process the message queue and write multiple file fragments and corresponding fragment information into the shared memory.
[0158] Multiple threads in the receiving device can be used to concurrently read multiple file fragments and corresponding fragment information in the shared memory, reorganize the multiple file fragments into files according to the fragment information, and send the reorganized files to the application layer of the receiving end.
[0159] Fig. 9 A structural diagram of a data processing system is provided for an embodiment of the present application, such as Fig. 9 As shown, in the case where the data to be transmitted is a file, the data processing method in the present application may also include the following S1-S13.
[0160] S1. The sending end device determines whether the size of the file meets a first preset condition.
[0161] The first preset condition may be that the memory size occupied by the file is larger than the memory size of any free memory block of the shared memory.
[0162] As a possible implementation, the sending end device may determine that the size of the file meets the first preset condition when the memory size occupied by the file is larger than the memory size of any free memory block of the shared memory. When the memory size occupied by the file is less than or equal to the memory size of any free memory block of the shared memory, it is determined that the size of the file does not meet the first preset condition.
[0163] It should be noted that the sending end device may execute the following S2 when it is determined that the size of the file meets the first preset condition.
[0164] S2. The sending device segments the file to obtain multiple file segments and segment information.
[0165] The fragment information includes the byte position occupied by the file fragment in the file and may also include the size of the file fragment.
[0166] It should be noted that the specific implementation of this step can refer to the description of S301 above, which will not be repeated here.
[0167] S3. The sending device determines whether the file has a priority.
[0168] As a possible implementation, the sending end device may determine whether there is priority information for the file. If there is no priority information, it is determined that the file does not have a priority. If there is priority information, it is determined that the file has a priority.
[0169] The priority information is used to indicate the priority level of the file.
[0170] It should be noted that the sending end device may execute the following S4 when it is determined that the file has a priority, and when the file does not have a priority, the receiving end device may execute the following S6.
[0171] S4. The sending device puts the multiple file fragments and fragment information into a priority queue.
[0172] As a possible implementation manner, the sending end device may put the file into the priority queue according to the priority information of the file and the data information in the priority queue.
[0173] In one example, the sending end device may put the file fragment and the fragment information into the target position in the priority queue. The priority of the data before the target position is greater than the priority of the file. The priority of the data after the target position is less than the priority of the file.
[0174] S5. The sending end device uses multiple threads to process multiple file fragments and fragment information in the priority queue.
[0175] The specific description of this step can refer to the above S401-S402 and will not be repeated here.
[0176] S6. The sending end device uses multiple threads to determine at least one free memory block address for storing multiple file fragments and fragment information.
[0177] The specific description of this step may refer to the above S501 and will not be repeated here.
[0178] It should be noted that, if there is at least one free memory block address for storing file fragments and fragment information, the following S7 is executed. If there is no at least one free memory block address for storing file fragments and fragment information, the above S4 is executed.
[0179] S7. The sending device writes the multiple file fragments and fragment information into the shared memory.
[0180] The specific description of this step may refer to the above S302 and will not be repeated here.
[0181] S8. The sending device sends a first indication message.
[0182] The specific description of this step may refer to the above S502 and will not be repeated here.
[0183] S9. The receiving device determines at least one free memory block address according to the first indication message.
[0184] In one example, the receiving device is configured with a virtual serial port, and the receiving device can determine at least one free memory block address according to the virtual serial port.
[0185] S10. The receiving end device reads a free memory block corresponding to at least one free memory block address to obtain multiple file fragments and fragment information.
[0186] The specific description of this step may refer to the above S601 and will not be repeated here.
[0187] S11. The receiving device puts multiple file fragments and fragment information into a message queue.
[0188] The message queue may be a priority queue in S4 or a non-priority queue, which will not be described in detail here.
[0189] S12. The receiving device uses multiple threads to process the file fragments and fragment information in the message queue.
[0190] The specific description of this step may refer to the above S601 and will not be repeated here.
[0191] S13. The receiving device reassembles the multiple file fragments into a file using the fragmentation information.
[0192] The specific description of this step may refer to the above S602 and will not be repeated here.
[0193] Fig.10 A structural diagram of a data processing system is provided for an embodiment of the present application, such as Fig.10As shown, the data processing system includes a sending end device and a receiving end device. The data to be transmitted is a message.
[0194] The application layer of the sending device may be used to fragment the message, obtain multiple message fragments and corresponding fragment information, and put the multiple message fragments and corresponding fragment information into a message queue.
[0195] Multiple threads in the sending device can be used to concurrently process the message queue and write multiple message fragments and corresponding fragment information into the shared memory.
[0196] Multiple threads in the receiving device can be used to concurrently read multiple message fragments and corresponding fragment information in the shared memory, reassemble the multiple message fragments into messages according to the fragment information, and send the reassembled messages to the application layer of the receiving end.
[0197] Fig.11 A flowchart of a data processing system is provided for an embodiment of the present application, such as Fig.11 As shown, in the case where the data to be transmitted is a message, the data processing method in the present application may also include the following S21-S33.
[0198] S21. The sending device determines whether the size of the message meets a first preset condition.
[0199] The first preset condition may be that the memory size occupied by the message is larger than the memory size of any free memory block of the shared memory.
[0200] As a possible implementation, the sending end device may determine that the size of the message meets the first preset condition when the memory size occupied by the message is larger than the memory size of any free memory block of the shared memory. When the memory size occupied by the message is less than or equal to the memory size of any free memory block of the shared memory, it is determined that the size of the message does not meet the first preset condition.
[0201] It should be noted that the sending end device may execute the following S22 if it is determined that the size of the message meets the first preset condition, and execute the following S23 if it does not meet the first preset condition.
[0202] S22. The sending end device fragments the message to obtain multiple message fragments and fragment information.
[0203] The fragment information includes the sequence number of the message fragment and may also include the size of the message fragment.
[0204] It should be noted that the specific implementation of this step can refer to the description of S301 above, which will not be repeated here.
[0205] S23. The sending device determines whether the message has a priority.
[0206] As a possible implementation, the sending end device may determine whether the message has priority information. If the message does not have priority information, it is determined that the message does not have priority. If the message does have priority information, it is determined that the message has priority.
[0207] The priority information is used to indicate the priority level of the message.
[0208] It should be noted that the sending end device may execute the following S24 when it is determined that the message has a priority, and when the message does not have a priority, the receiving end device may execute the following S26.
[0209] S24. The sending device puts the multiple message fragments and fragment information into a priority queue.
[0210] As a possible implementation manner, the sending end device may put the message into the priority queue according to the priority information of the message and the data information in the priority queue.
[0211] In one example, the sending end device may put the message fragment and the fragment information into the target position in the priority queue. The priority of the data before the target position is greater than the priority of the message. The priority of the data after the target position is less than the priority of the message.
[0212] S25. The sending end device uses multiple threads to process multiple message fragments and fragment information in the priority queue.
[0213] The specific description of this step can refer to the above S401-S402 and will not be repeated here.
[0214] S26. The sending end device uses multiple threads to determine at least one free memory block address for storing multiple message fragments and fragment information.
[0215] The specific description of this step may refer to the above S501 and will not be repeated here.
[0216] It should be noted that, if there is at least one free memory block address for storing multiple message fragments and fragment information, the following S27 is executed. If there is no at least one free memory block address for storing multiple message fragments and fragment information, the above S24 is executed.
[0217] S27. The sending device writes the multiple message fragments and fragment information into the shared memory.
[0218] The specific description of this step may refer to the above S302 and will not be repeated here.
[0219] S28. The sending device sends a first indication message.
[0220] The specific description of this step may refer to the above S502 and will not be repeated here.
[0221] S29. The receiving device determines at least one free memory block address according to the first indication message.
[0222] In one example, the receiving device is configured with a virtual serial port, and the receiving device can determine at least one free memory block address according to the virtual serial port.
[0223] S30. The receiving end device reads a free memory block corresponding to at least one free memory block address to obtain multiple message fragments and fragment information.
[0224] The specific description of this step may refer to the above S601 and will not be repeated here.
[0225] S31. The receiving device puts multiple message fragments and fragment information into a message queue.
[0226] Among them, the message queue can be a priority queue in S24 or a non-priority queue, which will not be described in detail here.
[0227] S32. The receiving device uses multiple threads to process message fragments and fragment information in the message queue.
[0228] The specific description of this step may refer to the above S601 and will not be repeated here.
[0229] S33. The receiving device reassembles the multiple message fragments into a message using the fragmentation information.
[0230] The specific description of this step may refer to the above S602 and will not be repeated here.
[0231] The various solutions in the above embodiments of the present application can be combined without contradiction.
[0232] The embodiment of the present application can divide the data processing device into functional modules or functional units according to the above method example. For example, each functional module or functional unit can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules or functional units. Among them, the division of modules or units in the embodiment of the present application is schematic, which is only a logical function division. There may be other division methods in actual implementation.
[0233] In the case of dividing each functional module into corresponding functional modules, Fig.12 A structural schematic diagram of a data processing device 800 is shown, which can be a sending end device in a data processing system, and the device includes: a processing unit 801; the processing unit 801 is used to fragment the data to be transmitted to obtain multiple data fragments to be transmitted and fragment information of each data fragment to be transmitted; the fragment information is used to indicate the position of the data fragment to be transmitted in the data to be transmitted; the processing unit 801 is also used to write the multiple data fragments to be transmitted and the fragment information into a shared memory, so that the receiving end device reads the multiple data fragments to be transmitted and the fragment information in the shared memory, and uses the fragment information to reorganize the multiple data fragments to be transmitted.
[0234] Optionally, the processing unit 801 is specifically used to: for each data slice to be transmitted, determine the first target thread of the data slice to be transmitted; the first target thread is the first thread or the second thread in the sending end device; the first thread is used to process the data slice to be transmitted according to the priority of the data to be transmitted; the second thread is used to process the data slice to be transmitted according to the sending time of the data to be transmitted; based on the first target thread, write the data slice to be transmitted and the slice information into the shared memory.
[0235] Optionally, the processing unit 801 is specifically used to: determine a first starting time for the first thread to process the data slices to be transmitted, and a second starting time for the second thread to process the data slices to be transmitted; when the first starting time is before the second starting time, determine the first thread as the first target thread; when the first starting time is after the second starting time, determine the second thread as the first target thread.
[0236] Optionally, the shared memory includes multiple memory blocks, and the device also includes: a determination unit 802 and a sending unit 803, wherein the determination unit 802 is used to determine at least one free memory block address for storing multiple data fragments to be transmitted and fragment information; the size of at least one free memory block is greater than or equal to the memory size required for the multiple data fragments to be transmitted; the sending unit 803 is used to send a first indication message to the receiving device; the first indication message is used to instruct the receiving device to read multiple data fragments to be transmitted and fragment information from at least one free memory block address.
[0237] Optionally, the processing unit 801 is specifically used to: determine whether the data to be transmitted meets a first preset condition; the first preset condition is that the memory size occupied by the data to be transmitted is larger than the memory size of any free memory block of the shared memory; if the first preset condition is met, the data to be transmitted is fragmented.
[0238] Fig.13A structural schematic diagram of another data processing device 900 is shown. The data processing device 900 can be a receiving end device in a data processing system. The data processing device 900 can be used to perform the functions of the data processing device involved in the above embodiment. The device includes: a reading unit 901 and a processing unit 902; the reading unit 901 is used to read multiple data fragments to be transmitted and fragment information in a shared memory; the multiple data fragments to be transmitted and the fragment information are obtained after the sending end device fragments the data to be transmitted; the fragment information is used to indicate the position of the data fragment to be transmitted in the data to be transmitted; the processing unit 902 is used to reorganize the multiple data fragments to be transmitted using the fragment information.
[0239] Optionally, the reading unit 901 is specifically used to: for each data slice to be transmitted, determine the second target thread of the data slice to be transmitted; the second target thread is the first thread or the second thread in the receiving end device; the first thread is used to process the data slice to be transmitted according to the priority of the data to be transmitted; the second thread is used to process the data slice to be transmitted according to the sending time of the data to be transmitted; and read multiple data slices to be transmitted and slice information based on the second target thread.
[0240] Optionally, the reading unit 901 is specifically used to: determine the third starting time for the first thread to process the data slices to be transmitted, and the fourth starting time for the second thread to process the data slices to be transmitted; when the third starting time is before the fourth starting time, determine the first thread as the second target thread; when the third starting time is after the fourth starting time, determine the second thread as the second target thread.
[0241] Optionally, the reading unit 901 is specifically used to: receive a first indication message; the first indication message is used to indicate at least one free memory block address for storing multiple data fragments to be transmitted and fragment information; the size of at least one free memory block is greater than or equal to the memory size required by the multiple data fragments to be transmitted; based on the second target thread, read multiple data fragments to be transmitted and fragment information from the free memory block address.
[0242] The embodiment of the present application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be completed by a computer program to instruct the relevant hardware, and the program can be stored in the above computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be an internal storage unit of the data processing device (including the data sending end and / or the data receiving end) of any of the above embodiments, such as a hard disk or memory of the data processing device. The above computer-readable storage medium can also be an external storage device of the above terminal device, such as a plug-in hard disk equipped on the above terminal device, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. Further, the above computer-readable storage medium can also include both the internal storage unit of the above data processing device and an external storage device. The above computer-readable storage medium is used to store the above computer program and other programs and data required by the above data processing device. The above computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.
[0243] It should be noted that the terms "first" and "second" in the specification, claims and drawings of the present application are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.
[0244] It should be understood that in the present application, "at least one (item)" means one or more, "more than one" means two or more, "at least two (items)" means two or three and more than three, and "and / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0245] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0246] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0247] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0248] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0249] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks or optical disks.
[0250] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A data processing method, characterized in that: Applied to a transmitting device, the method comprises: Slice the data to be transmitted to obtain a plurality of data slices to be transmitted and slice information of each of the data slices to be transmitted; the slice information is used to indicate the position of the data slice to be transmitted in the data to be transmitted; The multiple data fragments to be transmitted and the fragment information are written into the shared memory between the sending device and the receiving device, so that the receiving device reads the multiple data fragments to be transmitted and the fragment information in the shared memory, and uses the fragment information to reorganize the multiple data fragments to be transmitted into the data to be transmitted.
2. The method according to claim 1, characterized in that: The step of writing the plurality of data fragments to be transmitted and the fragment information into a shared memory between the transmitting end device and the receiving end device includes: For each of the data slices to be transmitted, determining a first target thread of the data slice to be transmitted; the first target thread is a first thread or a second thread in the sending end device; the first thread is used to process the multiple data slices to be transmitted according to the priority of the data to be transmitted; the second thread is used to process the multiple data slices to be transmitted according to the sending time of the data to be transmitted; Based on the first target thread of each of the data slices to be transmitted, the multiple data slices to be transmitted and the slice information are written into a shared memory.
3. The method according to claim 2, characterized in that When the first start time is before the second start time, the first target thread is the first thread; when the first start time is after the second start time, the first target thread is the second thread; the first start time is the start time when the first thread processes the data slice to be transmitted; the second start time is the start time when the second thread processes the data slice to be transmitted.
4. The method according to any one of claims 1 to 3, characterized in that The shared memory includes a plurality of memory blocks, and the method further includes: Determine at least one free memory block address for storing a plurality of data fragments to be transmitted and the fragment information; the size of the at least one free memory block is greater than or equal to the memory size required by the plurality of data fragments to be transmitted; A first indication message is sent to the receiving end device; the first indication message is used to instruct the receiving end device to read the multiple data fragments to be transmitted and the fragment information from the memory block corresponding to the at least one free memory block address.
5. The method according to claim 1, characterized in that The step of slicing the data to be transmitted includes: Determine whether the data to be transmitted meets a first preset condition; the first preset condition is that the memory size occupied by the data to be transmitted is greater than the memory size of any free memory block of the shared memory; When the first preset condition is met, the data to be transmitted is segmented.
6. A data processing method, characterized in that: Applied to a receiving device, the method comprises: Reading multiple data slices to be transmitted and the slice information of each data slice to be transmitted from the shared memory between the transmitting end device and the receiving end device; the multiple data slices to be transmitted and the slice information are obtained after the transmitting end device slices the data to be transmitted; the slice information is used to indicate the position of the data slice to be transmitted in the data to be transmitted; The plurality of data fragments to be transmitted are reassembled into the data to be transmitted using the fragment information.
7. The method according to claim 6, characterized in that The step of reading a plurality of data fragments to be transmitted and fragment information of each of the data fragments to be transmitted in a shared memory between the transmitting end device and the receiving end device includes: For each of the data slices to be transmitted, determining a second target thread of the data slice to be transmitted; the second target thread is the first thread or the second thread in the receiving end device; the first thread is used to process the multiple data slices to be transmitted according to the priority of the data to be transmitted; the second thread is used to process the multiple data slices to be transmitted according to the sending time of the data to be transmitted; Based on the second target thread of each of the data slices to be transmitted, the multiple data slices to be transmitted and the slice information are read.
8. The method according to claim 7, characterized in that When the third starting time is before the fourth starting time, the second target thread is the first thread; when the third starting time is after the fourth starting time, the second target thread is the second thread; the third starting time is the starting time when the first thread processes the data slices to be transmitted; the fourth starting time is the starting time when the second thread processes the data slices to be transmitted.
9. The method according to claim 7, characterized in that: The reading the plurality of data fragments to be transmitted and the fragment information based on the second target thread includes: Receive a first indication message; the first indication message is used to indicate at least one free memory block address storing the multiple data fragments to be transmitted and the fragment information; the size of the at least one free memory block is greater than or equal to the memory size required by the multiple data fragments to be transmitted; The multiple data fragments to be transmitted and the fragment information are read from the free memory block address based on the second target thread.
10. A data processing device, characterized in that: Applied to a transmitting end device, the apparatus comprises: a processing unit; The processing unit is used to slice the data to be transmitted to obtain a plurality of data slices to be transmitted and slice information of each of the data slices to be transmitted; the slice information is used to indicate the position of the data slice to be transmitted in the data to be transmitted; The processing unit is also used to write the multiple data fragments to be transmitted and the fragment information into the shared memory between the sending device and the receiving device, so that the receiving device reads the multiple data fragments to be transmitted and the fragment information in the shared memory, and uses the fragment information to reorganize the multiple data fragments to be transmitted into the data to be transmitted.
11. A data processing device, characterized in that: Applied to a receiving end device; the device comprises: a reading unit and a processing unit; The reading unit is used to read a plurality of data fragments to be transmitted and fragment information of each of the data fragments to be transmitted from a shared memory between a transmitting end device and the receiving end device; the plurality of data fragments to be transmitted and the fragment information are obtained after the transmitting end device fragments the data to be transmitted; the fragment information is used to indicate the position of the data fragment to be transmitted in the data to be transmitted; The processing unit is used to reorganize the multiple to-be-transmitted data fragments into the to-be-transmitted data by using the fragmentation information.
12. A data processing system, characterized in that: It comprises a sending end device and a receiving end device; the sending end device is used to execute the method as described in any one of claims 1-5; the receiving end device is used to execute the method as described in any one of claims 6-9.
13. A computer-readable storage medium, characterized in that: The readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 5 or any one of claims 6 to 9 is implemented.
14. An electronic device, characterized in that: include: A processor, a memory and a communication interface; wherein the communication interface is used for the electronic device to communicate with other devices or networks; The memory is used to store one or more programs, which include computer-executable instructions. When the electronic device is running, the processor executes the computer-executable instructions stored in the memory to enable the electronic device to perform any one of claims 1-5 or any one of claims 6-9.
15. A computer program product, comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the method according to any one of claims 1 to 5 or any one of claims 6 to 9 is implemented.
Citation Information
Cited By
Data transmission method and device, computer equipment, storage medium and computer program product
CN119806864A
