Data transmission method and related apparatus

Abstract

A data transmission method, which is applied to implement a data transmission process between different services. In the data transmission method, a process providing a certain service creates a virtualized environment at runtime, and runs in the virtualized environment a thread providing a target service. Because the thread providing the target service actually runs in a memory space corresponding to the process, the thread and the process can be considered as a whole. Therefore, it can be ensured that the thread and the process can access the address of the same memory space, and can then perform data read / write on the basis of the address of the same memory space to implement data transmission between different services. In addition, during data transmission, it is unnecessary to perform processes such as serialization, protocol encoding / decoding, and deserialization, thereby effectively improving the data transmission efficiency. That is, the solution substantially transforms data interaction between two peer processes into data interaction between a process and a thread in the process, thereby avoiding a cumbersome data transmission process between processes and improving the data transmission efficiency.

Description

Data transmission method and related device

[0001] The present application claims priority to the Chinese Patent Application No. 202410895764.5, filed on July 4, 2024, and entitled "A data transmission method and related device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of computers, and in particular to a data transmission method and related device. BACKGROUND

[0003] With the development of the Internet, software systems gradually evolve from monolithic application architecture to microservice architecture. Microservice architecture is to split an application into multiple small services, each of which can be independently deployed, run and expanded, and different services can communicate through service interfaces.

[0004] Microservice architecture solves the problems of monolithic application architecture, such as difficulty in expansion, maintenance, deployment and upgrading, because each service can be independently expanded, maintained, deployed and upgraded, and can be developed using different languages, so microservice architecture has been widely used in large software systems, cloud computing and other scenarios. However, microservice architecture is not only beneficial, but also has higher system complexity compared to monolithic application architecture, and problems such as service discovery, load balancing, service monitoring and business latency need to be considered and solved.

[0005] Specifically, under the microservice architecture, services are run in the form of processes, so the communication between two services is essentially the communication between two processes. Moreover, two services often need to support cross-server deployment, so when two services perform data interaction across servers, serialization, protocol encoding and decoding, and deserialization processes often need to be performed during data transmission, resulting in low data transmission efficiency between services. SUMMARY

[0006] The present application provides a data transmission method that can improve the data transmission efficiency between services.

[0007] In a first aspect, a data transmission method is provided, which is applied to implement a data transmission process between different services. The data transmission method comprises: first, creating a virtualization environment in a running process in a first process and running a target thread for providing a target service in the virtualization environment, the target thread being used to provide the target service. Since the target thread is created and run by the first process after calling a dynamic link library, the target thread runs in the memory space corresponding to the first process.

[0008] Then, the first process acquires the target address and reads or writes the target data in the target address. In the case that the first process reads the target data in the target address, the target data is the data written in the target address by the target thread after performing the data processing operation. In the case that the first process writes the target data in the target address, the target data is the data passed to the target thread by the first process.

[0009] That is, the data transmission between the target thread and the first process is realized by writing the target data into the target address of the memory space by the target thread and then reading the target data from the target address by the first process, or writing the target data into the target address of the memory space by the first process and then reading the target data from the target address by the target thread.

[0010] In the present solution, the process providing a certain service creates a virtualization environment at runtime, and runs the thread providing the target service in the virtualization environment. Since the thread providing the target service is actually running in the memory space corresponding to the process, the thread and the process can be regarded as a whole, so that the thread and the process can access the same piece of memory space address, and then perform data reading and writing based on the same piece of memory space address to realize data transmission between different services, and no serialization, protocol encoding and decoding, and deserialization processes are required during data transmission, effectively improving the data transmission efficiency. That is, the essence of the present solution is to transform the data interaction between two peer processes into the data interaction between the process and the thread in the process, thereby avoiding the cumbersome data transmission process between processes and improving the data transmission efficiency.

[0011] In addition, by running the thread providing the target service in the virtualization environment, the isolation of the thread can be realized, and the running of the thread will not affect the running of the process itself (for example, the thread crash will not cause the process itself to crash), thereby ensuring the independent running of different services.

[0012] In a possible implementation, the virtualization environment includes a virtual machine or a sandbox, which is used to provide an isolation environment and a runtime environment for the target thread, so as to ensure the isolated running between the target thread and the first process, and guarantee the smooth running of the target thread in the first process. In addition, the target address used by the target thread to write the target data can be the off-heap memory of the virtualization environment, which is located in the memory space corresponding to the first process.

[0013] In the scheme, since the target thread runs on the virtualization environment, the target thread uses the virtual memory address in the heap at runtime. For the first process, the first process does not recognize the virtual memory address used by the target thread at runtime, and the first process often needs additional cumbersome operations to access the virtual memory address. However, the memory outside the heap of the virtualization environment is directly accessible by the first process, so that the target data can be conveniently written into the memory outside the heap, and the first process can quickly obtain the target data, thereby improving the efficiency of data transmission.

[0014] In addition, since the memory outside the heap is the memory space allocated to the process, the process can realize data interaction between the threads by accessing the corresponding memory space, thereby ensuring efficient data transmission between different services, avoiding the processes of serialization, protocol encoding and decoding, and deserialization, and effectively improving the data transmission efficiency.

[0015] In a possible implementation, before the first process loads the dynamic link library, in order to realize the construction of the dynamic link library, the source code of the target service can be compiled to obtain the bytecode, or the bytecode of the target service provided in advance can be directly obtained; then, the bytecode of the target service and the library file relied on by the bytecode at runtime are compiled to obtain the machine code; and the machine code is compiled and linked with the machine code of the virtualization environment to obtain the dynamic link library. That is, the dynamic link library obtained by compiling and linking actually includes the machine code of the target service, the machine code of the library file relied on by the target service, and the machine code of the virtualization environment.

[0016] In the scheme, since the machine code can be directly recognized and run by the computer hardware, after the first process calls the dynamic link library, the creation of the virtualization environment and the start of the target thread can be realized by directly running the machine code in the dynamic link library, without the need to perform the compilation process of the bytecode, thereby improving the start performance of the target thread.

[0017] In a possible implementation, after the first process obtains the target data from the target address, the first process deletes the target thread. That is, after the first process obtains the target data generated by the target thread, the target thread has completed the work indicated by the first process, and the first process temporarily does not need to obtain the data generated by the target thread. Therefore, the first process can delete the target thread, thereby freeing up idle computing resources and memory resources, which helps to save the computing resources and memory resources of the device.

[0018] In a possible implementation, after the first process deletes the target thread, the first process can delete the virtualization environment, for example, the first process deletes a virtual machine or a sandbox used for running the target thread. Since the virtualization environment is mainly used for running the target thread that provides the target service, after the target thread is deleted, the virtualization environment has no actual use, and thus the virtualization environment can also be deleted, thereby freeing more computing resources and memory resources.

[0019] Alternatively, the first process releases the virtualization environment to a resource pool, where the virtualization environment released to the resource pool only includes the code of the target service. That is, the virtualization environment is not deleted, but only the target thread originally running in the virtualization environment is deleted, and the code of the target service is still retained in the virtualization environment, and the running of the target thread can be implemented by running the code of the target service. In the case where the virtualization environment is released to the resource pool, if the first process needs to continue to obtain the data generated by the target thread subsequently, the first process can directly obtain the virtualization environment from the resource pool and quickly start the target thread in the virtualization environment, without the need to perform the process of calling the dynamic link library to create the virtualization environment, thereby improving the efficiency of starting the target service in the first process.

[0020] In a possible implementation, the target address can be applied by the target thread. After the target thread completes the writing of the target data, the first process receives the target address sent by the target thread, thereby obtaining the target address.

[0021] In this scheme, the target thread applies the target address for writing the target data and returns the target address to the first process after completing the writing of the target data, so that the first process can obtain the data transmitted by the target thread by reading the data in the memory space corresponding to the first process, thereby improving the data transmission efficiency. Moreover, the target thread that generates the target data applies the target address (that is, the data generator applies the address for writing the data), so that the address can be applied on demand, the utilization rate of the memory space is improved, and the management of the memory space is facilitated.

[0022] In a possible implementation, the target address is applied by the first process, and the first process previously transmits the target address for writing the target data to the target thread. After the target thread completes the writing of the target data, the first process receives a notification message sent by the target thread, where the notification message is used to indicate that the target thread completes the writing of the target data. Then, the first process obtains the target address previously transmitted to the target thread, thereby determining the target data generated by the target thread from the target address.

[0023] In the scheme, the first process applies for a target address for writing target data in advance and delivers the target address to the target thread, so that the target thread writes data according to the target address in the memory space provided by the first process, the first process can manage its own memory space, and the first process can quickly obtain the target data generated by the target thread, and the efficiency of data transmission is improved.

[0024] In a possible implementation, the source code of the target service and the source code of the first process are written based on different programming languages. Since the service provided by the first process is different from the target service, and different services may use corresponding programming languages according to actual needs during development, the source code of the target service and the source code of the first process can be written based on different programming languages. For example, the source code of the first process is written based on the C++ language, and the source code of the target service can be written based on the Java language or the Python language.

[0025] In a possible implementation, the first process is configured to provide a service different from the target service. For example, the data transmission method can be applied to a micro-service architecture, and the first process is configured to provide a big data analysis service, and the target service provided by the target thread can be a data read-write service.

[0026] In a possible implementation, the first process can create the virtualization environment and the target thread by calling a dynamic link library. The dynamic link library includes the code of the virtualization environment and the code of the target service. The dynamic link library cannot be directly run, and needs to be run in the process when the process calls. The subject triggering the calling of the dynamic link library can be any one or more threads that have been run in the first process.

[0027] In the scheme, the code of the target service and the code of the virtualization environment are compiled into the form of a dynamic link library, so that another service process can call the dynamic link library to create a virtualization environment and run a thread providing the target service in the virtualization environment. Since the thread providing the target service is actually running in the memory space corresponding to the process, the thread and the process can be regarded as a whole, so that the thread and the process can access the same piece of memory space, and data read-write based on the same piece of memory space can be performed to realize data transmission between different services, and the processes such as serialization, protocol encoding and decoding, and deserialization do not need to be performed during data transmission, thereby effectively improving the data transmission efficiency.

[0028] In a second aspect, a data transmission apparatus is provided, comprising: a processing module configured to cause a first process to create a virtualization environment during runtime and a target thread running in the virtualization environment, the target thread configured to provide a target service, the target thread running in a memory space corresponding to the first process; an obtaining module configured to cause the first process to obtain a target address, the target address being a memory address accessible by both the first process and the target thread; and the processing module is further configured to cause the first process to read or write target data in the target address, wherein, in a case that the first process reads the target data in the target address, the target data is data written to the target address by the target thread after performing a data processing operation; and in a case that the first process writes the target data in the target address, the target data is data passed to the target thread by the first process.

[0029] In a possible implementation, the virtualization environment comprises a virtual machine or a sandbox, and the target address is an out-of-heap memory of the virtualization environment, the out-of-heap memory being located in the memory space corresponding to the first process.

[0030] In a possible implementation, the processing module is further configured to: perform compilation on bytecode of the target service and a library file relied on by a bytecode runtime to obtain machine code; and link the machine code with machine code of the virtualization environment to obtain a dynamic link library.

[0031] In a possible implementation, after the first process obtains the target data from the target address, the processing module is further configured to cause the first process to delete the target thread.

[0032] In a possible implementation, after the first process deletes the target thread, the processing module is further configured to cause the first process to: delete the virtualization environment; or release the virtualization environment to a resource pool, wherein the virtualization environment released to the resource pool only comprises code of the target service.

[0033] In a possible implementation, the target address is applied for by the target thread, and the obtaining module is further configured to cause the first process to:

[0034] After the target thread completes writing of the target data, the first process receives the target address sent by the target thread.

[0035] In a possible implementation, the target address is applied for by the first process, and the obtaining module is further configured to cause the first process to receive a notification message sent by the target thread, the notification message being configured to indicate that the target thread completes writing of the target data; and the first process obtains the target address previously passed to the target thread.

[0036] In a possible implementation, source code of the target service and source code of the first process are written based on different programming languages.

[0037] In a possible implementation, the first process is configured to provide a service different from the target service.

[0038] In a possible implementation, the first process can be configured to create the virtualization environment and the target thread by invoking a dynamic link library, the dynamic link library including code of the virtualization environment and code of the target thread.

[0039] In a third aspect, a data transmission apparatus is provided, which can include a processor, and a memory coupled to the processor, and the memory storing program instructions which, when executed by the processor, implement the method of the first aspect or any implementation of the first aspect. For the processor to execute the steps in each possible implementation of the first aspect, refer to the first aspect, which will not be repeated here.

[0040] In a fourth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores a computer program which, when executed on a computer, causes the computer to perform the method of any implementation of the first aspect.

[0041] In a fifth aspect, a circuit system is provided, and the circuit system includes a processing circuit configured to perform the method of any implementation of the first aspect.

[0042] In a sixth aspect, a computer program product is provided, and the computer program product, when executed on a computer, causes the computer to perform the method of any implementation of the first aspect.

[0043] In a seventh aspect, a chip system is provided, and the chip system includes a processor configured to support a server to implement the functions involved in any implementation of the first aspect, for example, to process the data and / or information involved in the method. In a possible design, the chip system further includes a memory configured to store program instructions and data necessary for the server. The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0044] The advantages of the second aspect to the seventh aspect can refer to the description of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0045] FIG. 1 is a process of implementing data transmission between services provided by the present application;

[0046] FIG. 2 is a schematic diagram of a system architecture provided by the present application;

[0047] FIG. 3 is a flowchart of a data transmission method provided by the present application;

[0048] FIG. 4 is a schematic diagram of data transmission between services in the related art;

[0049] FIG. 5 is a comparison diagram of a data transmission scheme provided by the present application and related technologies;

[0050] FIG. 6 is a flow diagram of compiling a dynamic link library provided by the present application;

[0051] FIG. 7 is a flow diagram of process A calling a dynamic link library provided by the present application;

[0052] FIG. 8 is a diagram of destroying a virtual machine provided by the present application;

[0053] FIG. 9 is a diagram of releasing a virtual machine to a resource pool provided by the present application;

[0054] FIG. 10A is a flow diagram of process A calling a Hudi engine to perform a read operation provided by the present application;

[0055] FIG. 10B is another flow diagram of process A calling a Hudi engine to perform a read operation provided by the present application;

[0056] FIG. 11 is a diagram of running a Hudi engine in a virtual machine provided by the present application;

[0057] FIG. 12A is a flow diagram of process A calling a Hudi engine to perform a write operation provided by the present application;

[0058] FIG. 12B is another flow diagram of process A calling a Hudi engine to perform a write operation provided by the present application;

[0059] FIG. 13 is a structural diagram of a data transmission device provided by the present application;

[0060] FIG. 14 is a structural diagram of a computing device provided by the present application;

[0061] FIG. 15 is a structural diagram of a computing device cluster provided by the present application;

[0062] FIG. 16 is another structural diagram of a computing device cluster provided by the present application;

[0063] FIG. 17 is a structural diagram of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments.

[0065] The terms "first", "second", "third", "fourth" and the like in the description and in the claims of the present application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms herein is to be construed to cover a generalised use of these terms to describe elements distinguishable from one another.

[0066] Moreover, the terms "comprising", "having", "including", and "containing" and any variations thereof used herein are intended to cover a non-exclusive inclusion, such that a process, method, system, product, or apparatus that comprises, has, includes or contains a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, system, product, or apparatus.

[0067] For the purpose of facilitating understanding, some technical terms related to the embodiments of the present application are introduced as follows.

[0068] (1) Microservice Architecture

[0069] Microservice Architecture (or simply microservices) is a cloud-native architecture that consists of numerous loosely coupled and independently deployable small services in a single application. In simple terms, microservice architecture is a software architecture pattern in which an application is broken down into small, autonomous service units that can be independently deployed, scaled, and maintained. Each service runs in its own process and can communicate with each other using inter-process communication mechanisms.

[0070] The core idea of microservice architecture is to break down an application into multiple small services, each with independent functionality and responsibility. Each service can be developed, tested, deployed, and scaled independently, improving the maintainability, scalability, and flexibility of the application. In addition, microservices allow different services to use different development languages, allowing developers to choose the development language that best suits their needs.

[0071] (2) Process

[0072] A process is a running activity of a program on a certain data set in a computer, and is the basic unit of resource allocation for the system and the basis of the structure of the operating system.

[0073] (3) Thread

[0074] A thread is the smallest unit of operation scheduling that can be performed by an operating system. It is contained in a process and is the actual operating unit in a process. A thread refers to a single sequence of control flow in a process. A process can have multiple threads running concurrently, each executing different tasks in parallel.

[0075] Generally, a process can contain multiple threads, and each thread can utilize the resources owned by the process. In an operating system that introduces threads, a process is usually taken as a basic unit of resource allocation, and a thread is taken as a basic unit of independent running and independent scheduling.

[0076] (4) Serialization

[0077] Serialization is a process of converting the state information of an object into a form that can be stored or transmitted. During serialization, an object writes its current state to a temporary or persistent storage. Later, the object can be recreated by reading or deserializing the state of the object from the storage.

[0078] (5) Deserialization

[0079] Deserialization refers to the process of decoding and restoring serialized data into the original data structure or object. Deserialization is the inverse process of serialization. In serialization, the state information of an object is converted into a byte stream for storage or transmission, while in deserialization, these byte streams will be reconverted into the state of the original object. Deserialization operations can be used in object recovery, distributed applications, remote calls, etc. scenarios, so that objects can restore their state from stored byte streams, or after transmission between different computers, the receiving party can convert the byte sequence into an operable object through the deserialization operation.

[0080] (6) Source code

[0081] Source code (also known as source program) refers to an uncompiled text file written in accordance with certain program design language specifications, which is a series of human-readable computer language instructions. In modern program languages, the most commonly used format of source code is a text file, and the purpose of this typical format is to compile a computer program. The ultimate goal of computer source code is to translate human-readable text into binary instructions that can be executed by a computer. This process is called compilation, which is completed through a compiler.

[0082] (7) Bytecode

[0083] Bytecode is a binary file containing an execution program composed of a sequence of op code / data pairs. Compared with machine instruction code that can be directly executed by hardware, bytecode is actually an intermediate code, i.e. instruction encoding that needs to be interpreted and executed by software code, and cannot be directly executed on hardware.

[0084] (8) Machine code

[0085] Machine code generally refers to machine instructions, which are instructions that can be directly recognized and executed by computer hardware (for example, a Central Processing Unit (CPU)), and its form is binary code. Machine code is usually composed of two parts: an operation code and an operand. The operation code indicates the operation to be performed by the machine code, that is, the function of the machine code; the operand indicates the object participating in the operation and the location where the operation result is stored, etc.

[0086] (9) Virtual machine

[0087] In this embodiment, a virtual machine is an important tool in a programming language (for example, a Java virtual machine), and is essentially a software module that can provide a running environment for code in a specific language and implement isolated running of the code.

[0088] In addition, a virtual machine instance refers to a virtual machine running on a hardware device (for example, a processor), that is, a virtual machine instance refers to a running virtual machine. Based on the same software code of a virtual machine, multiple virtual machine instances with the same configuration can be quickly created.

[0089] (10) Sandbox

[0090] In this application, a sandbox is an isolated environment for running program code, which can provide a running environment on which the program code depends and ensure that the running environment is isolated from the outside, thereby implementing isolated running of the program code.

[0091] (11) Dynamic link library

[0092] A dynamic link library is a way to implement the concept of a shared function library in an operating system. A dynamic link library appears to implement code reuse, and a dynamic link library actually includes some independent files, for example, functions that can be called by an executable program to complete some work. A dynamic link library is usually not directly executable, and only when a function in the dynamic link library is called by another module can the dynamic link library play a role.

[0093] (12) Heap memory

[0094] In the process of executing program code, a virtual machine divides the memory it manages in the main memory into multiple regions, each region storing different types of data, and the largest part of the space is the heap memory. The heap memory stores all object instances, and this region is created when the virtual machine starts and is shared by all threads in the virtual machine, and is the main working area of the garbage collector.

[0095] (13) Off-heap memory

[0096] Off-heap memory refers to memory space allocated and used outside the on-heap memory of a virtual machine. Off-heap memory is not managed by a garbage collector (GC) and is usually directly controlled by the developer of an application. Off-heap memory is mainly used to handle scenarios that require direct control over memory allocation. Allocation and release of off-heap memory are usually implemented through manual programming, for example, using a specific application programming interface (API).

[0097] Currently, under the micro-service architecture, services are run in the form of processes, so the communication between two services is essentially the communication between two processes. Moreover, two services often need to support cross-server deployment, so when two services perform data interaction across servers, serialization, protocol encoding and decoding, and deserialization processes often need to be performed during data transmission, resulting in low data transmission efficiency between services.

[0098] Referring to FIG. 1, FIG. 1 is a process of implementing data transmission between services provided by the present application. As shown in FIG. 1, service A runs in the form of process A, service B runs in the form of process B, and data transmission needs to be performed between service A and service B. In the data transmission process, the request object A in process A performs serialization on the request data to be transmitted, obtaining serialized data. Then, the serialized data is encoded according to the communication protocol, obtaining encoded data, and the encoded data is sent to process B. After process B obtains the encoded data, the encoded data is decoded and deserialized according to the communication protocol, so that the request object B can obtain the request data transmitted by process A.

[0099] After process B completes the data processing operation on the request data transmitted by process A and generates corresponding response data, the response object B in process B performs serialization on the response data to be transmitted, obtaining serialized data. Then, the serialized data is encoded according to the communication protocol, obtaining encoded data, and the encoded data is sent to process A. After process A obtains the encoded data, the encoded data is decoded and deserialized according to the communication protocol, so that the response object B can obtain the response data transmitted by process B.

[0100] That is, in the process of data transmission between services, serialization, protocol encoding and decoding, and deserialization processes need to be performed, resulting in low data transmission efficiency between services.

[0101] Therefore, the application provides a data transmission method. A process providing a certain service creates a virtualization environment during runtime, and runs a thread providing a target service in the virtualization environment. Since the thread providing the target service is actually running in the memory space corresponding to the process, the thread and the process can be considered as a whole, so that the thread and the process can access the same piece of memory space, and data transmission between different services can be implemented based on the address of the same piece of memory space by performing data reading and writing, without performing serialization, protocol encoding and decoding, and deserialization, thereby effectively improving the data transmission efficiency. That is, the essence of the present application is to transform the data interaction between two peer processes into the data interaction between a process and a thread in the process, thereby avoiding the cumbersome data transmission process between processes and improving the data transmission efficiency.

[0102] In addition, by running the thread providing the target service in the virtualization environment, the isolation of the thread can be achieved, and the running of the thread will not affect the running of the process itself (for example, the process itself will not crash after the thread crashes), thereby ensuring the independent running of different services.

[0103] Referring to FIG. 2, FIG. 2 is a schematic diagram of a system architecture provided by the application. As shown in FIG. 2, in the system architecture, the execution device 101 can be a physical host or a physical server, etc. The execution device 201 is in communication connection with the data storage system 202, and is used to obtain the program code stored in the data storage system 202, so as to implement the data transmission method provided by the application. The data storage system 202 can be implemented by a storage device deployed on the execution device 201, for example, the execution device 201 is a physical server, and the data storage system 202 is a hard disk deployed on the physical server. The data storage system 202 can also be implemented by a storage device independent of the execution device, for example, the execution device 201 is a computing server, and the data storage system 202 is a data server specially storing program code.

[0104] During the work, the execution device 201 can obtain the program code from the data storage system 202, and run the process on the execution device 201 and implement the data transmission in the process based on the data transmission method provided by the embodiment of the application.

[0105] Referring to FIG. 3, FIG. 3 is a flowchart of a data transmission method provided by the application. As shown in FIG. 3, the data transmission method includes the following steps 301-303.

[0106] In step 301, a virtualization environment is created in a first process during runtime, and a target thread running in the virtualization environment, the target thread is used to provide a target service, and the target thread runs in the memory space corresponding to the first process.

[0107] In the present application, the data transmission method can be applied to a micro-service architecture or other service architecture, for example. The first process can be a running form of a service in the micro-service architecture, i.e., the first process is used to provide a certain service in the micro-service architecture.

[0108] During the running of the first process, when the first process needs to obtain data provided by another service (i.e., a target service), the first process can call the dynamic link library, thereby implementing the running of the target service in the first process. The service provided by the first process is different from the target service. In addition, the subject triggering the calling of the dynamic link library can be any one or more threads that have been running in the first process. For example, when a certain thread in the first process needs to obtain data provided by the target service, the thread can trigger the calling of the dynamic link library.

[0109] Specifically, the dynamic link library includes the code of the virtualization environment and the code of the target service, and after the dynamic link library is called, the virtualization environment can be created and the target service can be run. In addition, the dynamic link library cannot be directly run, and needs to be called by a process to run in the process. In addition, the dynamic link library can be called by different processes, thereby running in different processes and providing corresponding services for different processes.

[0110] After the dynamic link library is called, the first process can create a virtualization environment based on the code of the virtualization environment included in the dynamic link library, and run the code of the target service in the virtualization environment, thereby implementing the running of the target thread for providing the target service in the virtualization environment. Since the target thread is created and run by the first process after the dynamic link library is called, the target thread is actually running in the memory space corresponding to the first process (i.e., the memory space allocated to the first process by the operating system), i.e., the running target thread is a thread belonging to the first process. Specifically, the target thread can include one or more threads, i.e., the one or more threads included in the target thread provide the target service. Alternatively, the first process can also create the virtualization environment and the target thread by calling the static link library, which is not limited in the present application.

[0111] After the first process runs the target thread through the virtualization environment, the first process can pass corresponding instructions to the target thread, so that the target thread can perform a data processing operation based on the instructions passed by the first process and obtain corresponding target data. Wherein, the first process can pass corresponding instructions to the target thread through the way of function call or the way of message notification. That is, the target thread performs a data processing operation based on the request data sent by the first process, and writes the target data (i.e. response data) obtained by performing the data processing operation into the target address, so that the first process obtains the target data from the target address.

[0112] Alternatively, the first process can pass corresponding instructions to the target thread to instruct the target thread to obtain the target data written by the first process into the target address from the target address specified by the first process.

[0113] In the case that the target thread runs in the virtualization environment, the target thread is actually running in isolation, that is, the running of the target thread does not affect the normal running of the first process. For example, in the case that the target thread abnormally crashes, it does not cause the first process to also abnormally crash. Therefore, the first process and the target thread can independently provide different services, and ensure that the services provided by the first process are not affected by the target thread, thereby ensuring that the target thread crashes still can provide normal services through the first process, and ensure that the scheme does not affect the normal running of the first process itself because of introducing the target thread in the first process.

[0114] Optionally, the virtualization environment described above can include a virtual machine or a sandbox. In this application, the virtualization environment can provide an isolation environment for the target thread on the one hand to ensure the isolated running between the target thread and the first process; on the other hand, the virtualization environment can provide a runtime environment for the target thread to ensure that the target thread can normally run, especially in the case that the source code of the first process and the source code corresponding to the target thread are written in different languages. Based on the virtualization environment, it can be ensured that the target thread can be executed smoothly in the first process.

[0115] Step 302, the first process obtains a target address, and the target address is a memory address that can be accessed by the first process and the target thread.

[0116] In the case that the target thread passes data to the first process, after the target thread completes writing the target data into the target address of the memory space, the first process can obtain the target address of the written target data, so as to take out the target data from the target address.

[0117] In the case that the first process transmits data to the target thread, the first process can also acquire the target address for writing the target data, so as to write the target data generated by the first process into the target address, and to facilitate the target thread to fetch the target data from the target address.

[0118] There can be multiple ways for the first process to acquire the target address. Hereinafter, the case that the target thread transmits data to the first process will be taken as an example to introduce how the first process acquires the target address.

[0119] In a possible implementation, the target address is applied for by the target thread.

[0120] For example, before performing the data processing operation, the target thread can apply for a piece of memory for writing the target data by calling an interface of the operating system, so as to obtain the target address allocated by the operating system in the memory space corresponding to the first process. In this way, in the process of performing the data processing operation, the target thread can continuously write the data obtained by performing the data processing operation into the target address, so that the target address can store the target data obtained by completing the data processing operation.

[0121] For another example, after completing the data processing operation, the target thread applies for a piece of memory for writing the target data to the operating system, and after the operating system returns the target address to the target thread, the target thread writes the target data obtained by performing the data processing operation into the target address.

[0122] In the case that the target address is applied for by the target thread, after the target thread completes the writing of the target data, the target thread can transmit the target address to the first process to indicate the address of the target data written by the target thread. That is, the first process acquires the target address by receiving the target address transmitted by the target thread. For example, the target thread can send a notification message to the first process, and the notification message includes the target address. The notification message is specifically used to indicate that the target thread has completed the writing of the target data and the target data is written in the target address. For another example, in the case that the first process transmits instructions to the target thread by means of function calling, the target thread can directly return the target address to the first process after completing the writing of the target data, so that the first process acquires the target address.

[0123] In the solution, the target thread applies for the target address for writing the target data and returns the target address to the first process after completing the writing of the target data, so that the first process can obtain the data transmitted by the target thread by reading the data in the memory space corresponding to the first process, thereby improving the data transmission efficiency. In addition, the target thread generating the target data applies for the target address (i.e., the data generator applies for the address for writing the data), so that the address can be applied on demand, the utilization rate of the memory space is improved, and the management of the memory space is facilitated.

[0124] In another possible implementation, the target address is applied for by the first process, and the first process needs to transmit the applied target address to the target thread.

[0125] For example, before the first process calls the target thread to execute the data processing operation, the first process can apply for the memory space for storing the data generated by the target thread by calling an operating system interface or other manners, so as to apply for the target address. Then, when the first process calls the target thread to execute the data processing operation, the first process can transmit the target address to the target thread to instruct the target thread to write the generated target data into the target address. In this way, in the process of executing the data processing operation, the target thread can continuously write the data obtained by executing the data processing operation into the target address. After the target thread completes the data processing operation, the first process can receive the notification message sent by the target thread, and the notification message is used to indicate that the target thread completes the writing of the target data. Based on the notification message, the first process obtains the target address transmitted to the target thread in advance, so as to determine the target data generated by the target thread from the target address.

[0126] Alternatively, in the case that the first process applies for the target address for the target thread in advance, the first process does not transmit the target address to the target thread when calling the target thread to execute the data processing operation. When the target thread executes the data processing operation, the target thread first stores the continuously generated data based on a cache space; after the target thread completes the data processing operation, the target thread returns the feedback message of the completed data processing to the first process. In this way, after confirming that the target thread completes the data processing, the first process transmits the target address applied in advance to the target thread, so that the target thread copies the cached target data to the target address. After the target thread completes the copying of the target data to the target address, the target thread sends a notification message to the first process, so as to notify the first process that the writing of the target data in the target address is completed.

[0127] In the scheme, the first process applies for the target address for writing the target data in advance and delivers the target address to the target thread, so that the target thread writes data according to the target address in the memory space provided by the first process, the first process can manage its own memory space, and the first process can quickly obtain the target data generated by the target thread, and the efficiency of data transmission is improved.

[0128] Similarly, in the case that the first process delivers data to the target thread, the target address can also be applied by the first process or the target thread.

[0129] Optionally, in the case that the virtualization environment is a virtual machine or a sandbox, the target address is a heap memory outside the virtualization environment. Specifically, although the first process runs the target thread in the first process by calling the dynamic link library, the target thread uses the memory of the first process, but since the target thread runs on the virtualization environment, the target thread uses the virtual memory address in the heap memory at runtime. For the first process, the first process does not recognize the virtual memory address used by the target thread at runtime, and the first process often needs additional cumbersome operations to access the virtual memory address. However, the heap memory outside the virtualization environment is directly accessible by the first process, so writing the target data into the heap memory outside the virtualization environment can facilitate the first process to quickly obtain the target data and improve the efficiency of data transmission.

[0130] Optionally, in the case of not using the heap memory outside for data transmission, a memory mapping technology can also be used to establish a mapping relationship between a certain block of heap memory in the virtualization environment and the actual memory address corresponding to the first process. In this way, the target thread can put the target data into the heap memory configured with the mapping relationship, so that the first process accesses the corresponding mapped actual memory address to obtain the target data.

[0131] In addition, the target address can also be an address outside the memory space corresponding to the first process, that is, the target address is not located in the memory space corresponding to the first process. For example, the target address can be a block of memory address outside the memory space corresponding to the first process, which is applied by the first process to the operating system and is specially used for data transmission between the first process and the target thread.

[0132] In step 303, the first process reads or writes the target data in the target address.

[0133] In the case that the first process reads the target data in the target address, the target data is the data written to the target address by the target thread after performing the data processing operation; in the case that the first process writes the target data in the target address, the target data is the data delivered by the first process to the target thread.

[0134] Specifically, in the case that the first process reads the target data in the target address, after obtaining the target address, the first process can obtain the target data stored in the target address by the target thread by reading the data in the target address. Since the first process and the target thread are respectively used to provide different services, after the first process obtains the target data stored in the target address by the target thread, it is equivalent to completing the data transmission process between the two services. And since the target thread is embedded into the first process in the form of calling the dynamic link library for execution, the first process can obtain the data generated by the target thread by directly accessing the memory allocated by itself, avoiding the cumbersome data transmission process such as serialization, protocol encoding and decoding, and deserialization, effectively improving the data transmission efficiency.

[0135] In the case that the first process writes the target data in the target address, the first process can write the target data needed to be transmitted to the target thread into the target address. Then, the first process can send an instruction to the target thread (such as sending an instruction in the form of function call) to instruct the target thread to read the target data transmitted by the first process to the target thread from the target address. In this way, when the target thread reads the target data from the target address based on the target address sent by the first process, the process of transmitting data from the first process to the target thread is completed.

[0136] Optionally, after the first process obtains the target data from the target address, the first process can delete the target thread. It can be understood that since the target thread used to provide the target service in the present application runs in the first process, the data generated by the target thread is mainly provided to the first process, and it is less likely to be provided to other processes. Therefore, after the first process obtains the target data generated by the target thread, the target thread has completed the work instructed by the first process, and the first process temporarily no longer needs to obtain the data generated by the target thread. In this way, the first process can delete the target thread, thereby freeing up idle computing resources and memory resources, which helps to save the computing resources and memory resources of the device.

[0137] Optionally, after the first process deletes the target thread, the first process can delete the virtualization environment, for example, the first process deletes the virtual machine or sandbox used to run the target thread. Since the virtualization environment is mainly used to run the target thread for providing the target service, after deleting the target thread, the virtualization environment has no actual use, and then the virtualization environment can also be deleted, thereby freeing up more computing resources and memory resources.

[0138] Alternatively, in the case that the computing resource and the memory resource are sufficient, the first process can release the virtualization environment to the resource pool. Among them, the virtualization environment released to the resource pool only includes the code of the target service. That is, the virtualization environment is not deleted, but only the target thread originally running in the virtualization environment is deleted, and the code of the target service is still retained in the virtualization environment, and the running of the target thread can be realized by running the code of the target service. In the case of releasing the virtualization environment to the resource pool, if the first process needs to continue to obtain the data generated by the target thread subsequently, the virtualization environment can be directly obtained from the resource pool and the target thread can be quickly started in the virtualization environment, without the need to perform the process of calling the dynamic link library to create the virtualization environment, so as to improve the efficiency of starting the target service in the first process.

[0139] The above introduces the process of running the target thread providing the target service by the first process through calling the dynamic link library, and the following will introduce the process of how to generate the dynamic link library.

[0140] In the process of generating the dynamic link library, the source code of the target service can be compiled first to obtain the bytecode; or the bytecode of the target service provided by the developer is directly obtained. Then, the bytecode of the target service and the library file relied on by the bytecode runtime are compiled to obtain the machine code. Finally, the compiled machine code and the machine code of the virtualization environment are compiled and linked to obtain the dynamic link library. Among them, the machine code of the virtualization environment can be provided by the developer, or can be obtained by compiling the source code of the virtualization environment provided by the developer. That is, the dynamic link library obtained by compiling and linking actually includes the machine code of the target service, the machine code of the library file relied on by the target service, and the machine code of the virtualization environment.

[0141] In the present scheme, since the machine code can be directly recognized and run by the computer hardware, after the first process calls the dynamic link library, the creation of the virtualization environment and the starting of the target thread can be realized by directly running the machine code in the dynamic link library, without the need to perform the compiling process of the bytecode, so as to improve the starting performance of the target thread.

[0142] Of course, in some possible embodiments, when generating the dynamic link library, the bytecode of the target service and the machine code of the virtualization environment (such as a virtual machine) can be compiled and linked together, so that the virtual machine runs the bytecode of the target service again at runtime, and then the starting of the target thread is realized.

[0143] Optionally, the source code of the target service and the source code of the first process are written based on different programming languages. Since the service provided by the first process is not the same as the target service, and different services tend to use corresponding programming languages according to actual needs during development, the source code of the target service and the source code of the first process are likely to be written based on different programming languages. For example, the source code of the first process is written based on the C++ language, and the source code of the target service can be written based on the Java language or the Python language.

[0144] In the case where the source code of the target service and the source code of the first process are written based on different programming languages, it is often difficult to directly embed the target service in the first process, so in this application, a virtualization environment is created by calling a dynamic link library, and the target service is run in the virtualization environment to provide a runtime-dependent environment for the target service, and then the code of the target service is run in the virtualization environment, so as to start and run the target thread providing the target service, and ensure that the thread written based on other programming languages is run in the first process.

[0145] The above introduces a data transmission method provided by the present application. In order to facilitate understanding, the specific execution process of the data transmission method under the micro-service architecture will be introduced in detail below with specific examples.

[0146] Specifically, in the case of rapid development of virtualization and cloud computing, the data flow between artificial intelligence (AI) and big data is often cross-service. When data is transmitted between different services, the data sending end often needs to perform serialization, encoding and transmission on the data, and the data receiving end needs to perform decoding and deserialization on the received data, thereby bringing a long time overhead, resulting in low efficiency of data transmission between services. In performance-sensitive scenarios, low data transmission efficiency between services will bring poor user experience. Based on this, in view of the problem of low transmission efficiency between services, the present application provides a data transmission method to improve the efficiency of data transmission between services.

[0147] Illustratively, please refer to FIG. 4, which is a schematic diagram of a data transmission between services in the related art. As shown in FIG. 4, in the AI, big data or database scenarios, the big data analysis service runs in the form of process A, the data read-write service runs in the form of process B, and process A needs to read and write data in the database through process B. Wherein, process A and process B are independently deployed, process A is used to provide big data analysis service, and process B is used to complete data writing and reading. Process A and process B can communicate through remote procedure call (RPC), so that process A can perform data read-write on the database through process B.

[0148] For example, refer to FIG. 5, which is a comparison diagram of a data transmission scheme provided by the present application and related technologies. As shown in FIG. 5, in related technologies, two different services are actually run in independent processes, and the data transmission between the services is implemented based on inter-process communication. In the data transmission scheme provided by the present application, the code of the data read-write service and the code of the virtual machine or sandbox are compiled and linked into a dynamic link library, and the big data analysis service is run in the form of process A, so that process A can run the data read-write service in the form of a thread in process A by calling the dynamic link library during running, and thus the data transmission between the two services is converted into a memory access operation within the process.

[0149] Specifically, refer to FIG. 6, which is a flowchart of compiling a dynamic link library provided by the present application. As shown in FIG. 6, in the present application, Hudi is a component of big data and can be used to provide data read-write services. For the Hudi source code written in Java, the Hudi source code can be first compiled into a Java Class file, i.e., Java bytecode, using a Java compiler. Optionally, in the case where the Hudi bytecode is provided by the developer, the compilation process of the source code can no longer be performed, and the subsequent compilation process can be directly performed on the Hudi bytecode.

[0150] Then, the Java bytecode and its dependent library files (Libraries) are compiled into machine code using a compiler (e.g., GraalVM Native Image Generator). That is, the Hudi bytecode is subjected to ahead-of-time (AOT) compilation, so as to compile the Hudi bytecode into machine code.

[0151] After obtaining the machine code of Hudi, the machine code is compiled and linked with the code of the virtual machine (e.g., the code of Substate VM) and the Java Development ToolKit (JDK) to generate a dynamic link library Hudi.so. The code of the virtual machine can include the code of the virtual machine Substate VM and the code of the execution environment of the components and libraries dependent by the Hudi runtime. The Java Development ToolKit includes the Java runtime environment, Java tools, and Java basic class library, etc.

[0152] Finally, process A creates a virtual machine inside process A and runs the Hudi thread in the virtual machine by calling the dynamic link library at runtime. That is, the Hudi component is actually embedded into the process as a dynamic link library Hudi.so, so as to convert the communication between process A and the Hudi process in the related art into the communication between process A and the dynamic link library Hudi.so.

[0153] Referring to FIG. 7, FIG. 7 is a flowchart of process A calling a dynamic link library according to an embodiment of the present application. As shown in FIG. 7, after the dynamic link library is compiled, the dynamic link library Hudi.so can be called by the running process A, so that process A can complete a data table reading service provided by Hudi once. Specifically, the process of process A calling the dynamic link library can include the following steps 701-706.

[0154] Step 701, process A calls the dynamic link library Hudi.so.

[0155] During the running of process A, when process A needs to read and write data in the database, process A can call the dynamic link library Hudi.so.

[0156] Step 702, process A creates a virtual machine based on the dynamic link library Hudi.so.

[0157] Since the dynamic link library Hudi.so includes the code of the virtual machine, process A can create the virtual machine by running the code of the virtual machine. In the present application, the virtual machine created by process A is a lightweight virtual machine, for example, a Substrate VM. The lightweight virtual machine starts quickly, has a certain degree of isolation, and has a necessary runtime environment, but generally does not have an operating system.

[0158] Step 703, process A initializes the Hudi engine in the virtual machine.

[0159] Based on the machine code of Hudi, process A can initialize the Hudi engine in the virtual machine. The Hudi engine is run in the form of a thread in process A. That is, the Hudi engine corresponds to the target thread in the above embodiment, and process A corresponds to the first process in the above embodiment.

[0160] It should be noted that in the related art, the Hudi engine is Java native and runs on the Java virtual machine, and the starting step is relatively more (for example, it includes loading the Java virtual machine executable file, loading the class from the file system, verifying the Hudi bytecode, interpreting the Hudi bytecode, running the static initializer, compiling the Hudi bytecode, and running the compiled machine code). Therefore, the starting time of the Hudi engine in the related art is often seconds (measured 1440ms). In this application, the dynamic link library Hudi.so is obtained after AOT compilation, so when starting the Hudi instance, only the executable file needs to be loaded, and the optimized machine code can be started immediately, greatly reducing the starting time (measured 78ms), and ensuring the efficiency of service response.

[0161] In addition, in addition to the optimization of the starting time, the application implementation can also achieve fine management of the memory used by Hudi. In the Java native scheme, each Hudi instance shares the memory of the Java virtual machine process, and cannot be finely managed. For example, assuming that the Java virtual machine is allocated 32G memory, then no matter how many Hudi instances are shared, only 32G memory can be shared, so that in the case of a large number of Hudi instances, memory overflow occurs, causing Hudi to malfunction. Based on the dynamic link library Hudi.so started by the AOT compiled Hudi instance, each Hudi instance runs on an independent virtual machine, and each virtual machine is independently allocated memory, thereby performing fine management of the memory usage of the Hudi instance.

[0162] Step 704, process A calls the Hudi engine to perform read-write operations.

[0163] After the Hudi engine completes the initialization operation, process A can call the Hudi engine to perform read-write operations through function calls or message notification. In this way, the Hudi engine can perform corresponding data read-write operations on the database based on the indication of process A.

[0164] Step 705, after completing the read-write operation, process A closes the Hudi engine.

[0165] After completing the read-write operation, process A first releases the resources (such as off-heap memory) applied by the Hudi engine, and then closes the Hudi engine, thereby releasing all resources related to the Hudi engine.

[0166] Step 706, process A destroys the virtual machine or releases the virtual machine to the resource pool.

[0167] After the Hudi engine is closed, the process A can destroy the virtual machine, thereby releasing all resources related to the virtual machine. Alternatively, the process A can release the virtual machine to a resource pool, so that when the Hudi engine needs to be called later, the virtual machine in the resource pool can be applied for and the Hudi engine can be quickly started in the virtual machine without the need to call the dynamic link library Hudi.so again.

[0168] Exemplarily, referring to FIG. 8, FIG. 8 is a schematic diagram of destroying a virtual machine provided by the present application. As shown in FIG. 8, after the dynamic link library Hudi.so is called, the process A can create one or more virtual machines (such as virtual machine 1 and virtual machine 2 in FIG. 8), and different virtual machines can run different Hudi instances. In any virtual machine, there is a heap memory for storing a Hudi image (i.e., the machine code of Hudi), a runtime heap memory for running the Hudi image, and a reserved memory space. After a virtual machine (for example, virtual machine 2 in FIG. 8) is destroyed, all the memory occupied by the virtual machine is released, and the virtual machine itself is also deleted.

[0169] Referring to FIG. 9, FIG. 9 is a schematic diagram of releasing a virtual machine to a resource pool provided by the present application. As shown in FIG. 9, when the virtual machine 2 created in the process A is released to the resource pool, the virtual machine 2 will still occupy a certain memory (i.e., the Hudi image heap memory) to save the Hudi image. In this way, when the process A still needs to call the Hudi engine to perform a data read / write operation, a virtual machine with a space can be applied for in the resource pool, and the Hudi engine can be quickly started based on the Hudi image reserved in the virtual machine, thereby avoiding the process of re-creating a virtual machine.

[0170] Referring to FIG. 10A, FIG. 10A is a flowchart of the process A calling the Hudi engine to perform a read operation provided by the present application. As shown in FIG. 10A, the step 704 can specifically include the following steps 7041-7044.

[0171] Step 7041, the process A sends a data read request to the Hudi engine.

[0172] When the process A calls the Hudi engine, a data read request can be sent to the Hudi engine to request the Hudi engine to read data in a certain data table in the database (i.e., the target data introduced in the above embodiment).

[0173] Step 7042, the Hudi engine applies off-heap memory and writes the data read from the database to the off-heap memory.

[0174] After obtaining the data read request, the Hudi engine first applies for the off-heap memory by calling the operating system interface. Since the Hudi engine is embedded in the process A, the off-heap memory applied for by the Hudi engine is also the memory of the process A. Based on the off-heap memory applied for, the Hudi engine reads the corresponding data from the database and writes the read data into the off-heap memory.

[0175] Exemplarily, refer to FIG. 11, which is a schematic diagram of running the Hudi engine in the virtual machine provided in the present application. As shown in FIG. 11, in addition to the memory allocated to the virtual machine, the Hudi engine also applies for the off-heap memory, which is located outside the memory managed by the virtual machine but still in the memory space allocated to the process A by the operating system.

[0176] In step 7043, the Hudi engine returns the address of the off-heap memory to the process A.

[0177] After the Hudi engine completes the reading of the data, the Hudi engine can return the address of the off-heap memory (i.e., the target address introduced in the above embodiment) to the process A, so that the process A can obtain the data based on the address of the off-heap memory.

[0178] In step 7044, the process A reads the data in the off-heap memory.

[0179] After obtaining the address of the off-heap memory, since the off-heap memory belongs to the memory space corresponding to the process A, the process A can directly read the data in the off-heap memory based on the address of the off-heap memory.

[0180] It should be noted that the embodiment corresponding to FIG. 10A introduces that when the process A reads the data of the database through the Hudi engine, the Hudi engine applies for the off-heap memory and returns the address of the off-heap memory to the process A. In some possible embodiments, the process A can also apply for the off-heap memory and return the address of the off-heap memory to the Hudi engine.

[0181] Exemplarily, refer to FIG. 10B, which is another flowchart of the process A calling the Hudi engine to perform the reading operation provided in the present application. As shown in FIG. 10B, the difference between the embodiment shown in FIG. 10B and the embodiment shown in FIG. 10A is that the process A first applies for the memory address to the operating system, then sends the data read request to the Hudi engine and carries the applied memory address in the data read request; in this way, the Hudi engine can write the data read from the database into the memory address based on the memory address carried in the data read request. Finally, the Hudi engine returns the notification message to the process A to indicate that the Hudi engine has written the data into the off-heap memory, thereby triggering the process A to read the data in the memory address.

[0182] Referring to FIG. 12A, FIG. 12A is a flowchart of a process A invoking a Hudi engine to perform a write operation according to an embodiment of the present application. As shown in FIG. 12A, the step 704 can include the following steps 7045-7048.

[0183] At step 7045, the process A applies for a memory address.

[0184] When the process A needs to write data to a database through the Hudi engine, the process A can first apply for a memory address through an operating system interface, so as to write the data to be written to the database into the memory address and pass the data to the Hudi engine.

[0185] At step 7046, the process A writes the data to be written to the database into the applied memory address.

[0186] After the memory address is applied, the process A writes the data to be written to the database into the applied memory address (i.e., the target address).

[0187] At step 7047, the process A sends a data write request to the Hudi engine, and carries the applied memory address in the data write request.

[0188] After the data is written into the memory address, the process A can send a data write request to the Hudi engine, and carry the applied memory address in the data write request, so as to instruct the Hudi engine to write the data in the memory address to the database.

[0189] At step 7048, the Hudi engine reads the data in the memory address, and writes the read data to the database.

[0190] After receiving the data write request, the Hudi engine can read the data in the memory address based on the memory address carried in the data write request, and write the read data to the database, so as to complete the data write operation.

[0191] It should be noted that the embodiment corresponding to FIG. 12A introduces that when the process A writes data to the database through the Hudi engine, the process A applies for a memory address and returns the memory address to the Hudi engine. In some possible embodiments, the Hudi engine can also apply for a memory address and return the applied memory address to the process A.

[0192] Exemplarily, please refer to FIG. 12B, which is another flowchart provided by the present application, showing a process A calling a Hudi engine to perform a write operation. As shown in FIG. 12B, the difference between the embodiment shown in FIG. 12B and the embodiment shown in FIG. 12A is that the process A sends a data write request to the Hudi engine; then, the Hudi engine applies for off-heap memory and returns the address of the applied off-heap memory to the process A. In this way, the process A can write the data to be written into the database into the off-heap memory applied by the Hudi engine, and send a notification message to the Hudi engine to indicate that the process A has completed writing data into the off-heap memory. Finally, the Hudi engine can read the data in the off-heap memory and write the read data into the database.

[0193] In the present application, a lightweight virtual machine is used to embed a certain service into the process corresponding to another service, which essentially avoids the transfer of data between different systems, and therefore has a wide range of use scenarios in the fields of big data, AI, databases, etc. In addition, in the near-data computing scenario, efficient data transfer can also be achieved. For example, assuming that in the near-data computing scenario, the data on the storage server needs to be transmitted to a certain computing server for operation, at this time, the scheme provided by the present application can be used to directly start a lightweight virtual machine on the storage server, and deploy the corresponding computing service on the computing server on the lightweight virtual machine to complete the corresponding computation, thereby avoiding the flow of data between two systems and saving the data transmission time. It should be noted that in the above example, the scheme of the present application is used to shorten the data transmission time between servers, and the scheme of the present application can also be applied between chips or other devices.

[0194] The above describes in detail the method provided by the present application. Next, the device provided by the present application for executing the above method will be introduced.

[0195] Please refer to FIG. 13, which is a structural schematic diagram of a data transmission device provided by the present application. As shown in FIG. 13, the data transmission device comprises: a processing module 1301, configured to cause a target thread to be created in a virtualization environment during the running of a first process and run in the virtualization environment, the target thread being configured to provide a target service, and the target thread running in a memory space corresponding to the first process; an acquisition module 1302, configured to cause the first process to acquire a target address, the target address being a memory address accessible by both the first process and the target thread; and the processing module 1301 is further configured to cause the first process to read or write target data in the target address; wherein, in the case that the first process reads the target data in the target address, the target data is data written into the target address by the target thread after performing a data processing operation; and in the case that the first process writes the target data in the target address, the target data is data passed by the first process to the target thread.

[0196] In a possible implementation, the virtualization environment includes a virtual machine or a sandbox, and the target address is an out-of-heap memory of the virtualization environment, which is located in a memory space corresponding to the first process.

[0197] In a possible implementation, the processing module 1301 is further configured to perform compilation on the bytecode of the target service and a library file relied on by the bytecode runtime to obtain machine code; and perform compilation and linking of the machine code and machine code of the virtualization environment to obtain a dynamic link library.

[0198] In a possible implementation, after the first process obtains the target data from the target address, the processing module 1301 is further configured to cause the first process to delete the target thread.

[0199] In a possible implementation, after the first process deletes the target thread, the processing module 1301 is further configured to cause the first process to delete the virtualization environment, or cause the first process to release the virtualization environment to a resource pool, wherein the virtualization environment released to the resource pool only includes the code of the target service.

[0200] In a possible implementation, the target address is applied for by the target thread, and the obtaining module 1302 is further configured to cause the first process to receive the target address sent by the target thread.

[0201] After the target thread completes writing of the target data, the first process receives the target address sent by the target thread.

[0202] In a possible implementation, the target address is applied for by the first process, and the obtaining module 1302 is further configured to cause the first process to receive a notification message sent by the target thread, the notification message being used to indicate that the target thread completes writing of the target data; and cause the first process to obtain the target address previously passed to the target thread.

[0203] In a possible implementation, the source code of the target service and the source code of the first process are written based on different programming languages.

[0204] In a possible implementation, the first process is configured to provide a service different from the target service.

[0205] In a possible implementation, the first process can create the virtualization environment and the target thread by calling the dynamic link library, and the dynamic link library includes the code of the virtualization environment and the code of the target thread.

[0206] The obtaining module 1302 or the processing module 1301 can be implemented by software or hardware. For example, the implementation of the processing module 1301 is described below. Similarly, the implementation of the obtaining module 1302 can refer to the implementation of the processing module 1301.

[0207] As an example of the software function unit, the processing module 1301 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the A module can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers running the code can be distributed in the same region, or in different regions. Further, the multiple hosts / virtual machines / containers running the code can be distributed in the same availability zone (AZ), or in different AZs. Each AZ includes a data center or multiple data centers in a similar geographical location. Generally, one region can include multiple AZs.

[0208] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed in the same virtual private cloud (VPC), or in multiple VPCs. Generally, one VPC is set in one region, and communication between two VPCs in the same region or between VPCs in different regions needs to be realized through a communication gateway set in each VPC.

[0209] As an example of a hardware functional unit, the processing module 1301 can include at least one computing device, such as a server or the like. Alternatively, the processing module 1301 can also be a device implemented by a central processing unit (CPU), or implemented by an application-specific integrated circuit (ASIC), or implemented by a programmable logic device (PLD), and the like. The PLD can be implemented by a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system on chip (SoC), an offload card, an acceleration card, or any combination thereof.

[0210] The multiple computing devices included in the processing module 1301 can be distributed in the same region or in different regions. The multiple computing devices included in the processing module 1301 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the processing module 1301 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offload cards, acceleration cards, and the like.

[0211] Referring to FIG. 14, FIG. 14 is a structural schematic diagram of a computing device provided by the present application. The computing device 1400 shown in FIG. 14 can be used to execute the data transmission method provided by the present embodiment. As shown in FIG. 14, the computing device 1400 includes a bus 1402, a processor 1404, a memory 1406, and a communication interface 1408. The processor 1404, the memory 1406, and the communication interface 1408 communicate through the bus 1402. The computing device 1400 can be a server or a terminal device. It should be understood that the number of processors and memories in the computing device 1400 is not limited by the present application.

[0212] The bus 1402 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one line is represented in FIG. 14, but it does not mean that there is only one bus or only one type of bus. The bus 1402 can include a path for transmitting information between various components (e.g., the memory 1406, the processor 1404, the communication interface 1408) of the computing device 1400.

[0213] The processor 1404 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), or the like.

[0214] The memory 1406 can include a volatile memory (e.g., a random access memory (RAM)), and the processor 1404 can further include a non-volatile memory (e.g., a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD)).

[0215] The memory 1406 stores executable program code, and the processor 1404 executes the executable program code to respectively implement the functions of the foregoing acquisition module and processing module, thereby implementing the model processing method described above. That is, the memory 1406 has instructions for executing the model processing method.

[0216] The communication interface 1408 uses a transceiver module such as, but not limited to, a network interface card, a transceiver, and the like, to implement communication between the computing device 1400 and other devices or communication networks.

[0217] It should be understood that the computing device 1400 according to the present application is used to execute the data transmission method as shown in FIGS. 3 to 12B, and can correspond to the execution of the corresponding subject in the method according to the present application. For the sake of brevity, it will not be repeated here.

[0218] The application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0219] Referring to FIG. 15, FIG. 15 is a structural schematic diagram of a computing device cluster provided by the application. As shown in FIG. 15, the computing device cluster includes at least one computing device 1400. The memory 1406 in one or more computing devices 1400 in the computing device cluster can store the same instructions for performing the data transmission method.

[0220] In some possible implementations, the memory 1406 in one or more computing devices 1400 in the computing device cluster can also respectively store partial instructions for performing the data transmission method. In other words, the combination of one or more computing devices 1400 can collectively execute the instructions for performing the data transmission method.

[0221] It should be noted that the memory 1406 in different computing devices 1400 in the computing device cluster can store different instructions, respectively used for performing partial functions of the data processing apparatus. That is, the instructions stored in the memory 1406 in different computing devices 1400 can implement the functions of one or more of the above-mentioned obtaining module and processing module.

[0222] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc. FIG. 16 shows a possible implementation. FIG. 16 is a structural schematic diagram of another computing device cluster provided by the application. As shown in FIG. 16, in the computing device cluster 1600, two computing devices 1400A and 1400B are connected through a network. Specifically, the communication interface in each computing device is connected to the network. In this type of possible implementation, the memory 1406 in the computing device 1400A stores instructions for performing the functions of the obtaining module. Meanwhile, the memory 1406 in the computing device 1400B stores instructions for performing the functions of the processing module.

[0223] It should be understood that the functions of the computing device 1400A shown in FIG. 16 can also be completed by multiple computing devices 1400. Similarly, the functions of the computing device 1400B can also be completed by multiple computing devices 1400.

[0224] It should be appreciated that the computing device 1400 or the computing device cluster 1600 in the present application can correspond to the data transmission apparatus in the present application of FIG. 13, and can correspond to the respective subject performing the data transmission method in the present application of FIG. 3. Also, the above and other operations and / or functions of the respective modules in the computing device 1400 or the computing device cluster 1600 are respectively for implementing the respective flows of the data transmission method in the present application of FIG. 3, and for brevity, will not be repeated here.

[0225] Referring to FIG. 17, there is shown a structural diagram of a computer readable storage medium according to the present application. The present application also provides a computer readable storage medium, in some embodiments, the method disclosed in FIG. 3 above can be implemented as computer program instructions encoded in a computer readable storage medium or other non-transitory medium or article in a machine readable format.

[0226] FIG. 17 schematically illustrates a conceptual partial view of an example computer readable storage medium arranged to include a computer program for execution on a computing device to perform a computer process, according to at least some embodiments presented herein. In one embodiment, the computer readable storage medium 1700 is provided using a signal bearing medium 1701. The signal bearing medium 1701 can include one or more program instructions 1702, which, when executed by one or more processors, can provide at least some of the functionality described above with respect to FIG. 3.

[0227] In some examples, the signal bearing medium 1701 can embody a computer readable medium 1703, such as, but not limited to, a hard disk drive, a compact disk (CD), a digital video disk (DVD), a digital tape, memory, ROM, RAM, etc.

[0228] In some implementations, the signal bearing medium 1701 can embody a computer recordable medium 1704, such as, but not limited to, memory, a read / write (R / W) CD, a R / W DVD, etc. In some implementations, the signal bearing medium 1701 can embody a communication medium 1705, such as, but not limited to, a digital and / or an analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communication link, a wireless communication link, etc.). Thus, for example, the signal bearing medium 1701 can be conveyed by a wireless form of the communication medium 1705 (e.g., a wireless communication medium complying with the IEEE 1602.X standards or other transmission protocols).

[0229] The one or more program instructions 1702 can be, for example, computer-executable instructions or logic-implementing instructions. In some examples, a computing device of a computing device can be configured to provide various operations, functions, or actions in response to the program instructions 1702 being communicated to the computing device by one or more of the computer-readable medium 1703, the computer-recordable medium 1704, and / or the communication medium 1705.

[0230] It should be understood that the computer-readable storage medium 1700 in the present application can be deployed on the data transmission apparatus introduced in FIG. 13, or the computing device 1400 or the computing device cluster 1600 of the present application. In this way, the data transmission apparatus, the computing device 1400 or the computing device cluster 1600 provided by the present application realizes the data transmission method introduced in FIG. 3 by reading the one or more program instructions 1702 on the computer-readable storage medium 1700.

[0231] In addition, it should be noted that the apparatus embodiments described above are only schematic, and units described as separate components can or can not be physically separate, and components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the apparatus embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0232] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary general hardware, and of course can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and specific hardware structures for implementing the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including a plurality of instructions for causing a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods of various embodiments of the present application.

[0233] In the above-described embodiments, all or part can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product.

[0234] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the procedures or functions according to the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can store or be integrated into a data storage device such as a training device, a data center, etc. containing one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0235] The above is only a specific embodiment of the present application. Those skilled in the art can think of changes or replacements based on the specific embodiments provided by the present application, which should be covered within the protection scope of the present application.

Claims

1. A data transmission method, characterized by, The method comprises: a first process creates a virtualization environment and a target thread running in the virtualization environment during running, the target thread is used to provide a target service, and the target thread runs in a memory space corresponding to the first process; the first process obtains a target address, the target address being a memory address accessible to the first process and the target thread; the first process reads or writes target data in the target address; in a case where the first process reads the target data in the target address, the target data is data written to the target address by the target thread after performing a data processing operation; in a case where the first process writes the target data in the target address, the target data is data transmitted by the first process to the target thread.

2. The method of claim 1, wherein, The virtualization environment comprises a virtual machine or a sandbox, the target address is a heap memory of the virtualization environment, and the heap memory is located in the memory space corresponding to the first process.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: performing compilation on bytecode of the target service and a library file relied on by the bytecode runtime to obtain machine code; linking the machine code with machine code of the virtualization environment to obtain the dynamic link library.

4. The method according to any one of claims 1 to 3, characterized in that, After the first process obtains the target data from the target address, the method further comprises: the first process deletes the target thread.

5. The method of claim 4, wherein, After the first process deletes the target thread, the method further comprises: the first process deletes the virtualization environment; or, the first process releases the virtualization environment to a resource pool, wherein the virtualization environment released to the resource pool only comprises code of the target service.

6. The method according to any one of claims 1 to 5, characterized in that, The target address is applied for by the target thread, and the first process obtains the target address, comprising: after the target thread completes writing of the target data, the first process receives the target address sent by the target thread.

7. The method according to any one of claims 1 to 5, characterized in that, The target address is applied for by the first process, and the first process obtains the target address, comprising: the first process receives a notification message sent by the target thread, the notification message being used to indicate that the target thread completes writing of the target data; the first process obtains the target address transmitted to the target thread in advance.

8. The method according to any one of claims 1 to 7, characterized in that, Source code of the target service and source code of the first process are written based on different programming languages.

9. The method according to any one of claims 1 to 8, characterized in that, The first process is used to provide a service different from the target service.

10. The method according to any one of claims 1 to 9, characterized in that, The first process creates the virtualization environment and the target thread by calling a dynamic link library, the dynamic link library comprising code of the virtualization environment and code of the target thread.

11. A data transmission apparatus, characterized by comprising: The method comprises: a processing module is configured to cause a first process to create a virtualization environment and a target thread running in the virtualization environment during running, the target thread being used to provide a target service, and the target thread running in a memory space corresponding to the first process. The acquisition module is configured to cause the first process to acquire a target address, the target address being a memory address accessible to both the first process and the target thread. The processing module is further configured to cause the first process to read or write target data in the target address. In a case where the first process reads the target data in the target address, the target data is data written to the target address by the target thread after performing a data processing operation. In a case where the first process writes the target data in the target address, the target data is data passed to the target thread by the first process.

12. The apparatus of claim 11, wherein, The virtualization environment includes a virtual machine or a sandbox, and the target address is an out-of-heap memory of the virtualization environment, the out-of-heap memory being located in a memory space corresponding to the first process.

13. The apparatus of claim 11 or 12, wherein, The processing module is further configured to: perform compilation on bytecode of the target service and a library file relied on by the bytecode runtime to obtain machine code; and link the machine code with machine code of the virtualization environment to obtain the dynamic link library.

14. The apparatus of any one of claims 11-13, wherein, After the first process acquires the target data from the target address, the processing module is further configured to cause the first process to delete the target thread.

15. The apparatus of claim 14, wherein, After the first process deletes the target thread, the processing module is further configured to cause: the first process to delete the virtualization environment; or the first process to release the virtualization environment to a resource pool, wherein the virtualization environment released to the resource pool only includes code of the target service.

16. The apparatus of any one of claims 11-15, wherein, The target address is applied for by the target thread, and the acquisition module is further configured to cause: the first process to receive the target address sent by the target thread after the target thread completes writing of the target data.

17. The apparatus of any one of claims 11-15, wherein, The target address is applied for by the first process, and the acquisition module is further configured to cause: the first process to receive a notification message sent by the target thread, the notification message being used to indicate that the target thread completes writing of the target data; and the first process to acquire the target address passed to the target thread in advance.

18. The apparatus of any of claims 11-17, wherein, The source code of the target service and the source code of the first process are written based on different programming languages.

19. The apparatus of any of claims 11-18, wherein, The first process is used to provide a service different from the target service.

20. The apparatus of any of claims 11-19, wherein, The first process creates the virtualization environment and the target thread by calling a dynamic link library, the dynamic link library including code of the virtualization environment and code of the target thread.

21. A data transmission apparatus, characterized by comprising: The apparatus includes a memory and a processor, the memory stores code, and the processor is configured to execute the code, when the code is executed, the apparatus performs the method of any one of claims 1 to 10.

22. A computer storage medium, comprising, The computer storage medium stores instructions, when the instructions are executed by a computer, the computer implements the method of any one of claims 1 to 10.

23. A computer program product, characterised in that, The computer program product stores instructions, when the instructions are executed by a computer, the computer implements the method of any one of claims 1 to 10.

Images