A system and method for quickly materializing Java objects from database data

Abstract

The present invention relates to a device for extracting data from a database for object initialization, a device for writing object data into a database, a data extraction method, and a method for writing data into a database. The present invention facilitates fast and memory-efficient data exchange between a native side and a virtual machine (VM) side. In particular, the data extraction kernel on the native side uses a database interface to read database data from multiple locations in the database, and uses a native side buffer application programming interface (API) to write the database data into a buffer. The object initialization kernel on the VM side uses the native side buffer API to read the database data from the buffer and initialize multiple objects. Information indicating one of the multiple storage areas in the buffer in which the data extraction kernel writes the database data is known to the native side buffer API and the VM side buffer API.

Description

[0001] The invention relates to a system and method for efficiently materializing objects from database data. Background Art

[0002] The embodiments of the present invention relate to the field of computer software engineering, application servers and database systems, and more particularly to a system and method for efficiently materializing Java objects from database data, which is necessary in the implementation of object-relational mapping (ORM).

[0003] There are many existing ORM solutions for Java in general (e.g. "Hibernate", more details are available online at http: / / hibernate.org / ), and for Android in particular (e.g. "GreenDAO", see Figure 1A , more details at http: / / greenrobot.org / greendao / or the "RoomPersistence Library", more details available online at https: / / developer.android.com / topic / libraries / architecture / room.html). There are also ORMs for other platforms, such as AppleCoreData (see Figure 1B ), which provides object graph management and persistence. More details are available online at https: / / developer.apple.com / documentation / coredata. Google also recognized the importance of this mapping by providing a room persistence API in the recently released AndroidO.

[0004] Typical roles of an ORM system are Figure 1A As shown in FIG. 1 , greenDAO represents an ORM system that mediates between a database and an object. In the figure, the database is represented by a Sqlite (or "SQLite") database, and a Java object is shown as an exemplary data object. The goal of ORM is to provide data transfer between database rows and Java objects in memory. Different mapping strategies (class per table, class per hierarchy, etc.) are used to establish relationships between database tables and Java objects.

[0005] An example of a "class per table" mapping strategy is Figure 2A, where the tables correspond to Structured-query language (SQL) tables. SQL is a high-level non-procedural query language for relational databases. On the left side of the figure, a database model is shown containing two tables "suppliers" (with two columns, "id" and "address") and "orders" (with three columns, "name", "price" and "supplierId"). On the right side, the mapping between the tables and the corresponding Java objects representing the data from the tables is shown through the database table definition and the Java class definition. In particular, it can be seen that the SQL tables correspond to the Java classes, and the columns correspond to the data fields (i.e., attributes / properties) of the classes. Figure 2A What is not shown is the correspondence between the rows of the table and the data objects (i.e., instances of the class). An example of an extended object-relational mapping is shown in Figure 2B shown.

[0006] Interoperation (especially data access) between Java applications and DBMS can be inter-process or intra-process, depending on whether the application (or applications) and DBMS have their own operating system processes, or whether they share the same process, i.e., a common process running the application and DBMS.

[0007] The difference between inter-process and intra-process data access (also called "interoperability") is as follows Figure 3 As shown in Figure 1. As can be seen, in the case of inter-process interoperation, there are two separate operating system processes: the application process running the virtual machine (VM) and the DBMS process running the DBMS row buffer. Inter-process interoperation is performed through serialization and deserialization. In the case of intra-process interoperation, the VM and DBMS row buffer are executed in the application process.

[0008] When the DBMS is implemented as a native component (e.g., using C / C++) and executed in the same process as the application Java Virtual Machine (JVM), the interaction mechanism between Java (i.e., part of the process in the virtual machine) and the native part of the process will use the Java Native Interface (JNI). In this case, JNI can be used to call the public API functions of the DBMS, which is loaded into the application process as a dynamically linked library. This approach can be used for the Android data access layer of the Sqlite package android.database.sqlite, and more information is available online at https: / / developer.android.com / reference / android / database / sqlite / package-summary.html.

[0009] Figure 4 The data transmission pipeline in a typical ORM is shown in the figure. The application process includes Java VM and Sqlite DBMS. Summary of the invention

[0010] A system and method are presented that significantly speeds up Java object creation and database object initialization.

[0011] This can be achieved by the features of the independent claim. Advantageous embodiments of the invention are subject matter of the dependent claims.

[0012] On the one hand, a device for extracting data from a database is provided. The device includes a database interface for enabling communication with the database; a virtual machine (VM); a buffer including multiple storage areas; a data extraction kernel on the native side; and an object initialization kernel on the VM side. The data extraction kernel includes a native side buffer application programming interface (API) for reading database data from multiple locations in the database using the database interface and writing the database data to the buffer using the native side buffer API, wherein the database data corresponds to a query result. The object initialization kernel includes a VM side buffer API for reading the database data from the buffer using the VM side buffer API and initializing multiple objects using the database data read from the buffer. Information indicating one of the multiple storage areas in the buffer in which the data extraction kernel writes the database data is known to the native side buffer API and the VM side buffer API.

[0013] This offers the advantage of enabling fast object initialization based on database data.

[0014] According to some embodiments, the data extraction kernel includes a native side multi-cursor to extract the database data from the database, the native side multi-cursor including a plurality of cursors. The object initialization kernel includes a VM side multi-cursor to extract the database data from the buffer, the VM side multi-cursor including a plurality of cursors. The cursor of the native side multi-cursor is used to iterate the items of the database data at one of a plurality of locations in the database independently of the positions of other cursors of the native side multi-cursor. The cursor of the VM side multi-cursor is used to iterate the items of the database data synchronously with the iteration of the cursor of the native side multi-cursor.

[0015] This provides the advantage that database regions can be traversed in a coordinated manner when fetching database data for object initialization.

[0016] For example, the cursor of the VM side multi-cursor is used to pass an iteration call to the cursor of the native side multi-cursor; and the cursor of the native side multi-cursor is used to execute cursor steps synchronously with the VM side database cursor when receiving the iteration call by reading items of the database data from the database and writing the items of the database data to the buffer.

[0017] This provides a fast and efficient data pipeline from the database to the VM.

[0018] In one example, the multiple storage areas are located on the native side; the VM side buffer API includes a direct buffer on the VM side; the cursor of the native multi-cursor is also used to pass the location of a storage area among the multiple storage areas to the direct buffer when receiving the iteration call; and the object initialization kernel is used to obtain the database data for initializing the object by accessing the direct buffer.

[0019] This provides the advantage of allowing the VM side to actually fetch data from the storage area of ​​the buffer.

[0020] In another example, the multiple storage areas are located on the native side; the cursor of the native side multi-cursor is also used to create a new direct buffer in the VM side buffer API when receiving the iteration call, and pass the direct address of one of the multiple storage areas to the direct buffer through the buffer API; and the object initialization kernel is used to obtain the database data for initializing the object by accessing the direct buffer.

[0021] This provides the advantage of being able to flexibly allocate direct buffers for actually fetching data from the buffer storage area.

[0022] In yet another example, the plurality of storage areas are located on the VM side, and the data extraction kernel is used to write the database data into the buffer through the native side buffer API.

[0023] This provides the advantage that the virtual machine accesses database data for object initialization in an efficient manner.

[0024] For example, the object initialization kernel is used to: for each cursor in the native-side multi-cursor, initialize an object in the multiple objects using database data in the cursor step.

[0025] This provides the advantage of coordinating efficient object initialization against database data using pipelined cursors.

[0026] In some embodiments, the apparatus further comprises a kernel generator for generating the object initialization kernel and the data extraction kernel. The kernel generator is used to: generate the object initialization kernel based on a first isomorphism between the respective object types of the plurality of objects and the data format of the regional layout in the buffer; and generate the data extraction kernel based on a second isomorphism between the data format of the database data in the database interface and the data format of the regional layout in the buffer. The first isomorphism and the second isomorphism are input to the kernel generator, and the isomorphism is a data structure representing program code.

[0027] This provides the advantage that the data extraction kernel and the object initialization kernel can be provided with a database and buffers and a corresponding mapping between buffers and data objects.

[0028] For example, the kernel generator also inputs a data model, object-relational mapping (ORM), a data access application programming interface (API), the definition of the native side buffer API and / or the definition of the VM side buffer API.

[0029] For example, the kernel generator dynamically generates the object initialization kernel and the data extraction kernel at runtime.

[0030] This offers the advantage of flexibility in the timing of object initialization during operation of the device.

[0031] For example, the object is a Java object, and the VM is a Java VM.

[0032] This offers the advantage of compatibility with mature technologies.

[0033] On the other hand, a device for writing data to a database is provided. The device includes a database interface for enabling communication with the database; a virtual machine (VM); a buffer including multiple storage areas; an object processing kernel on the VM side; and a data writing kernel on the native side. The object processing kernel includes a VM-side buffer application programming interface (API) for writing object data of multiple objects to the buffer using the VM-side buffer API. The data writing kernel includes a native-side buffer application programming interface (API) for reading the object data from the buffer using the native-side buffer API and writing the object data to multiple locations in the database using the database interface. Information indicating one of the multiple storage areas in the buffer in which the object processing kernel writes the object data is known to the native-side buffer API and the VM-side buffer API.

[0034] This provides the advantage of fast data transfer from VM objects to the database.

[0035] In another aspect, a method for extracting data from a database is provided. The method comprises the following steps: reading database data from multiple locations in the database through a database interface, the database data corresponding to query results; writing the database data to a buffer including multiple storage areas using a native side buffer application programming interface (API); reading database data from the buffer using a virtual machine (VM) side buffer API; and initializing multiple objects using the database data read from the buffer. Information indicating one of the multiple storage areas in the buffer in which the database data is written is known to the native side buffer API and the VM side buffer API.

[0036] For example, the step of extracting data from the database includes: independently of the positions of other cursors of the native side multi-cursor, at one of the multiple positions in the database, a cursor among the multiple cursors included in the native side multi-cursor iterates the items in the database data; the step of reading the database data from the buffer includes: a cursor among the multiple cursors included in the VM side multi-cursor iterates the items in the database data synchronously with the iteration of the cursor of the native side multi-cursor.

[0037] For example, iteration of the cursor of the VM-side multi-cursor includes: passing an iteration call to the cursor of the native-side multi-cursor; and iteration of the cursor of the native-side multi-cursor includes: upon receiving the iteration call, executing cursor steps synchronously with the VM-side database cursor by reading items of the database data from the database and writing the items of the database data to the buffer.

[0038] In one example, the iteration of the cursor of the native-side multi-cursor also includes: when receiving the iteration call, passing the position of the one storage area among the multiple storage areas to the direct buffer included in the VM-side buffer API on the VM side, the multiple storage areas are located on the native side, and the iteration of the cursor of the VM-side buffer API includes: obtaining the database data for initializing the object by accessing the direct buffer.

[0039] In another example, the iteration of the cursor of the native-side multi-cursor also includes: when receiving the iteration call, creating a new direct buffer in the VM-side buffer API, and passing the direct address of one of the multiple storage areas to the direct buffer through the buffer API, and the iteration of the cursor of the VM-side buffer API includes: obtaining the database data for initializing the object by accessing the direct buffer.

[0040] In yet another example, the multiple storage areas are located on the VM side, and the database data is written into the buffer through the native side buffer API.

[0041] In some embodiments, for each cursor in the native-side multi-cursor, an object of the plurality of objects is initialized using the database data in a cursor step.

[0042] In some embodiments, the method of extracting data from a database further includes: generating the object initialization kernel based on a first isomorphism between the respective object types of the multiple objects and the data format of the regional layout in the buffer, and generating the data extraction kernel based on a second isomorphism between the data format of the database data in the database interface and the data format of the regional layout in the buffer, wherein the first isomorphism and the second isomorphism are input to generate the data extraction kernel and the object initialization kernel, and the isomorphism is a data structure representing program code.

[0043] For example, a data model, an object-relational mapping (ORM), a data access application programming interface (API), the definition of the native side buffer API and / or the definition of the VM side buffer API are also input to generate the data extraction kernel and the object initialization kernel.

[0044] For example, the object initialization kernel and the data extraction kernel are dynamically generated at runtime.

[0045] In some embodiments, the plurality of objects are Java objects and the VM is a Java VM.

[0046] In another aspect, a method for writing data to a database is provided. The method comprises the following steps: writing object data of multiple objects to a buffer including multiple storage areas using a virtual machine (VM) side buffer application programming interface (API); reading the object data from the buffer using a native side buffer API; and writing the object data to multiple locations in the database through a database interface, wherein information indicating one of the multiple storage areas in the buffer where the object data is written is known to the native side buffer API and the VM side buffer API.

[0047] In addition, a computer-readable medium storing instructions is provided, wherein the instructions enable a computer to execute the above method of extracting data from a database or the above method of writing data into a database.

[0048] The following drawings and description will set forth one or more embodiments in detail. Other features, objects, and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Hereinafter, various embodiments of the present invention will be described in more detail with reference to the accompanying drawings, in which:

[0050] Figure 1A is a schematic diagram of an exemplary ORM system;

[0051] Figure 1B is a flow chart of an exemplary object graph management system;

[0052] Figure 2A is a diagram showing an example of mapping between database tables and data objects;

[0053] Figure 2B is a diagram showing another mapping example between database tables and data objects;

[0054] Figure 3 is a block diagram illustrating inter-process and intra-process data access;

[0055] Figure 4 is a block diagram illustrating a data transfer pipeline in object-relational mapping;

[0056] Figure 5 is a block diagram illustrating an exemplary data extraction path from a database to a user interface;

[0057] Figure 6 is a table diagram showing an exemplary data layout of a database management system;

[0058] Figure 7 is a block diagram of a device for extracting data from a database;

[0059] Figure 8 is a block diagram of an apparatus including a kernel generator for extracting data from a database;

[0060] Fig. 9 is a block diagram of a device for writing data into a database;

[0061] Fig.10 is a flow chart of a method for extracting data from a database;

[0062] Fig.11 is a flow chart of a method for writing data into a database;

[0063] Fig. 12A It is the definition of the method of the present invention supported in Sqlite;

[0064] Fig. 12B It is the definition of the method of the present invention supported in Sqlite;

[0065] Fig.13A is an example of code slicing for a data transfer pipeline;

[0066] Fig. 13B It is a block diagram of the library that implements the data transfer pipeline;

[0067] Fig.14 It is a schematic diagram of the isomorphic concrete data representation of DBMS rows and buffers;

[0068] Fig.15 It is a flowchart of data modeling and class mapping;

[0069] Fig.16 It is a flowchart of the method for implementing the data API;

[0070] Fig.17 It is a reuse flow chart of the data API generator for buffer API implementation and data access API implementation;

[0071] Fig.18 is a flow chart of a method for implementing a data conversion kernel;

[0072] Fig.19 is a flow chart for reusing the converter generator to generate the kernel;

[0073] Fig. 20 is a block diagram of a device for extracting data from a database;

[0074] Fig.21 is a block diagram of a device for extracting data from a database;

[0075] Fig. 22 is a diagram showing benchmark results for a typical data access scenario;

[0076] Fig.23 A table showing the type and number of SQL statements executed in a typical month, as well as the query characteristics used in each statement;

[0077] Fig.24 is a statistical table of database usage during the startup process; and

[0078] Fig.25 Code examples for cross-platform code. DETAILED DESCRIPTION

[0079] The present invention relates to the execution environment of the same application process of a VM (such as JVM) and a DBMS sharing an operating system, such as Figure 3 (in-process interoperation).

[0080] In order to be materialized as Java objects, database data may need to be transmitted through multiple layers of the software stack, such as Figure 5 For example, as can be seen in the figure, the following steps may be required:

[0081] 1) Read the data page from the database file and put it into memory (in a page pool, such as the last recently used (LRU) page pool).

[0082] 2) Row data should be located in pages according to the DBMS's index and / or other navigation data structures. An example of data layout in SQLite is as follows: Figure 6 Further, more details of the sample data layout in Sqlite can be found in G. Allen and M. Owens, "The Definitive Guide to SQLite (second edition)", Apress, 2010, pp. 303-308.

[0083] 3) The row data should be decoded from the DBMS internal format to the DBMS public application programming interface (API) format. The decoded information is Figure 5 denoted by the "decoded row" block.

[0084] 4) The decoded data should be converted into Java type format and transferred to Java storage space.

[0085] 5) A Java object should be created and its fields initialized with the transferred data.

[0086] 6) User interface (UI) components are bound to Java objects by copying data from Java object fields or using the data binding library (for more information about the data binding library, visit: https: / / developer.android.com / topic / libraries / data-binding / index.html).

[0087] The performance of this data transfer process is mainly limited by memory, so the storage hierarchy (especially cache, L1, L2, L3, ...) should be considered during implementation.

[0088] Furthermore, on mobile hardware (e.g., phones or tablets), hardware performance is limited by memory access. Therefore, it is desirable to have both temporal and spatial cache locality. Data transfers should be pipelined so that buffers created at each layer should be immediately accessible downstream.

[0089] The above steps 1) to 3) are performed by the DBMS, for example, when the database cursor moves to the next row, the values ​​of all columns are retrieved. This is demonstrated by the list shown in the following code snippet for extracting data from a Sqlite row using the public API:

[0090]

[0091] In the code snippet, operations starting with sqlite_ belong to the Sqlite public API. The sample code also shows that the extracted (decoded) data is appended to the output buffer data structure associated with the cursor. This code snippet for extracting data from a DBMS such as Sqlite will also be referred to as an extraction kernel hereinafter.

[0092] Corresponding to this extraction kernel, there should be another code segment, as shown below, which reads the decoded data from the buffer associated with the cursor and transfers it to the fields of a Java object:

[0093]

[0094] In other words, the latter code segment performs the materialization or initialization of Java objects. This code segment for object initialization is also called InitKernel.

[0095] The present invention provides a new system and method for transferring data from a database and materializing it into data objects, such as Java objects. Among them, a device for implementing ORM to establish cache-efficient data transfer between DBMS and Java is provided. Further, a method for implementing data transfer code (codekernels) that is easy to develop / support and efficient at runtime is described.

[0096] For example, in an ORM such as greenDAO, the initialization kernel code snippet shown above is generated based on the table-to-class mapping configuration. Figure 4 When InitKernel is executed in the data transfer pipeline of the typical ORM shown, the data of each class field is requested from the cursor. Android Database Cursor (ADC) is a Java class implemented in the package android.database.sqlite of the Android SDK. It is paired with the corresponding ADC native side object implemented in C++ and resides in a native library linked to the public API of Sqlite.

[0097] Two cursor operations are relevant to the present invention:

[0098] 1) The moveNext operation moves the cursor forward one step to the next row in the query result set; and

[0099] 2) getXXX operation set, extracting field values ​​from the current row of the result set.

[0100] In the actual implementation of android.database.sqlite, the rows of the database are fetched in chunks, and each chunk is put into a storage buffer. The native side ADC is moving on the current chunk, and when that chunk is complete, it retrieves the next chunk from the database and then moves to the first row of the new chunk.

[0101] When retrieving the next block of data, the following steps are performed:

[0102] For each database row loaded into the data block, and

[0103] For each column of a row:

[0104] 1) Native ADC uses relevant Sqlite statements and calls Sqlite API methods to extract column data from the database;

[0105] 2) The Sqlite API function uses the internal Sqlite API to access and decode column data and store it in the format of the public Sqlite API;

[0106] 3) The native ADC receives the decoded column data and saves it in a data block buffer managed by the native Android database library, which converts the data into the format of the data block buffer.

[0107] During the InitKernel execution, the following steps are performed for each column of the table mapped to the Java class:

[0108] 1) Call the method of Android Database Cursor (ADC) on the Java side, and then use JNI to call the corresponding method of Android Database Cursor on the Sqlite side (native side).

[0109] 2) The native ADC extracts the column value from the data block buffer, or advances the data fetch to the next data block and then extracts the column value.

[0110] The following design features of typical implementations lead to suboptimal performance of the data transfer pipeline:

[0111] Too many JNI calls are performed for each column access. Although the rows are blocked on the native ADC side, JNI calls are still performed to access the blocked data and transfer it to the Java memory space.

[0112] The data value is decoded from the internal Sqlite representation to the public Sqlite representation and then immediately converted to the internal representation of the data chunk buffer for subsequent transfer to Java (during the JNI call).

[0113] Data is transferred column by column to the (Java) VM storage space, reducing the locality of the temporal cache.

[0114] Linearization of row data in a single byte array (for more details, see A. Shivarudraiah and E. Shirk, "System and method for marshaling massive database data from native layer to java using linear array." Patent US 9600546B2, March 21, 2017); copying this array to the Java / VM side has at least two negative effects on performance: 1) the extra data copying increases cache misses; 2) linear buffers are suboptimal for column-based data storage.

[0115] Based on the performance issues of these typical implementations, the present invention aims to provide a system for implementing ORM so as to establish memory efficient data transfer and CPU efficient control transfer between DBMS and VM / Java. In particular, the VM side database cursor should be pipelined with the native side database cursor, which makes the data transfer cache efficient and can reduce cache misses. The native side database cursor can also be advantageously used to create DirectByteBuffer objects on the Java side. DirectByteBuffer objects should be directly accessible from Java code to facilitate inline and just in time (JIT) compiler optimization.

[0116] Yet another objective is to describe a method for implementing data transfer code kernels that is: 1) simple to use (development / support); 2) flexible to support different DBMS and buffer data layouts; and 3) efficient at runtime.

[0117] The following will be combined Figure 7 General aspects of the invention are described.

[0118] According to a general aspect, the present invention provides an apparatus for extracting data from a database. The apparatus includes a database interface for enabling communication with the database, a virtual machine (VM) 710, and a buffer 770 including a plurality of storage areas. In addition, the apparatus for extracting data from a database table includes a data extraction kernel 760 on a native side 750 and an object initialization kernel 720 on a VM side 710. The data extraction kernel 760 includes a native side buffer application programming interface (API) for reading database data from a plurality of locations in the database using the database interface and writing the database data to the buffer 770 using the native side buffer API. The database data corresponds to a query result. The object initialization kernel 720 includes a VM side buffer API for reading the database data from the buffer 770 using the VM side buffer API and initializing a plurality of objects using the database data read from the buffer. Information indicating one of the plurality of storage areas in the buffer 770 in which the data extraction kernel 760 writes the database data is known to the native side buffer API and the VM side buffer API.

[0119] The database data extracted from the database corresponds to an instance of a query result, which may include a set of instances. The instance of the query result may consist of different items located in different database tables. Then, the instance of the query result corresponds to the VM object instance.

[0120] For example, database tables may be organized in a master-detail relationship, such as Order(master)-lineitem(detail), where a master item may be associated with multiple detail items. For example, an order table includes a column "id" containing an order ID number. In another table "lineitem", several items may have the same order id as indicated by an entry in the "order_id" column of the lineitem table corresponding to an entry in the id column of the order table.

[0121] This aspect of the invention includes the following way of transferring information of each master item through the data pipeline (database->data extraction kernel->buffer->object initialization kernel->object):

[0122] 1) Each master item is transferred in a single query, and then a separate query is made based on each master item to extract the related detail items one by one.

[0123] 2) Transfer each master item in a single query, then, using a VM-native round trip, extract all related detail items (i.e., detail items that are related to all master items) and save them in a buffer that can hold many detail items simultaneously.

[0124] 3) A single master item and all related detail items are round-tripped by a single VM-native through the corresponding structured buffers.

[0125] The buffer contains multiple storage areas (or blocks). That is, the buffer does not need to correspond to a linear array. In particular, some or all of the storage areas or blocks of the buffer may be adjacent to each other (e.g., physically, logically and / or with respect to a memory allocation strategy), however, the buffer may also or may contain one or more areas that are away from other storage areas included in the buffer. That is, storage areas are not necessarily allocated continuously. They may be distributed in memory, which may be shared with other data. It is sufficient for the buffer to be written to by the data extraction kernel and read by the object initialization kernel. No further access to the buffer is required.

[0126] Rather than using a linear array of buffers, the VM side buffer API and the native side buffer API are used, where the buffer APIs on both sides know information indicating one of the multiple storage areas where the database data is written. The data extraction kernel and the object initialization kernel can agree (in their definitions) on the structure of the buffer, such as: a) how many rows are stored at any given time; b) coordinated control flow between the VM and native parts; c) the life cycle and allocation location of each area of ​​the buffer. As described in more detail later, the common knowledge of the data extraction kernel and the object initialization kernel, that is, their consensus on the buffer structure, can be established through kernel generation.

[0127] The VM side and native side buffer APIs constitute the implementation of the general buffer API. Accordingly, the general buffer API can encapsulate the control flow transmission mechanism between the VM side and native side implementations of the buffer API. Depending on the implementation of the general buffer API, the buffer API can use different memory layouts (data formats) composed of regions and memory allocation strategies. For each selected specific memory layout, at least the VM side implementation (VM side buffer API) and the native side implementation of the general buffer API (native side buffer API) are required. The present invention is not limited to the specific area structure of the buffer, but provides the possibility of having an arbitrary buffer structure according to the general buffer API and its instantiation on the VM side and the native side. The storage area included in the buffer corresponds to the memory block in the specific buffer implementation.

[0128] The data extraction kernel (extraction kernel) and the object initialization kernel (initialization kernel) include buffer APIs (native side / VM side) respectively. That is, the data extraction kernel implements the native side buffer API, and the object initialization kernel implements the VM side buffer API. The data extraction kernel can extract data from several specified locations in the database corresponding to different database items (such as tables, views, queries, etc.). When data is moved from the native side to the VM, the native side buffer is used to implement writing to the buffer from the native side. When the object is initialized, the VM side buffer is used to implement reading the buffer from the VM side.

[0129] Although in the above description, reference is made to Figure 7 , but the present invention is not limited to the embodiments shown in the figure. Figure 7 The illustrated embodiment and further embodiments.

[0130] According to some embodiments, the data extraction kernel includes a native side multi-cursor to extract data from a database. The multi-cursor includes a plurality of cursors. Accordingly, the object initialization kernel includes a VM side multi-cursor, and the VM side multi-cursor includes a plurality of cursors to extract database data from a buffer. The cursor (i.e., at least one cursor) of the native side multi-cursor is used to iterate the items of the database data at one of the plurality of positions of the database independently of the positions of the other cursors of the native side multi-cursor. The cursor (i.e., at least one cursor) of the VM side multi-cursor is used to iterate the items of the database data synchronously with the independently iterated native side cursor.

[0131] Typically, databases typically have the concept of cursors to traverse internal data structures, or other objects to represent data locations, depending on the specific data access API. According to an embodiment using multiple cursors instead of a single cursor, a set of cursors containing multiple such database cursors (which can be viewed as "cursors in buckets") can be maintained that traverse the database in a coordinated manner. It has been observed that using two coordinated cursors can improve performance by up to 50%.

[0132] The term "multi-cursor" refers to a collection of at least two cursors. In other words, a multi-cursor is a collection of ordinary cursors. Each ordinary cursor can iterate over a collection of items in a database (e.g., tables, views, queries) and extract data from the corresponding specified location in the database. In addition, each cursor in a multi-cursor can operate independently. For example, a multi-cursor can be used to implement all three extraction strategies for the master-detail relationship described above. In the order-lineitem example, the multi-cursor includes two cursors (i.e., a cursor for the order and a cursor for the lineitem).

[0133] Having multiple cursors to access the database allows implementing multiple data extraction strategies. In particular, multiple VM objects can be initialized in a coordinated manner. For each object to be initialized, an object-cursor mapping can be defined. In addition, pipelined cursor pairs are defined (a pair consisting of a VM-side cursor of a VM-side multi-cursor and a native-side cursor of a native-side multi-cursor). Accordingly, each cursor pair has its own object-cursor mapping. The multi-cursors constitute an implementation of a consensus between the object initialization kernel and the data extraction kernel on the buffer structure, i.e., a common sense of where in the buffer the database data is written and read.

[0134] For example, the VM-side cursor included in the VM-side multi-cursor can be used to pass an iteration call to a corresponding cursor of the native-side multi-cursor, and the VM-side multi-cursor is synchronized with the iteration of the cursor from the native-side multi-cursor. The cursor of the native-side multi-cursor is used to perform cursor steps synchronously with the VM-side database cursor by reading the items of the database data from the database and writing the items of the database data into the buffer when receiving the iteration call.

[0135] The present invention provides several possibilities for the location of the storage area of ​​the buffer. Two examples are provided below where the storage area is located on the native side.

[0136] Example a) Multiple storage areas may be located on the native side. In this case, the VM side buffer API may include a direct buffer on the VM side. The cursor of the native multi-cursor is also used to pass the location of one storage area in the multiple storage areas to the direct buffer when receiving the iteration call. The object initialization kernel may be used to obtain the database data for initializing the object by accessing the direct buffer.

[0137] In this example, the object initialization kernel retrieves database data by accessing a direct buffer. Figure 7 The direct buffer 730 shown contains memory addresses indicating where data extracted from the database has been written to multiple storage areas, the data having been received from the native side multi-cursor or from one of the cursors of the native side multi-cursor. In this example, information indicating the storage area where the data extraction kernel writes the database data is contained on the VM side in the direct buffer.

[0138] Example b) Although the multiple storage areas are still located on the native side, the cursor of the multiple cursors on the native side is used to create a new direct buffer in the VM side buffer API when receiving the iteration call, and pass the direct address of one of the multiple storage areas to the direct buffer through the buffer API. The object initialization kernel is used to obtain the database data for initializing the object by accessing the direct buffer.

[0139] According to examples a) and b), the VM side buffer API knows which direct buffer or which field of the direct buffer needs to be accessed in order to obtain the database data according to the memory address contained in the field. In addition, in both examples, there can be one direct buffer item or multiple direct buffer items at one point in time or during the operation time of the device.

[0140] According to example b), instead of passing said memory location to a previously created direct buffer, for example at the beginning of a full database iteration or during compilation, a new direct buffer is created by said iteration step containing said memory address accessed in the current iteration step.

[0141] However, the present invention is not limited to the multiple memory areas of the buffer located on the native side. In another example, the multiple memory areas of the buffer are located on the VM side, and the data extraction kernel is used to write the database data to the buffer through the native side buffer API.

[0142] In the above examples a) and b), the VM side buffer API knows information indicating the storage area in the buffer where the database data is written by accessing the direct buffer where the storage area is stored. However, in examples where multiple storage areas are located on the VM side, the native side (especially the native side database cursor) passes the database data itself to the VM side through the native side buffer API. Accordingly, the native side knows the database data and does not need to use a memory address pointing to the native side location to access the data. The VM side does not need a direct buffer to store the memory address.

[0143] As mentioned above, the present invention implements object-cursor mapping, wherein the object is initialized by a cursor pair consisting of a cursor of a VM-side multi-cursor and an associated cursor of a native-side multi-cursor. Accordingly, for example, the object initialization kernel is used to initialize an object in the plurality of objects using database data in a cursor step for each cursor in the native-side multi-cursor.

[0144] In addition, as mentioned above, the information indicating one of the multiple storage areas in the buffer where the data extraction kernel writes the database data is known to the native side buffer API and the VM side buffer API. Accordingly, the object initialization kernel and the data extraction kernel (in their definitions) agree on the buffer structure. To this end, the generation of the two kernels can be connected by isomorphism. For example, each extraction strategy (for example, the above-mentioned extraction strategies 1) to 3) for the master-detail relationship) can be described by the definition of a corresponding isomorphic pair, that is, an isomorphism between the data object and the buffer and an isomorphism composed of a corresponding isomorphism between the buffer and the database. Generally, isomorphism is a structure-preserving mapping between structures with an inverse mapping. Isomorphism represents a corresponding relationship between structures, especially a one-to-one correspondence. In the context of the present invention, isomorphism is a data structure representing program code.

[0145] Therefore, according to Figure 8 In the example shown, the apparatus further includes a kernel generator 880 for generating an object initialization kernel and a data extraction kernel. The kernel generator 880 is used to generate the object initialization kernel 720 based on a first isomorphism between the object types of the plurality of objects and the data format of the regional layout in the buffer, and to generate the data extraction kernel 720 based on a second isomorphism between the data format of the database data in the database interface and the data format of the regional layout in the buffer. The first isomorphism and the second isomorphism are input to the kernel generator 880, and the isomorphism is a data structure representing a program code. The program code can be provided, for example, as an abstract syntax tree (AST), an intermediate representation (IR), or a text file.

[0146] For example, Figure 8 As shown, in addition to the isomorphism, the kernel generator also inputs a data model, an object-relational mapping (ORM), a data access application programming interface (API), the definition of the native side buffer API and / or the definition of the VM side buffer API. Among them, the data model is a data structure representing a class with methods and properties. ORM is a data structure representing the mapping of the data model to the items (tables, views, etc.) of the database. The data access API is a data structure representing the interface, methods and properties of the data access library. In other words, the data access API is an instance of the API descriptor data type. The definition of the buffer API is other instances of the API descriptor data type.

[0147] The object initialization kernel 720 of the VM side 710 includes a VM side buffer API, and the data extraction kernel 760 of the native side 750 provides a native side buffer API. The VM side buffer API and the native side buffer API are provided by the object initialization kernel and the data extraction kernel, respectively. The buffer structure can be an arbitrary buffer structure, including an arbitrary structure / layout of a storage area, and is determined by a first isomorphism between the buffer layout and the data object (i.e., between the database data extracted by the cursor of the native side multi-cursor and the buffer structure / layout) and a second isomorphism between the format of the database data in the database interface and the buffer layout (i.e., between the buffer structure / layout and the multiple objects initialized from the database data). Since two kernels are generated based on the data model, data access API, buffer API definition and ORM, isomorphism is inherent to the two linked kernels, i.e., the kernels are linked by providing respective implementations of the same general buffer API. If these two isomorphisms are provided, the structure of the buffer can be arbitrary.

[0148] The kernel generator can dynamically generate the object initialization kernel and the data extraction kernel at runtime. However, it is also possible to generate the individual kernels during, for example, compilation.

[0149] For example, the object or objects being generated may be Java objects such as Figure 7 and 8 As shown. Accordingly, the VM can be a JVM. However, although the present invention provides several examples using Java, the present invention is not limited to this technology. Other possible examples are JavaScript or Microsoft.NET framework. The present invention is applicable to systems including a JIT compiler.

[0150] In addition, some examples in the description use the Android execution environment and the Sqlite database management system (DBMS). However, it can also be applied in other contexts, such as in the context of Apple iOS technology.

[0151] So far, the case of extracting database data and initializing data objects from the database has been described. However, the present invention is also applicable to the case of modifying data in the database or writing new data to the database based on operations such as field modification or instantiation of data objects in the VM. When data is moved from the VM to the native side, the roles of the buffer API implementations on the VM side and the native side and their respective kernels are exchanged.

[0152] In particular, the present invention also provides a device for writing data into a database, such as Fig. 9As shown. The device for writing data to a database includes a database interface for enabling communication with the database; a VM 710; and a buffer 770 including multiple storage areas. The device for writing data to a database also includes an object processing kernel 920 on the VM side, which includes a VM-side buffer application programming interface (application programming interface, API for short); and is used to write object data of multiple objects to the buffer using the VM-side buffer API. In addition, the device for writing data to a database includes a data writing kernel 960 on the native side, which includes a native-side buffer API; and is used to read object data from the buffer using the native-side buffer API, and write object data to multiple locations in the database using the database interface. Information indicating one of the multiple storage areas in the buffer in which the object processing kernel writes the database data is known to the native-side buffer API and the VM-side buffer API.

[0153] For example, object data corresponds to attributes of a data object. Relative to the above-mentioned device for reading data from a database, in the device for writing data to a database, reading data from a buffer is replaced by writing data to a buffer. In addition, writing data to a native side buffer is replaced by reading data from a buffer, and extracting data from a database is replaced by writing data to a database. In addition, the device for writing data to a database can be implemented similarly to the device for extracting data from a database. In particular, the device can have read and write capabilities in both directions on the VM side and the native side (i.e., objects on the VM side <-> buffer and buffer on the native side <-> database), thereby including a device for extracting data from a database, such as Figure 7 As shown and described previously, and means for writing data into a database, such as Fig. 9 shown.

[0154] The device may include a database that communicates with the device for extracting data from the database and / or the device for writing data to the database via the database interface. Figure 6 The file system of the database file shown may be stored locally on a device operating as a device. However, the present invention is not limited to a database located on the same device as the device for extracting and / or writing data. In particular, the database may be located on a remote server to which the device is connected via a wireless or wired connection.

[0155] The device for reading data from the database or writing data into the database may be included in a mobile device, such as a tablet computer, a laptop or a mobile phone / smartphone. However, the method may also be implemented in a personal computer.

[0156] Corresponding to the device for extracting data, the present invention also provides a method for extracting data from a database. The flowchart of the method is as follows: Fig.10 As shown. The method includes the steps of: reading database data from multiple locations in the database through a database interface S1010, wherein the database data corresponds to the query result. The method also includes the steps of: writing the database data into a buffer including multiple storage areas using a native side buffer application programming interface (application programming interface, referred to as API) S1020. The method also includes the steps of: reading database data from a buffer S1030 using a virtual machine (virtual machine, referred to as VM) side buffer API, and initializing S1040 multiple objects using the database data read from the buffer. In the steps of the method, information indicating one of the multiple storage areas in the buffer in which the database data is written is known to the native side buffer API and the VM side buffer API.

[0157] In addition, the present invention provides a method for writing data into a database. The flowchart of the method is as follows Fig.11 As shown. The method for writing data to a database comprises the steps of: writing object data of a plurality of objects into a buffer comprising a plurality of storage areas using a VM-side buffer API S1110. The method further comprises the step of: reading object data from a buffer using a native-side buffer API S1120. In addition, the method comprises the step of: writing the object data into a plurality of locations in the database S1130 using the database interface. In the steps of the method, information indicating one of the plurality of storage areas in the buffer in which the object data is written is known to the native-side buffer API and the VM-side buffer API.

[0158] The method of reading data from a database and writing data into a database is performed by a device for reading data from a database and writing data into a database. In addition, the present invention provides a computer-readable medium storing instructions, which enables a computer to execute the method of reading data from a database and / or the method of writing data into a database.

[0159] The specific details of the apparatus and method provided by the present invention will be further described below. A method for initializing / materializing objects from database data and a system for instantiating the method will be described to improve the performance of a data transmission pipeline.

[0160] In particular, a method for implementing fast data object (e.g., Java object) initialization / materialization is proposed, comprising the following steps:

[0161] 1. Describe data models and class mapping using known methods;

[0162] 2. Implementing DBMS-specific data access APIs and buffer APIs in a code generation framework that supports definition of abstract data types and related functions in an object-oriented programming (OOP) language;

[0163] 3. Use the transformer generator to take the data model, class map, data access API, and buffer API as inputs of its source code and generate the extraction kernel (data extraction kernel) and initialization kernel (object initialization kernel) as outputs;

[0164] 4. Compile the generated code and package the initialization kernel and the extraction kernel into the dynamic link library (corresponding Java / VM library and native);

[0165] 5. Link the libraries statically at compile time or dynamically at runtime so that they can be used by components of the system at runtime.

[0166] Source code is a textual representation of a source program using a programming language. A source program is a computer program used as input to a compiler. The program is converted into executable machine code.

[0167] A system implementing the method may include the following components:

[0168] 1. VM-side database cursors, which can be included in VM-side multi-cursors that are pipelined with native-side database cursors, which move together over the rows of the query result set.

[0169] 2. Native database cursors can be included in native multi-cursors and:

[0170] a. Access the internal functions of the DBMS and read database data

[0171] b. Execute the extraction kernel to fill the data buffer storage with information from the database

[0172] c. Create at least one direct buffer (i.e., DirectByteBuffer in Java) on the VM side using JNI (Java Native Interface as an example of a VM-Native Interface) and pass the direct address of the data buffer to it.

[0173] 3. Initialize the kernel to execute inside the VM and execute:

[0174] a. Call the moveNext operation of the VM-side database cursor

[0175] b. Receive the direct buffer pointing to the next buffer data, that is, the storage area pointing to the next buffer data.

[0176] c. Perform operations on the direct buffer to read database data (e.g., row data) and initialize data object fields / attributes.

[0177] 4. A kernel generator that:

[0178] a. Support the definition of abstract data types and related functions;

[0179] b. Configured with data model, ORM class mapping, data access API, buffer API, i.e., general buffer API and / or its native and VM side implementation;

[0180] c. When called, a pair of synthesizable kernels are generated: 1) an extraction kernel that fills the data buffer using the native side buffer API; 2) an initialization kernel that reads data from the data buffer through the VM side buffer API.

[0181] Figure 8 An exemplary system (device) for implementing a method for rapid object materialization from database data is shown. The components of the system are described in detail below:

[0182] VM-side database cursor: When a Java application needs to access data, it issues a SQL query and returns a cursor (VM-side multi-cursor) of the result as a collection of database data items (e.g. rows). The cursor is then iterated to retrieve rows or database entries / values ​​one by one. The Java-side cursor is pipelined with the native-side database cursor, moving synchronously on the rows of the result set together. As we all know, cursors are often used internally in ORM implementations. At each step, the cursor's data is transferred to the fields / attributes of the Java object.

[0183] Native-side database cursors: The lifecycle of a native-side (or simply "native") database cursor can correspond to the lifecycle of a VM-side database cursor, but can also be different through pooling and / or other optimization mechanisms. The minimum requirement is to bind a native-side cursor to a corresponding VM-side cursor during iteration on the VM-side cursor. In the present invention, such binding is assumed.

[0184] The minimal patch for Sqlite to support this method is Fig. 12A and Fig. 12B As shown. Fig. 12AThe definition of is added to sqliteInt.h. Fig. 12B Only necessary internal functions are marked in . A total of 6 files were changed, 18 were added and 16 were deleted. Therefore, the marked functions became exportable and can be linked from other libraries (for example, directsqlite.so).

[0185] Native side cursors access internal DBMS functions to generate row data at each step. Fig.13A The interaction between the code segments executed in each step of the data transfer pipeline is illustrated. However, it should be noted that the present invention is not limited to SQLite. Other DBMSs such as Cassandra or Apache Spark can also be used in cases where data needs to be transferred between native and VM environments. In this example, the code segments belong to different software libraries, and the relationship between these software libraries is as follows: Fig. 13B shown.

[0186] The extraction kernel (data extraction kernel) is dynamically generated by the kernel generator component at application compile time or runtime, depending on the embodiment and using a method described in more detail below.

[0187] The kernel is executed once for each step of the cursor, which may also vary in different embodiments and is not a critical design choice.

[0188] The kernel is given two pointers, such as Fig.14 As shown. Pointer to DBMS RowApi (corresponding to database interface) and pointer to native side buffer API. During the execution of the kernel, data is copied from DBMS to the buffer.

[0189] Direct Buffer: A direct byte buffer as an example of a direct buffer is a special Java object used to efficiently access off-heap storage areas from Java code. DirectByteBuffer is a special Java object used to provide a "shared" area of ​​memory between the VM and the native side, avoiding data copying. During the execution of native code using the JNI interface, a direct byte buffer can be created by calling the following code.

[0190] env->NewDirectByteBuffer(pBuffer,bufferSize)

[0191] Where env is the JNIEnv object, pBuffer is a pointer to the buffer storage, and bufferSize is the size of the buffer storage in bytes.

[0192] Depending on the specific buffer layout (such as Fig.14As shown), one or more direct byte buffers can be created. When there are multiple direct byte buffers, each DBB represents a storage area within the buffer. They should form a list. The head of this list is passed to the VM side, and this buffer list is used in the initialization kernel to extract data from the buffer. During the execution of the extraction kernel, the content of the buffer memory will be filled. During the execution of the data transfer pipeline and between the cursor steps, different allocation / reallocation strategies known in the art can be used to manage the direct buffer list.

[0193] According to this embodiment, the initialization kernel (object initialization kernel) is dynamically generated by the kernel generator component when the application is compiled or run.

[0194] For each step of the cursor and an object, the kernel is executed once on the VM side; however, this can also change in different embodiments. For example, in a complex ORM mapping model, a data object can be mapped to multiple tables. In this case, the initialization kernel will use multiple cursors and multiple buffers (or buffer storage areas) when initializing a single Java object.

[0195] The initialization kernel has two pointers: a pointer to the VM-side buffer API object and a pointer to the ORM object / data object that needs to be initialized. When executing the initialization kernel, the data is copied from the buffer to the fields of the object.

[0196] Direct buffers provide access to native storage areas. Lists of direct buffers are created to contain specific data from a database in some specific layout, which is used in the buffer API implementation (see Fig.14 See the layout examples and the following method description (step 2) for details on the buffer API implementation).

[0197] Each direct buffer object is used in the initialization kernel code and accessed by calling the corresponding method of the VM-side buffer API. In order to provide good performance, the code for initializing the kernel should contain direct calls to the methods of the direct buffer, which are inserted by the JIT compiler and optimized by the compiler.

[0198] The initialization kernel and the extraction kernel are generated by the kernel generator 880. The steps of kernel generation will be described in more detail below, especially in steps 3 and 4 of the following method description.

[0199] An important point of the present invention is that Figures 7 to 9In the system shown, the VM side cursor is pipelined with the native side cursor to improve cache efficiency. The buffer (RowBuffer) is filled by the generated extraction kernel according to the database specific storage design. In addition, a direct buffer (DirectByteBuffer) is created in the VM to avoid data transmission from the native side to the VM side, which may help improve storage efficiency. The direct buffer is unpacked by the generated materialization kernel (or object initialization kernel) on the Java side. A single common metadata-driven Codegen (kernel generator) is used to generate extraction kernels and initialization / materialization kernels that work together.

[0200] Provides a method that can be used to implement an efficient data transfer pipeline from a database to a Java object. The method is instantiated by the following steps:

[0201] Step 1: Describe the data model and class mapping

[0202] This step is performed by those skilled in the art according to Fig.15 The flowchart shown is implemented using known tools. The process is as follows:

[0203] 1) ORM users perform data modeling and configuration known to those skilled in the art.

[0204] 2) The generated artifacts are stored in a data model and class mapping repository (e.g., the repository can be as simple as an XML file or as complex as some cloud storage service).

[0205] Step 2: Implement the Data Access API and Buffer API

[0206] Both the data access API (an implementation of the database interface) and the buffer API use Fig.16 The same method is implemented as outlined in . This method is instantiated as follows for the data access API and the buffer API respectively:

[0207] For data access API: the physical memory data layout is defined as the memory layout of the DBMS, and the logical data model is defined as the model of the database. The instantiation of this method generates a data access library for the DBMS.

[0208] For the buffer API: The physical memory data layout is defined as the layout of the buffer data in memory, and the logical data model is the corresponding fragment of the database model (sub-model). For example, if the buffer contains only a single row of a DBMS table, then this sub-model will be the model of this table. The instantiation of this method produces the implementation of the buffer data structure, including the traversal and construction operations in the API. In this step 2, the buffer API corresponds to the implementation of the above-mentioned general buffer API, especially the implementation of the native side buffer API.

[0209] The process proceeds as follows:

[0210] 1) The logical data model is defined using any data definition language known in the art (this step is performed by the ORM developer).

[0211] 2) Using any API definition framework known in the art (for API definition frameworks, see, for example, the "Open API Initiative" available online at https: / / www.openapis.org / ), define a data access API that connects the data access API to the logical data model by defining a model-API mapping artifact. (This step is performed by the ORM developer.) An example of a model-API mapping is shown in Table 1.

[0212] Table 1: Model-API mapping example

[0213]

[0214] 3) The logical data model is implemented using any physical data layout known in the art: in particular, row-based layouts, column-based layouts, and hybrid layouts can be used, see Fig.16 The specific model implementation in . Known implementation techniques can be used. For example, each logical model can be associated one-to-one with the class and method of the OOP language interface. In this case, the specific model implementation can include a manual implementation of the corresponding interface using a given physical memory data layout.

[0215] 4) The concrete model implementation, data API definition and model-API mapping are used to generate the data API implementation as an instance of IR in object-oriented languages ​​such as Java / Scala. This can be an automatic step or a manual step.

[0216] 5) The generated data API implementation is optionally prepared for use in a framework with staged evaluation, i.e. virtualized as described in "AUTOMATIC CODE VIRTUALIZATION FOR A COMPILER", patent application WO2017095257.

[0217] When the process is executed according to items 1) to 5), the result of the application will be a fully functional API implementation that conforms to the logical model and keeps the specific data in the memory layout.

[0218] The idea behind this approach is to use the same process, i.e. reuse the Data API generator, when generating the data access layer for the database and doing additional buffering of the extracted information. Fig.17As shown in the figure. For the buffer API implementation, the physical buffer layout and logical buffer model are adopted; correspondingly, for the data access API implementation, the physical database data layout and logical database model are adopted. This design promotes technology reuse within the DBMS and provides many implementation choices to produce different data-dependent buffer models and implementations to build efficient data transfer pipelines as part of ORM and similar data access tools.

[0219] Step 3: Generate kernel source code

[0220] use Fig.18 The same method outlined in impl implements the initialization kernel and the extraction kernel. This method is instantiated as follows:

[0221] For the generation of the initialization kernel: define A API library as the buffer Java library, B API library as the ORM Java library, and isomorphic type repository as the isomorphic set between Java types and buffer items.

[0222] For the generation of the extraction kernel: define A API library as the data access library of the DBMS, B API library as the buffer C++ library, and the isomorphic type repository as the isomorphic set between the database abstraction layer (DAL) type and the buffer item. Examples of isomorphic data types are shown in Table 2, where a Scala-like language is used, corresponding to the isomorphic data types in the following literature: [1] A. Slesarenko, A. Filippov, and A. Romanov, "First-class isomorphic specialization by stagedevaluation", 10th ACM SIGPLAN workshop on Generic programming, Gothenburg, 2014, incorporated herein by reference.

[0223] Table 2: Homogeneous API data types

[0224] Buffer Type Row Type BufferInt RowInt BufferDouble RowDouble BufferString RowText ByteArray RowBlob BufferItem RowCell Collection[BufferItem] Collection[RowCell] Class Buffer(Val items:Collection[BufferItem] class Row(Val cells:Collection[RowCell])

[0225] Fig.18 An important observation and idea of ​​the approach shown is to realize that both kernels are converters between types, so the same process can be used to generate the initialization kernel that executes on the Java side of the ORM and the extraction kernel that executes on the native side of the DBMS. Fig.19As shown, the method is implemented by a converter generator artifact. Among them, the converter generator is reused to generate the initialization kernel and the extraction kernel respectively. In addition, it can be seen that a pair of isomorphic input kernel generators: the isomorphism between the object type and the buffer item (i.e., the data format of the area layout in the buffer) and the isomorphism between the DAL type (correspondingly, the data format of the database data in the database interface) and the buffer item. Accordingly, the kernel generation can be performed in such a way that the buffer API on the native side and the VM (Java) side know the information of the storage area of ​​the buffer where the data is located.

[0226] Fig.18 The process of implementing the data conversion kernel shown is performed as follows:

[0227] 1) Have two libraries: library A and library B, choose two types accordingly: type A and type B, and then implement a converter between type A and type B, as shown in the example code implementation of a converter between line and buffer API types shown below. In many cases, in a framework with first-class isomorphism (see also [1] and Table 2), a converter between type A and type B can be automatically generated. For example, the getConverter function used in the example code can be implemented in the Scalan framework (see "General Structure of Data Converters" available online,

[0228] https: / / github.com / scalan / scalan / blob / master / core / src / main / scala / scalan / Converters.scala#L159 ). The first type of isomorphism is "legal" objects. That is, like other data objects, they can be created, modified, composed, stored, etc. Scalan is a domain specific language (DSL) development framework based on staged evaluation of Scala compilation. Scalan uses a graph-based intermediate representation. Scalan further supports graph rewriting, slicing, isomorphic specialization and other general program transformations.

[0229]

[0230] 2) The converter code is virtualized in a manner known in the art (for more details, see "AUTOMATIC CODE VIRTUALIZATION FOR A COMPILER", patent application WO2017095257).

[0231] 3) Virtualization converters are used to specialize specific implementations of types A and B using specific data layouts. Examples of two different but isomorphic layouts are Fig.14The specialization is performed because it is known to use the technique of isomorphic specialization in the process of hierarchical evaluation of the abstract converter code shown in the above example code section of the converter between line and buffer API types.

[0232] Hierarchical evaluation, as defined in [1], is the process of symbolically evaluating program terms to produce a graph-based IR. This process can be combined with a set of rewriting rules (RWs). When these rules are applied to the nodes of a graph constructed by staged evaluation, the graph is transformed into another graph that is specialized for a predefined set of isomorphisms.

[0233] 4) Generate kernel code by using a target language code generator (Target Language Codegen) known in the art, and further process the intermediate representation (IR) of the dedicated converter.

[0234] The above items 1) to 4) have the effect of data transmission from type A specific layout to type B specific data layout, and can be used in the data transmission pipeline provided by the present invention.

[0235] Step 4: Package the kernel into a library:

[0236] The generated source codes of initialization kernel and extraction kernel are compiled by Java and C++ compilers respectively and packaged into domain-known libraries.

[0237] Step 5: Use the kernel in the data transfer pipeline:

[0238] As is known in the art, during runtime, the initialization kernel and extraction kernel libraries are dynamically linked with the ORM and DBMS components.

[0239] Next, combine Fig. 20 and Fig.21 The examples shown are used to emphasize some key points of the present invention. Fig. 20 As shown in Figure 1, data duplication (from the native side to the VM side) is avoided. Fig.21 In , it is shown how to generate database specific kernels. In this design, layout specific kernels are used to extract entity data. In addition, a single common metadata driven Codegen for both native and VM sides is provided. Different objects and tables can use different row buffer formats.

[0240] The prototype implementation of the present invention (hereinafter also referred to as DirectSqlite) is compared with the existing data access Java API android.database.sqlite provided by Android SDK. The TPC-H decision support benchmark database model is selected as the benchmark database, and a 1.5Mb database size is used.

[0241] For the existing Android API (left columns) and the DirectSqlite prototype implementation (right columns), the execution time of typical data access scenarios is as follows: Fig. 22 and Table 3. The results show a significant performance advantage when using DirectSqlite data access implementation. It should be noted that if the processing time exceeds 50 milliseconds, Google recommends offloading it to a separate thread. This makes the application code more complex and also consumes more power.

[0242] Table 3: Benchmark results for typical data access scenarios

[0243] Scenario Android,msec Direct,msec accelerate Open Database 106 29 3,66 Handling a single customer object 84 25 3,36

[0244] The potential impact of DirectSqlite usage is discussed below. Recently, a study was conducted to understand the usage statistics of Sqlite on the Android platform (O. Kennedy, A. Ajay, G. Challen, and L. Ziarek, "Pocket data: The need for TPC-MOBILE", Technology Conference on Performance Evaluation and Benchmarking, 2015), and the summary statistics of the results are as follows: Fig.23 As shown in the figure, most database queries are simple SELECT statements, and data is usually loaded from the database. The second main operation is simple INSERT and UPSERT. Table 4 shows the CPU time to process database statements.

[0245] Table 4: Total time spent on database operations

[0246]

[0247]

[0248] Taking into account SELECT, INSERT, UPSERT, and UPDATE operations, all of which require data transfer from Java to the native side, the total time can be estimated to be 10.5+1.3+1.9+1.9=15.6 hours. Since the DirectSqlite method of data transfer is about 3.5 times faster than the standard Android API, it is further estimated that the monthly CPU time reduction under typical usage may be 15.6–15.6 / 3.5=11.1 hours. For example, in the case of SELECT statements alone, if 3 hours are used instead of 10.5 hours during the speedup process, 7 hours of CPU time is saved, which can also save battery power. In addition, in Fig.24 In FIG. 5 , the type and number of SQL statements executed during startup and the query characteristics used are shown.

[0249] It should also be noted that the present invention provides cross-platform support. Examples of such cross-platform code are Fig.25 In particular, it supports running the same code on both Android and JVM.

[0250] In general, the present invention relates to an apparatus for extracting data from a database for object initialization, an apparatus for writing object data into a database, a data extraction method, and a method for writing data into a database. The present invention facilitates fast and memory-efficient data exchange between a native side and a virtual machine (VM) side. In particular, the data extraction kernel on the native side uses a database interface to read database data from multiple locations in the database and writes the database data into a buffer using a native side buffer application programming interface (API). The object initialization kernel on the VM side uses the native side buffer API to read the database data from the buffer and initialize multiple objects. Information indicating one of the multiple storage areas in the buffer in which the data extraction kernel writes the database data is known to the native side buffer API and the VM side buffer API.

Claims

1. A device for extracting data from a database, characterized in that: include: a database interface for enabling communication with said database; Virtual machine (VM); A buffer, including multiple storage areas; a kernel generator for generating an object initialization kernel based on a first isomorphism between respective object types of a plurality of objects and a data format of a region layout in the buffer; and generating a data extraction kernel based on a second isomorphism between a data format of the database data in the database interface and the data format of the regional layout in the buffer; wherein the first isomorphism and the second isomorphism are input to the kernel generator, and the isomorphism is a data structure representing a program code; The data extraction kernel of the native side, the data extraction kernel including a native side buffer application programming interface (API) for reading the database data from multiple locations in the database using the database interface and writing the database data to the buffer using the native side buffer API, the database data corresponding to the query result; the native side is an operating system process running a database management system DBMS; The object initialization kernel on the VM side includes a VM side buffer API, which is used to read the database data from the buffer using the VM side buffer API, and initialize the multiple objects using the database data read from the buffer; the VM side is an operating system process running a virtual machine VM; Wherein, information indicating one of the plurality of storage areas in the buffer in which the data fetch kernel writes the database data is known to the native-side buffer API and the VM-side buffer API.

2. The device according to claim 1, characterized in that: The data extraction kernel includes a native side multi-cursor to extract the database data from the database, the native side multi-cursor including a plurality of cursors; The object initialization kernel includes a VM-side multi-cursor to extract the database data from the buffer, and the VM-side multi-cursor includes a plurality of cursors; The cursor of the native-side multi-cursor is used to iterate items of the database data at one of the plurality of positions in the database independently of positions of other cursors of the native-side multi-cursor; as well as The cursor of the VM-side multi-cursor is used to iterate the items of the database data synchronously with the iteration of the cursor of the native-side multi-cursor.

3. The device according to claim 2, characterized in that Also includes: The cursor of the VM-side multi-cursor is used to pass an iteration call to the cursor of the native-side multi-cursor; as well as The cursor of the native side multi-cursor is used to execute cursor steps synchronously with the VM side database cursor by reading the items of the database data from the database and writing the items of the database data into the buffer when receiving the iteration call.

4. The device according to claim 3, characterized in that The plurality of storage areas are located on the native side; the VM side buffer API includes a direct buffer on the VM side; The cursor of the native-side multi-cursor is further used to transfer the position of one storage area in the plurality of storage areas to the direct buffer when receiving the iteration call; and The object initialization kernel is used for obtaining the database data for initializing the object by accessing the direct buffer.

5. The device according to claim 3, characterized in that The plurality of storage areas are located on the native side; The cursor of the native-side multi-cursor is further used to create a new direct buffer in the VM-side buffer API when receiving the iteration call, and pass a direct address of one of the multiple storage areas to the direct buffer through the buffer API; and The object initialization kernel is used for obtaining the database data for initializing the object by accessing the direct buffer.

6. The device according to claim 1, characterized in that The multiple storage areas are located on the VM side, and the data extraction kernel is used to write the database data into the buffer through the native side buffer API.

7. The device according to any one of claims 2 to 5, characterized in that The object initialization kernel is used for initializing an object in the plurality of objects by using the database data within a cursor step for each cursor in the native-side multi-cursor.

8. The device according to claim 1, characterized in that The kernel generator also inputs a data model, an object-relational mapping (ORM), a data access application programming interface (API), a definition of the native side buffer API and / or a definition of the VM side buffer API.

9. The device according to claim 1 or 8, characterized in that The kernel generator dynamically generates the object initialization kernel and the data extraction kernel at runtime.

10. The device according to any one of claims 1 to 9, characterized in that The multiple objects are Java objects, and the VM is a Java VM.

11. A device for writing data into a database, characterized in that: include: a database interface for enabling communication with said database; Virtual machine (VM); A buffer, including multiple storage areas; an object processing kernel on the VM side, the object processing kernel including a VM side buffer application programming interface (API) for writing object data of a plurality of objects into the buffer using the VM side buffer API; the VM side is an operating system process running a virtual machine VM; a data writing kernel on the native side, the data writing kernel comprising a native side buffer application programming interface (API) for reading the object data from the buffer using the native side buffer API and writing the object data to a plurality of locations in the database using the database interface; the native side being an operating system process running a database management system DBMS; Wherein, information indicating one of the plurality of storage areas in the buffer in which the object processing kernel writes the object data is known to the native-side buffer API and the VM-side buffer API.

12. A method for extracting data from a database, characterized in that: The following steps are involved: Reading database data from a plurality of locations in the database through a database interface, the database data corresponding to the query result; Using a native side buffer application programming interface (API) to write the database data into a buffer including multiple storage areas; the native side is an operating system process running a database management system DBMS; Using a virtual machine (VM) side buffer API to read database data from the buffer; the VM side is an operating system process running the virtual machine VM; and Initializing a plurality of objects using the database data read from the buffer; Wherein, information indicating one of the plurality of storage areas in the buffer in which database data is written is known to the native-side buffer API and the VM-side buffer API.

13. A method for writing data into a database, characterized in that: The following steps are involved: Writing object data of multiple objects into a buffer including multiple storage areas using a buffer application programming interface (API) on a virtual machine (VM) side; the VM side is an operating system process running the virtual machine VM; Using a native side buffer API to read the object data from the buffer; the native side is an operating system process running a database management system DBMS; and writing the object data into a plurality of locations in the database through the database interface; Herein, information indicating one of the plurality of storage areas in the buffer in which the object data is written is known to the native-side buffer API and the VM-side buffer API.

14. A computer-readable medium storing instructions, characterized in that: The instructions cause the computer to execute the method according to claim 12 or 13.

Images