A bus data processing system, method and device

Abstract

Embodiments of the present specification provide a bus data processing system, method and device, the system comprising a data storage layer and a data processing layer, the data storage layer comprising a memory module and an external storage module, the memory module being configured to store index data of bus data, and the external storage module being configured to store the bus data; the data processing layer being configured to perform a first operation and a second operation in parallel on real-time updated bus data, the first operation comprising updating the index data in the memory module according to the real-time updated bus data, and the second operation comprising storing the real-time updated bus data to the external storage module.

Description

TECHNICAL FIELD

[0001] The present document relates to the technical field of data processing, and particularly relates to a bus data processing system, method and device. BACKGROUND

[0002] A bus vehicle is a common traffic tool. In the process of running of the bus vehicle, real-time updating and storage of bus data are generally required, so that when a user needs to query bus data, a query service can be provided based on the stored bus data. In the updating and storage of the bus data, generally, in order to reduce the performance load of the system, a first-fall-disk-then-query external storage scheme is usually adopted, that is, the real-time updated bus data is first stored in the external storage, and then data query is performed based on the bus data in the external storage. However, with the intensive development of the bus system, more and more bus data need to be updated and stored in real time, and the traditional scheme has relatively low query efficiency when facing massive bus data, and cannot meet the real-time query demand of the user for the bus data. SUMMARY

[0003] The embodiments of the present specification provide a bus data processing system, method and device, which are used to solve the problem that the query efficiency is low when querying bus data based on the current updating and storage scheme of bus data, and the real-time query demand of the user for the bus data cannot be met.

[0004] To solve the above technical problems, the embodiments of the present specification are implemented as follows:

[0005] In a first aspect, a bus data processing system is provided, comprising a data storage layer and a data processing layer, wherein:

[0006] The data storage layer comprises a memory module and an external storage module, the memory module is used to store index data of bus data, and the external storage module is used to store bus data.

[0007] The data processing layer is used to perform a first operation and a second operation on real-time updated bus data in parallel, the first operation comprises updating index data in the memory module according to the real-time updated bus data, and the second operation comprises storing the real-time updated bus data to the external storage module.

[0008] In a second aspect, a bus data processing method is provided, comprising:

[0009] Receiving real-time updated bus data;

[0010] The first operation and the second operation are performed in parallel on the real-time updated bus data, the first operation comprises updating index data in an update memory module according to the real-time updated bus data, and the second operation comprises storing the real-time updated bus data to an external storage module.

[0011] In a third aspect, a bus data processing apparatus is provided, comprising:

[0012] a receiving module configured to receive real-time updated bus data;

[0013] a data processing module configured to perform a first operation and a second operation in parallel on the real-time updated bus data, the first operation comprising updating index data in an update memory module according to the real-time updated bus data, and the second operation comprising storing the real-time updated bus data to an external storage module.

[0014] In a fourth aspect, an electronic device is provided, comprising:

[0015] a processor; and

[0016] a memory arranged to store computer executable instructions that, when executed, cause the processor to perform the following operations:

[0017] receive real-time updated bus data;

[0018] perform a first operation and a second operation in parallel on the real-time updated bus data, the first operation comprising updating index data in an update memory module according to the real-time updated bus data, and the second operation comprising storing the real-time updated bus data to an external storage module.

[0019] In a fifth aspect, a computer readable storage medium is provided, the computer readable storage medium stores one or more programs, when the one or more programs are executed by an electronic device comprising a plurality of applications, the electronic device is caused to perform the following method:

[0020] receive real-time updated bus data;

[0021] perform a first operation and a second operation in parallel on the real-time updated bus data, the first operation comprising updating index data in an update memory module according to the real-time updated bus data, and the second operation comprising storing the real-time updated bus data to an external storage module.

[0022] The above at least one technical solution adopted by one or more embodiments of the present specification can achieve the following technical effects:

[0023] By dividing the storage of the bus data into index storage and data storage, when the bus data is updated, the real-time updated bus data is stored in the external storage module, and the index data corresponding to the bus data is stored in the internal storage module. In this way, when the real-time updated bus data is queried, the corresponding index can be queried in the internal storage module, and then the corresponding bus data is queried from the external storage module, so that the query efficiency is faster. Since the complete bus data is stored in the external storage module, the occupation of the internal storage module can be reduced. Therefore, by using the hybrid storage mechanism of internal and external storage, the query timeliness and memory occupation can be balanced. In addition, since the data storage operation and the index storage operation on the real-time updated bus data can be performed in parallel, the execution efficiency can also be improved. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present specification, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0025] Figure 1 is a structural schematic diagram of a bus data processing system according to an embodiment of the present specification;

[0026] Figure 2 is a flowchart of a bus data processing method according to an embodiment of the present specification;

[0027] Figure 3 is a structural schematic diagram of a bus data processing system according to an embodiment of the present specification;

[0028] Figure 4 is a flowchart of a bus data processing method according to an embodiment of the present specification;

[0029] Figure 5 is a structural schematic diagram of an electronic device according to an embodiment of the present specification;

[0030] Figure 6 is a structural schematic diagram of a bus data processing device according to an embodiment of the present specification. DETAILED DESCRIPTION

[0031] In order to make the person in the art better understand the technical solutions in the embodiments of the present specification, the technical solutions in the embodiments of the present specification will be clearly and completely described below in combination with the drawings in one or more embodiments of the present specification. Obviously, the described embodiments are only a part of the embodiments of the present specification, not all. Based on the embodiments in the present specification, all other embodiments obtained by those of ordinary skill in the art without creative labor should belong to the protection scope of the present document.

[0032] The technical solutions provided by the embodiments of the present specification will be described in detail below in combination with the drawings.

[0033] Figure 1 is a structural schematic diagram of a bus data processing system according to an embodiment of the present specification. Figure 1 The bus data processing system 10 shown includes a data storage layer 11 and a data processing layer 12, wherein:

[0034] The data storage layer 11 includes a memory module 111 and an external storage module 112, the memory module 111 is used to store index data of bus data, and the external storage module 112 is used to store bus data.

[0035] The data processing layer 12 is used to perform a first operation and a second operation on the real-time updated bus data in parallel, the first operation includes updating the index data in the memory module 111 according to the real-time updated bus data, and the second operation includes storing the real-time updated bus data to the external storage module 112.

[0036] The memory module 111 can be a system memory of the data processing system 10, and the external storage module 112 can be other storage devices in the data processing system 10 except the system memory, such as a database and the like. When storing bus data, the storage of bus data can be divided into index storage and data storage, the memory module 111 is used to store the index data corresponding to the bus data (i.e. index storage), and the external storage module 112 is used to store the complete bus data (i.e. data storage). Among them, the index data in the memory module 111 can be used to query the bus data in the external storage module 112.

[0037] The data processing layer 12 can process the continuously real-time updated bus data. Specifically, after the bus data management system collects the real-time updated bus data, the bus data management system can send the real-time updated bus data to the data processing layer 12, and the data processing layer 12 can perform a storage operation on the real-time updated bus data. The storage operation includes performing a first operation and a second operation in parallel. The first operation can be an index storage operation, that is, storing index data corresponding to the real-time updated bus data into the memory module 111, that is, updating the index data in the memory module 111 according to the real-time updated bus data. The second operation can be a data storage operation, that is, storing the bus data into the external storage module 112.

[0038] When the data processing layer 12 performs the first operation and the second operation in parallel, the data processing layer 12 can optionally be implemented by multiple threads, that is, when the data processing layer 12 stores the real-time updated bus data, the data processing layer 12 can start two threads, and the two threads perform in parallel, one thread is responsible for performing the first operation, and the other thread is responsible for performing the second operation. Alternatively, it can also be implemented in a distributed manner, for example, the data processing layer 12 includes multiple distributed data processing devices, and two data processing devices are responsible for performing the first operation and the second operation in parallel, that is, one data processing device performs the first operation, and the other data processing device performs the second operation. Of course, in other possible implementation manners, the parallel execution of the first operation and the second operation can also be implemented in other manners, which will not be illustrated one by one here.

[0039] By dividing the storage of the bus data into index storage and data storage, when the bus data is updated, the real-time updated bus data is stored into the external storage module, and the index data corresponding to the bus data is stored into the memory module. In this way, when the real-time updated bus data is queried, the corresponding index can be queried in the memory module first, and then the corresponding bus data is queried from the external storage module, and the query efficiency is relatively fast. Since the complete bus data is stored in the external storage module, the occupation of the memory module can be reduced. Therefore, by using the hybrid storage mechanism of internal and external storage fusion, the query timeliness and memory occupation can be balanced. In addition, since the data storage operation and the index storage operation on the real-time updated bus data can be performed in parallel, the execution efficiency can also be improved.

[0040] Optionally, the memory module 111 is configured to store index data of the bus data, and can be configured to store the index data according to a preset index structure. On this basis, the data processing layer 12 is configured to update the index data in the memory module according to the real-time updated bus data, and can be configured to update the index data in the memory module 111 according to the real-time updated bus data according to the preset index structure.

[0041] The preset index structure can represent a correspondence between at least two of a vehicle trajectory, a real-time position point of the vehicle, a region grid, and a region cube. The memory module 111 stores index data according to the preset index structure, which can be a mapping table in the memory module 111 that records a correspondence between at least two of a vehicle trajectory, a real-time position point of the vehicle, a region grid, and a region cube, to facilitate fast querying of real-time data. The vehicle trajectory can be a trajectory of a bus vehicle, and the real-time position point of the vehicle can correspond to a real-time position of the bus vehicle. Generally, the position of the bus vehicle is updated in real time during vehicle operation, the vehicle trajectory can be a trajectory formed by positions of all points passed by the bus vehicle, and the vehicle trajectory extends by one point each time the real-time position of the bus vehicle is updated. The region grid can be a grid obtained by dividing a region range in which the bus vehicle travels, and the region cube can also be a grid obtained by dividing the region range in which the bus vehicle travels. The difference is that the region grid does not contain time information, while the region cube contains time information. When updating the index structure related to the region grid, historical data is erased, and the updated index structure corresponds to the latest bus data. When updating the index structure related to the region cube, historical data is not erased, and the updated index structure includes multiple index structures at different times, having a time dimension.

[0042] Optionally, as an embodiment, the preset index structure can include at least one of the following (1) to (6):

[0043] (1) A first index structure representing a correspondence between a vehicle trajectory and a real-time position point of the vehicle.

[0044] The first index structure can be represented as TP-index, which records information of a real-time position point corresponding to a trajectory of a bus vehicle. In the memory module 111, a list is maintained for the first index structure, and the list records information of all sampling points (i.e., real-time position points) of the entire trajectory.

[0045] (2) A second index structure representing a correspondence between a region grid and a vehicle trajectory passing through the region grid.

[0046] The second index structure can be represented as GT-index, which records information of all trajectories passing through a grid corresponding to the grid. In the memory module 111, a trajectory list is maintained for the second index structure, and the trajectory list records information of all trajectories passing through the grid.

[0047] (3) A third index structure representing a correspondence between a vehicle trajectory and a region cube through which the vehicle trajectory passes.

[0048] The third index structure can be denoted as TC-index, which records information of all cubes through which a trajectory passes. Compared with a grid, a cube adds information of a time dimension, which supports a user to query data from a time range.

[0049] (4) A fourth index structure representing a correspondence between a region cube and a vehicle trajectory passing through the region cube.

[0050] The fourth index structure can be denoted as CT-index, which records information of a trajectory passing through a cube. The fourth index structure supports a user to query a trajectory in a specific space and at a specific time, and improves query efficiency.

[0051] (5) A fifth index structure representing a correspondence between a region cube and a real-time location point of a vehicle in the region cube.

[0052] The fifth index structure can be denoted as CP-index, which records information of a real-time location point of a vehicle passing through a cube. The fifth index structure supports a user to query location information at a latest time.

[0053] (6) A sixth index structure representing a correspondence between a vehicle trajectory and a latest location point of the vehicle.

[0054] The sixth index structure can be denoted as TIP-index, which records information of a latest location point of a vehicle in a trajectory. The sixth index structure facilitates a user to quickly locate a latest location point of a specified trajectory.

[0055] It should be noted that, in other possible implementations, the index structure can also be a correspondence between other combinations of a vehicle trajectory, a real-time location point of a vehicle, a region grid, and a region cube. Here, the index structures are not enumerated one by one, and the embodiments of the present disclosure are only described by taking the six commonly used index structures as examples.

[0056] Optionally, the index data stored in the memory module 111 can be real-time index data, and the external storage module 112 is further configured to store historical index data, and the data processing layer 12 can be further configured to transfer the historical index data in the memory module 111 to the external storage module 112. In actual application scenarios, the bus data is constantly updated in real time, and when new bus data is generated, the data processing layer 12 can update the index data in the memory module 111 according to the newly generated bus data, and other index data in the memory module 111 can be regarded as historical index data compared with the real-time updated index data, and the data processing layer 12 can transfer these historical index data to the external storage module 112, and the external storage module 112 stores these historical index data. In this way, only real-time index data can be stored in the memory module 111, thereby reducing memory occupation and improving real-time query efficiency of bus data.

[0057] When the data processing layer 12 transfers the historical index data to the external storage module 112, the historical index data can be directly stored in the external storage module 112, or for performance considerations, the historical index data can be first processed by merging, and then the merged historical index data is stored in the external storage module 112, such as using a merging algorithm for time slicing of the historical index data, and merging and storing the empty slices or sparse slices (time slices corresponding to less total amount of index data), and adding “Merge-” as the index name prefix.

[0058] It should be noted that the division of historical index data and real-time index data can include various ways, for example, as described above, whenever new bus data is generated, the index data corresponding to these new bus data is real-time index data, and other index data can be regarded as historical index data. For example, a time threshold can be set, and if the generation time of the index data is within the time threshold from the current time, it can be regarded as real-time index data, and other index data is regarded as historical index data. For example, a time threshold of 1 hour can be set, and the index data generated within the last hour can be regarded as real-time index data, and the remaining index data is regarded as historical index data. Here, the division method of historical index data and real-time index data will not be illustrated one by one.

[0059] In the embodiments of the present specification, Figure 1 The data processing system 10 shown can further include a data query layer 13, which can be configured to receive a query request of bus data and implement query of the bus data according to the query request. Specifically, the data query layer 13 can include an index query module 131 and a data query module 132, the index query module 131 is configured to query the corresponding index according to the query request of the bus data, and the data query module 132 is configured to query the corresponding bus data according to the index of the bus data.

[0060] The query request of the bus data can include a real-time query request and a historical query request. The data query layer 13 can implement real-time query and historical query of the bus data. Specifically:

[0061] In the case where the query request includes the real-time query request, the index query module 131 is configured to query the corresponding first index in the memory module 111 according to the real-time query request, and the data query module 132 is configured to query the corresponding bus data in the external storage module 112 according to the first index;

[0062] In the case where the query request includes the historical query request, the index query module 131 is configured to query the corresponding second index in the external storage module 112 according to the historical query request, and the data query module 132 is configured to query the corresponding bus data in the external storage module 112 according to the second index.

[0063] Based on the above description, it can be known that the memory module 111 can be used to store real-time index data, the external storage module 112 can be used to store historical index data and complete bus data, the real-time index data is used for real-time query, the historical index data is used for historical query, and the complete bus data contains both real-time updated bus data and historical bus data. Therefore, when the query request is based on the query request to query the bus data, in the case where the query request includes the real-time query, the index query module 131 can query the corresponding real-time index, i.e., the first index, from the memory module 111 according to the real-time query request, and then the data query module 132 queries the corresponding bus data in the external storage module 112 according to the first index; in the case where the query request includes the historical query, the index query module 131 can query the corresponding historical index, i.e., the second index, from the external storage module 112 according to the historical query request, and then the data query module 132 continues to query the corresponding bus data in the external storage module 112 according to the second index.

[0064] The index query module 131 can query according to the query condition contained in the real-time query request when querying the corresponding first index from the memory module 111 according to the real-time query request. The query condition can be input by the user, or can be provided by the data query layer 13 and selected by the user. The query condition can contain time information, location information and the like corresponding to the bus data to be queried. When querying the first index according to the query condition, the query condition can be mapped to a regional grid or a regional square, and then further mapped to a vehicle trajectory or a real-time location point of a vehicle, so that the first index can be obtained. The first index can be any one or a combination (i.e., a hybrid index) of the above-mentioned six kinds of preset index structures. For example, if the user wants to query the latest bus data passing through location A, the index query module 131 can determine the regional square in which location A is located when querying the corresponding first index, map location A to the regional square, obtain the ID of the regional square in which location A is located, and then further query the trajectory ID according to the ID and the CT-index, and further query the ID of the real-time location point according to the trajectory ID and the TIP-index, so that the hybrid index corresponding to the latest bus data of location A can be obtained, and the hybrid index is the first index.

[0065] Of course, the above only exemplarily illustrates how to determine the corresponding first index according to the real-time query request. In other possible implementation manners, the first index can also be determined by other manners, which will not be exemplarily illustrated here.

[0066] The external storage module 112 can be a relational database, and optionally can be a distributed relational database, such as OceanBase. When storing the bus data to the external storage module 112, the data processing layer 12 can store the bus data and the key for querying the bus data correspondingly, and the key and the bus data can be in a one-to-one correspondence. In this way, when querying the corresponding bus data in the external storage module 112 according to the first index, the data query module 132 can query the bus data uniquely corresponding to the key in the external storage module by taking the first index as the key. The bus data is the bus data requested to be queried by the real-time query request.

[0067] When the index query module 131 queries the corresponding second index from the external storage module 112 according to the historical query request, and the data query module 132 queries the corresponding bus data in the external storage module 112 according to the second index, the specific implementation manner can be the same as that when the index query module 131 queries the first index and the data query module 132 queries the corresponding bus data according to the first index, which will not be described in detail here.

[0068] In the embodiments of the present specification, for Figure 1The data processing layer 12 shown can also execute a third operation in parallel while performing the first and second operations (i.e., index storage and data storage operations). This third operation includes sending real-time updated bus data to at least one client, which can be used to display the real-time updated bus data. Thus, since index storage, data storage, and data display can be performed synchronously for real-time updated bus data, execution efficiency can be effectively improved. The parallel execution of the first, second, and third operations can be implemented using three parallel threads, or it can be implemented in a distributed manner; no specific limitation is made here.

[0069] Based on the aforementioned public transport data processing system, embodiments of this specification also provide a public transport data processing method based on this system, such as... Figure 2 As shown. Figure 2 This is a flowchart illustrating a bus data processing method according to one embodiment of this specification. Figure 2 The method shown can be derived from Figure 1 The processing system shown is executed, and the method may include the following steps.

[0070] S202: Receives real-time updated bus data.

[0071] Real-time updated public transport data can be collected or gathered in real time by a public transport data management system. When new public transport data is generated, the management system can send this real-time updated data to the data processing system, which can then receive the updated data. The public transport data management system can be a GTFS (Government-to-Footage System), which can easily obtain a large amount of basic information about the urban public transport system (such as bus routes, stop locations, vehicle schedules (timetables), fares, etc.) and real-time data (such as the real-time location of buses, information on the next stop, and the current operating status of vehicles).

[0072] S204: Perform the first operation and the second operation in parallel on the real-time updated bus data. The first operation includes updating the index data in the memory module according to the real-time updated bus data. The second operation includes storing the real-time updated bus data in the external storage module.

[0073] After receiving real-time updated bus data, the data processing system can perform indexing and storage operations and data storage operations on the bus data in parallel, namely the first and second operations mentioned above. The parallel execution of the first and second operations can be achieved through multiple threads or a distributed approach.

[0074] Optionally, when storing the index data, the memory module can store the index data according to a preset index structure, so that when updating the index data in the memory module according to the real-time updated bus data, the following operations can be included:

[0075] According to the real-time updated bus data, the index data in the memory module is updated according to a preset index structure.

[0076] The preset index structure can represent a correspondence between at least two of a vehicle trajectory, a real-time position point of a vehicle, a region grid, and a region cube. The memory module 111 stores the index data according to the preset index structure, which can be a mapping table that records the correspondence between at least two of the vehicle trajectory, the real-time position point of the vehicle, the region grid, and the region cube in the memory module 111, facilitating fast querying of real-time data. Optionally, the preset index structure can include at least one of the following:

[0077] The first index structure represents a correspondence between a vehicle trajectory and a real-time position point of a vehicle;

[0078] The second index structure represents a correspondence between a region grid and a vehicle trajectory passing through the region grid;

[0079] The third index structure represents a correspondence between a vehicle trajectory and a region cube through which the vehicle trajectory passes;

[0080] The fourth index structure represents a correspondence between a region cube and a vehicle trajectory passing through the region cube;

[0081] The fifth index structure represents a correspondence between a region cube and a real-time position point of a vehicle in the region cube;

[0082] The sixth index structure represents a correspondence between a vehicle trajectory and a latest position point of a vehicle.

[0083] For specific descriptions of the above preset index structures, please refer to the specific descriptions of the corresponding contents in the embodiments shown in Figure 1 The specific descriptions of the corresponding contents in the embodiments shown in

[0084] In the embodiments of the present specification, the index data stored in the memory module can be real-time index data, and the external storage module is further configured to store historical index data. In this way, when updating the index data in the memory module according to the real-time updated bus data, the historical index data in the memory module can also be transferred to the external storage module, thereby reducing memory occupation and improving real-time query efficiency of bus data.

[0085] In addition to storing real-time updated bus data using the methods described above, the data processing system can also enable real-time and historical queries of bus data. Specifically, this can include the following steps:

[0086] Receive public transport data query requests, including real-time query requests and historical query requests;

[0087] In the case of a query request including a real-time query request, the corresponding first index is queried in the memory module according to the real-time query request; and the corresponding bus data is queried in the external storage module according to the first index.

[0088] If the query request includes historical query requests, the corresponding second index is retrieved from the external storage module based on the historical query requests; and the corresponding bus data is retrieved from the external storage module based on the second index.

[0089] When a user wants to query bus data, they can send a query request to the data processing system. This query request can be a real-time query for current bus data or a historical query for historical bus data. Considering that the memory module stores real-time index data and the external storage module stores historical index data and bus data (including both real-time and historical bus data), if the query request is a real-time request, the corresponding first index can be retrieved from the memory module first, and then the corresponding bus data can be retrieved from the external storage module based on the first index. If the query request is a historical query, the corresponding second index can be retrieved from the external storage module, and then the corresponding bus data can be retrieved from the external storage module based on the second index. For detailed implementation details, please refer to [link to implementation details]. Figure 1 The specific implementation of the corresponding steps in the illustrated embodiments will not be repeated here.

[0090] The external storage module can be a relational database. When storing public transport data, a relational database can store the public transport data and the keys used to query the public transport data in a corresponding manner. In this way, when querying the corresponding public transport data in the external storage module based on the first index / second index, the first index / second index can be used as the key to retrieve the public transport data that uniquely corresponds to the key.

[0091] Optionally, while the data processing system executes the first and second operations in parallel, a third operation can also be executed in parallel. The third operation includes sending real-time updated bus data to at least one client, which is used to display the real-time updated bus data.

[0092] It should be noted that, Figure 2 For detailed implementation methods of each step shown, please refer to [link / reference]. Figure 1 The specific implementation of the corresponding content in the illustrated embodiments will not be described in detail here.

[0093] In the embodiments of the present specification, by dividing the storage of the bus data into index storage and data storage, when the bus data is updated, the real-time updated bus data is stored in the external storage module, and the index data corresponding to the bus data is stored in the internal storage module. In this way, when the real-time updated bus data is queried, the corresponding index can be queried in the internal storage module first, and then the corresponding bus data is queried from the external storage module, and the query efficiency is faster. Since the complete bus data is stored in the external storage module, the occupation of the internal storage module can be reduced. Therefore, by using the hybrid storage mechanism of internal and external storage, the query timeliness and memory occupation can be balanced. In addition, since the data storage operation and the index storage operation on the real-time updated bus data can be performed in parallel, the execution efficiency can also be improved.

[0094] In order to facilitate understanding of the technical solutions provided by the embodiments of the present specification, the following Figure 3 and Figure 4 may be referred to. Figure 3 and Figure 4 are a structure diagram of a bus data processing system and a schematic diagram of a bus data processing method in a more specific implementation manner of the technical solutions provided by the present specification.

[0095] As shown in Figure 3 , GTFS is a management system of bus data, and the data processing system provided by the embodiments of the present specification can be deployed in the management system, or the management system can be connected with the data processing system provided by the embodiments of the present specification. When real-time updated bus data (i.e. A city data and B city data) enters the GTFS, the data processing system can perform index storage and data storage operations on the real-time updated bus data in parallel, that is, update the real-time index data in the internal storage according to the real-time updated bus data and store the bus data in the external storage. The external storage is an OceanBase database. In addition, the real-time updated bus data can also be displayed in the front end of the GTFS in parallel, and the user can access the real-time updated bus data in the front end.

[0096] For the index storage portion, the embodiments in this specification can employ LSM-Tree technology. This technology is a multi-level indexing technique that leverages the speed advantage of sequential writes, sacrificing a small amount of read performance for highly efficient write performance, and provides excellent support for high-speed write scenarios such as real-time data streams. Specifically, the index storage portion can be divided into real-time index storage and historical index storage. The real-time index is stored in memory and can be used for real-time queries of public transport data, while the historical index is stored in external storage, i.e., OceanBase, and can be used for historical queries of public transport data. The real-time data index (i.e., real-time index data) can be considered a first-level index. When the real-time data index becomes a historical data index, it can be stored in external storage as a second-level index. The structure of the second-level index is the same as that of the first-level index. Optionally, the second-level index can be an index merged from the first-level index.

[0097] based on Figure 3 The bus data processing system shown can process bus data in the following manner: Figure 4 As shown. Figure 4 In this context, the real-time data processing engine can be... Figure 1 As shown in the data processing layer, when real-time updated bus data enters the real-time data processing engine, the real-time data processing engine can... Figure 4 The three threads shown execute three operations in parallel: a client-side real-time data distribution thread sends real-time updated bus data to at least one client so that at least one client can display the real-time updated bus data; a backend OceanBase storage thread stores the real-time updated bus data in the OceanBase database; and a backend LSM-Tree indexing thread updates the index data in memory. The index structure of the index data in memory can include various methods.

[0098] For performance reasons, only the unique key is stored when storing index data; the entire data is not saved. After a user retrieves the unique key of specific data using the index, they need to read the complete data structure from the database. Any database system that supports unique key queries can be used as support, such as... Figure 4 As shown, OceanBase, an enterprise-grade native distributed database, is used as the storage system.

[0099] When a user queries public transportation data, they can enter a query request through the web service frontend. The web service backend can then process the query request and return the corresponding query results (i.e., Figure 4 (The response request is shown). Figure 4Taking a real-time query request as an example, after receiving the real-time query request, the Web service backend can query the corresponding index in the memory, and then query the corresponding bus data from the OceanBase database according to the index.

[0100] Figure 4 The OceanBase database in the memory can also store historical index data. Specifically, when the index in the memory is discarded (data that is too old is discarded; the memory space is insufficient and the data that is the longest time ago needs to be cleaned up, etc.), the non-real-time index data needs to be transferred to the external storage to continue to provide query acceleration for historical data of users, that is, when the query request is a historical query request, the corresponding historical index can be queried from the OceanBase database, and then the corresponding bus data is queried in the OceanBase database according to the historical index. When storing historical data in the OceanBase database, an index structure with the same effect still needs to be maintained, but for performance considerations, the time slicing of the historical data will use a merging algorithm to merge and store empty slices or sparse slices (time slices with less total data), and add "Merge-" as a prefix of the index name.

[0101] Figure 3 and Figure 4 The specific implementation of the embodiments shown in Figure 1 and Figure 2 will not be described in detail here.

[0102] The technical solution provided by the embodiments of the present specification adopts a hybrid storage index mechanism of internal and external storage fusion. Compared with other solutions, the index of real-time data is directly updated to the memory, and the query efficiency is faster. On the other hand, the complete data is still in the external relational database, and only the unique key is retained as the index basis, which reduces the system occupation. The hybrid storage index mechanism of internal and external storage fusion can achieve a good balance between efficiency and space occupation. The sharing mechanism of critical resources by using multi-thread programming realizes parallel operation of real-time data streams, which can overcome the problem of low efficiency caused by sequential processing of real-time data in the traditional way of "first falling into the database and then querying". In theory, it can achieve synchronous execution of multiple tasks such as building index, data falling into database, data updating, etc., and greatly improves the execution efficiency.

[0103] The above describes particular embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order than the order in which they are recited and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or necessary.

[0104] Figure 5 Figure 1 is a structural schematic diagram of an electronic device according to an embodiment of the present specification. Please refer to Figure 5 At the hardware level, the electronic device includes a processor, and optionally further includes an internal bus, a network interface, and a memory. The memory can include a memory, such as a high-speed random-access memory (RAM), and can also include a non-volatile memory, such as at least one disk memory. Of course, the electronic device can also include other hardware required by the business.

[0105] The processor, the network interface, and the memory can be connected to each other through the internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0106] The memory is used to store programs. Specifically, the program can include program code, and the program code includes computer operation instructions. The memory can include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0107] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs, and forms a bus data processing device at the logical level. The processor executes the program stored in the memory, and is specifically used to perform the following operations:

[0108] Receive real-time updated bus data;

[0109] The first operation and the second operation are performed in parallel on the real-time updated bus data, the first operation comprises updating index data in an update memory module according to the real-time updated bus data, and the second operation comprises storing the real-time updated bus data to an external memory module.

[0110] The bus data processing apparatus disclosed in the embodiments of the present specification can perform the method as described above. Figure 5 The method performed by the bus data processing apparatus disclosed in the embodiments of the present specification can be applied in a processor or implemented by the processor. The processor can be an integrated circuit chip with processing capability. In the implementation process, each step of the method described above can be completed by hardware integrated logic circuit or software form of instructions in the processor. The processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the present specification can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present specification can be directly embodied as a hardware coding processor for execution, or a combination of hardware and software modules in the coding processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the memory is read by the processor, and the hardware thereof is combined to complete the steps of the method described above.

[0111] The electronic device can also perform the method of Figure 2 and realize the functions of the bus data processing apparatus in the embodiments as shown in Figure 2 The embodiments of the present specification will not be described here.

[0112] Of course, in addition to the software implementation, the electronic device of the embodiments of the present specification does not exclude other implementation manners, such as logic devices or a combination of software and hardware, etc. That is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0113] The embodiments of the present specification also provide a computer readable storage medium storing one or more programs, the one or more programs comprising instructions which, when executed by a portable electronic device comprising a plurality of applications, can cause the portable electronic device to perform the method of the embodiments shown above, and specifically to perform the following operations: Figure 2 The embodiments of the present specification also provide a computer readable storage medium storing one or more programs, the one or more programs comprising instructions which, when executed by a portable electronic device comprising a plurality of applications, can cause the portable electronic device to perform the method of the embodiments shown above, and specifically to perform the following operations:

[0114] receiving real-time updated public transportation data;

[0115] performing a first operation and a second operation on the real-time updated public transportation data in parallel, the first operation comprising updating index data in an update memory module according to the real-time updated public transportation data, and the second operation comprising storing the real-time updated public transportation data to an external storage module.

[0116] Figure 6 Fig. 1 is a schematic diagram of a structure of a public transportation data processing apparatus 60 according to an embodiment of the present specification. Please refer to Figure 6 In a software implementation, the public transportation data processing apparatus 60 can comprise a receiving module 61 and a data processing module 62, wherein:

[0117] The receiving module 61 receives real-time updated public transportation data.

[0118] The data processing module 62 performs a first operation and a second operation on the real-time updated public transportation data in parallel, the first operation comprising updating index data in an update memory module according to the real-time updated public transportation data, and the second operation comprising storing the real-time updated public transportation data to an external storage module.

[0119] Optionally, the data processing module 62 updates index data in an update memory module according to the real-time updated public transportation data, comprising:

[0120] updating the index data in the memory module according to the real-time updated public transportation data in a preset index structure.

[0121] Optionally, the preset index structure comprises at least one of the following:

[0122] a first index structure representing a corresponding relationship between a vehicle trajectory and a real-time position point of a vehicle;

[0123] a second index structure representing a corresponding relationship between a region grid and a vehicle trajectory passing through the region grid;

[0124] a third index structure representing a corresponding relationship between a vehicle trajectory and a region grid through which the vehicle trajectory passes;

[0125] a fourth index structure representing a corresponding relationship between a region grid and a vehicle trajectory passing through the region grid.

[0126] a fifth index structure representing a correspondence between the area grid and a real-time position point of the vehicle in the area grid;

[0127] a sixth index structure representing a correspondence between the vehicle trajectory and the latest position point of the vehicle;

[0128] wherein the area grid does not contain time information, and the area grid contains time information.

[0129] Optionally, the data processing module 62 further comprises:

[0130] transferring the historical index data in the memory module to the external storage module.

[0131] Optionally, the data processing module 62 further comprises:

[0132] receiving a query request of bus data, the query request comprising a real-time query request and a historical query request;

[0133] in a case where the query request comprises the real-time query request, querying a corresponding first index in the memory module according to the real-time query request; querying corresponding bus data in the external storage module according to the first index;

[0134] in a case where the query request comprises the historical query request, querying a corresponding second index in the external storage module according to the historical query request; querying corresponding bus data in the external storage module according to the second index.

[0135] Optionally, the data processing module 62, according to the first index, queries corresponding bus data in the external storage module, comprising:

[0136] querying, as a key, the first index in the external storage module to obtain bus data corresponding to the key uniquely.

[0137] Optionally, the data processing module 62, in a case where the first operation and the second operation are executed in parallel, further comprises:

[0138] executing a third operation in parallel, the third operation comprising sending the real-time updated bus data to at least one client, the at least one client being configured to display the real-time updated bus data.

[0139] The bus data processing apparatus 60 provided by the embodiments of the present specification can also execute the method of Figure 2 , and realize the functions of the bus data processing apparatus in the embodiments shown in Figure 2 . The embodiments of the present specification will not be described here again.

[0140] In conclusion, the above merely describes preferred embodiments of the present specification, and is not intended to limit the protection scope of the present document. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of the present specification shall be included in the protection scope of the present document.

[0141] The systems, apparatuses, modules or units illustrated by the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0142] The computer readable medium includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to computing devices. According to the definition herein, computer readable medium does not include transitory computer readable media, such as modulated data signals and carriers.

[0143] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0144] The various embodiments described in this specification are presented by way of example, and each embodiment is not inherently more important than any other embodiment. The same can hold true for the various implementations of the systems and methods within each embodiment.

Claims

1. A public transportation data processing system, comprising a data storage layer, a data processing layer, and a data query layer, wherein: The data storage layer includes a memory module and an external storage module. The memory module is used to store index data of the bus data, and the external storage module is used to store the bus data itself. The data processing layer is used to perform a first operation and a second operation in parallel on the real-time updated bus data. The first operation includes updating the index data in the memory module according to the real-time updated bus data, and the second operation includes storing the real-time updated bus data in the external storage module. The data query layer includes an index query module and a data query module. The index query module is used to query the corresponding index according to the query request of the bus data, and the data query module is used to query the corresponding bus data according to the index of the bus data. The query request includes real-time query requests and historical query requests; When the query request includes a real-time query request, the index query module is used to query the corresponding first index in the memory module according to the real-time query request, and the data query module is used to query the corresponding bus data in the external storage module according to the first index. When the query request includes a historical query request, the index query module is used to query the corresponding second index in the external storage module according to the historical query request, and the data query module is used to query the corresponding public transport data in the external storage module according to the second index.

2. The system as described in claim 1, wherein the memory module is used to store index data of the public transportation data according to a preset index structure; The data processing layer is used to update the index data in the memory module according to the preset index structure based on the real-time updated bus data.

3. The system as described in claim 2, wherein the preset index structure includes at least one of the following: The first index structure represents the correspondence between the vehicle trajectory and the vehicle's real-time location. The second index structure represents the correspondence between the regional grid and the vehicle trajectory passing through the regional grid; The third index structure represents the correspondence between the vehicle trajectory and the area grid through which the vehicle trajectory passes; The fourth index structure represents the correspondence between the area grid and the vehicle trajectories passing through the area grid; The fifth index structure represents the correspondence between the area grid and the real-time location points of the vehicles within the area grid; The sixth index structure represents the correspondence between the vehicle trajectory and the vehicle's latest location. The regional grid does not contain time information, while the regional squares do contain time information.

4. In the system as described in claim 1, the index data stored in the memory module is real-time index data, the external storage module is also used to store historical index data, and the data processing layer is also used to transfer the historical index data in the memory module to the external storage module.

5. The system as described in claim 1, wherein the external storage module includes a relational database; in, The data query module is used to query the corresponding bus data in the external storage module according to the first index, including: The data query module is used to use the first index as a key to query the bus data that uniquely corresponds to the key in the external storage module.

6. The system of claim 1, wherein the data processing layer is further configured to execute a third operation in parallel while executing the first operation and the second operation in parallel; in, The third operation includes sending the real-time updated bus data to at least one client, the at least one client being used to display the real-time updated bus data.

7. A bus data processing method based on a bus data processing system as described in any one of claims 1 to 6, comprising: Receive real-time updated bus data; The first operation and the second operation are performed in parallel on the real-time updated bus data. The first operation includes updating the index data in the memory module according to the real-time updated bus data. The second operation includes storing the real-time updated bus data in the external memory module. Receive query requests for public transport data, including real-time query requests and historical query requests; If the query request includes a real-time query request, the corresponding first index is queried in the memory module according to the real-time query request; The corresponding bus data is queried in the external storage module according to the first index; If the query request includes a historical query request, the corresponding second index is queried in the external storage module according to the historical query request; and the corresponding bus data is queried in the external storage module according to the second index.

8. The method as described in claim 7, wherein updating the index data in the memory module according to the real-time updated bus data includes: Based on the real-time updated bus data, the index data in the memory module is updated according to the preset index structure.

9. The method of claim 8, wherein the preset index structure includes at least one of the following: The first index structure represents the correspondence between the vehicle trajectory and the vehicle's real-time location. The second index structure represents the correspondence between the regional grid and the vehicle trajectory passing through the regional grid; The third index structure represents the correspondence between the vehicle trajectory and the area grid through which the vehicle trajectory passes; The fourth index structure represents the correspondence between the area grid and the vehicle trajectories passing through the area grid; The fifth index structure represents the correspondence between the area grid and the real-time location points of the vehicles within the area grid; The sixth index structure represents the correspondence between the vehicle trajectory and the vehicle's latest location. The regional grid does not contain time information, while the regional squares do contain time information.

10. The method of claim 7, further comprising: The historical index data in the memory module is transferred to the external storage module.

11. The method of claim 7, wherein querying the corresponding bus data in the external storage module according to the first index includes: Using the first index as the key, query the bus data that uniquely corresponds to the key in the external storage module.

12. The method of claim 7, wherein, when the first operation and the second operation are performed in parallel, the method further comprises: A third operation is performed in parallel, the third operation including sending the real-time updated bus data to at least one client, the at least one client being used to display the real-time updated bus data.

13. A public transport data processing device, comprising: The receiving module receives real-time updated bus data; The data processing module performs a first operation and a second operation in parallel on the real-time updated bus data. The first operation includes updating the index data in the memory module according to the real-time updated bus data. The second operation includes storing the real-time updated bus data in the external storage module. The data query module receives query requests for bus data, including real-time query requests and historical query requests. If the query request includes a real-time query request, it queries the corresponding first index in the memory module based on the real-time query request; and queries the corresponding bus data in the external storage module based on the first index. If the query request includes a historical query request, it queries the corresponding second index in the external storage module based on the historical query request; and queries the corresponding bus data in the external storage module based on the second index.

14. An electronic device comprising: processor; as well as Memory configured to store computer-executable instructions that, when executed, cause the processor to perform the following operations: Receive real-time updated bus data; The first operation and the second operation are performed in parallel on the real-time updated bus data. The first operation includes updating the index data in the memory module according to the real-time updated bus data. The second operation includes storing the real-time updated bus data in the external memory module. Receive query requests for public transport data, including real-time query requests and historical query requests; If the query request includes a real-time query request, the corresponding first index is queried in the memory module according to the real-time query request; The corresponding bus data is queried in the external storage module according to the first index; If the query request includes a historical query request, the corresponding second index is queried in the external storage module according to the historical query request; and the corresponding bus data is queried in the external storage module according to the second index.

15. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of applications, cause the electronic device to perform the following method: Receive real-time updated bus data; The first operation and the second operation are performed in parallel on the real-time updated bus data. The first operation includes updating the index data in the memory module according to the real-time updated bus data. The second operation includes storing the real-time updated bus data in the external memory module. Receive query requests for public transport data, including real-time query requests and historical query requests; If the query request includes a real-time query request, the corresponding first index is queried in the memory module according to the real-time query request; The corresponding bus data is queried in the external storage module according to the first index; If the query request includes a historical query request, the corresponding second index is queried in the external storage module according to the historical query request; and the corresponding bus data is queried in the external storage module according to the second index.

Images