Data service system, method, device, medium and product
By using a background thread in the data service system to update the second memory, the problem of data update affecting query performance is solved, and efficient data query performance and real-time performance are achieved.
Patent Information
- Application Number
- CN202410319113.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-19
- Publication Date
- 2025-09-19
AI Technical Summary
Data updates affect data query performance, resulting in poor query performance.
The first data processing device is used to obtain the second data set and store it in the first memory. The second data processing device updates the second memory through the background thread, and the main thread independently performs data query.
It avoids the mutual influence between data update and query, and improves query performance and real-time performance.
Smart Images
Figure CN120670431A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data service system, method, device, medium, and product. Background Art
[0002] With the development of computer technology, the storage and query of massive amounts of data has become a hot topic. In practical applications, this problem can be achieved through databases such as MySQL (a relational database management system) or KV (a database that stores data in key-value pairs).
[0003] For example, the data collector can first upload the queryable data to the database, and then the data query party can query the database to obtain the required data. When the queryable data is updated, the data collector needs to update the data in the database accordingly. However, this process may affect the data query process and lead to poor query performance. Summary of the Invention
[0004] The embodiments of the present application provide a data service system, method, device, medium and product to avoid affecting the query process when data is updated and improve query performance.
[0005] The embodiments of this application disclose the following technical solutions:
[0006] In a first aspect, an embodiment of the present application provides a data service system, comprising: a first data processing device, a second data processing device, a first memory, and a second memory, wherein the first memory and the second memory both include a first data set;
[0007] The first data processing device is configured to obtain a second data set and send the second data set to the first memory for storage, wherein the second data set includes an update result of the first data set;
[0008] The second data processing device is used to call a background thread to obtain the second data set from the first memory and update the second memory based on the second data set; in response to a query request for target data in the second data set, call the main thread to obtain the target data from the second memory.
[0009] In a second aspect, an embodiment of the present application provides a data service method, which is applied to a data service system. The data service system includes a first data processing device, a second data processing device, a first memory, and a second memory. The first memory and the second memory both include a first data set. The data service method includes:
[0010] The first data processing device acquires a second data set and sends the data to the first memory for storage, wherein the second data set includes an update result of the first data set;
[0011] The second data processing device calls a background thread to obtain the second data set from the first memory, and updates the second memory based on the second data set;
[0012] In response to a query request for target data in the second data set, the second data processing device calls a main thread to obtain the target data from the second memory.
[0013] In a third aspect, an embodiment of the present application provides a data service device, the device including a processor and a memory:
[0014] The memory is used to store a computer program and transmit the computer program to the processor;
[0015] The processor is configured to execute the steps performed by the first data processing device or the steps performed by the second data processing device according to the instructions in the computer program.
[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the steps performed by the above-mentioned first data processing device, or the steps performed by the second data processing device.
[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program or instructions, which, when executed by a processor, implement the steps performed by the above-mentioned first data processing device, or the steps performed by the second data processing device.
[0018] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:
[0019] In an embodiment of the present application, a data service system can be implemented by including a first data processing device, a second data processing device, a first memory, and a second memory, wherein the first memory and the second memory both include a first data set; the first data processing device can be used to obtain the second data set and send it to the first memory for storage, and the second data set includes an update result of the first data set; the second data processing device can be used to call a background thread to obtain the second data set from the first memory and update the second memory based on the second data set; in response to a query request for target data in the second data set, the main thread is called to obtain the target data from the second memory. In this way, with the help of the first memory and the second data processing device, the second data processing device can independently complete the update and query of the second memory, that is, the background thread can be used to store the second data set in the second memory, and the main thread can be used to query the required target data from the second memory. Therefore, it is possible to avoid mutual influence when different data processing devices perform data updates and queries, thereby improving query performance. Moreover, since the data update process of the second memory is completed by the second data processing device using a background thread, the data update process can be executed without affecting the main thread's task of executing data query, and the main thread can also perform data query services at any time without waiting for data update, which can further improve data query performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 A schematic diagram of the structure of a data service system provided in an embodiment of the present application;
[0021] Figure 2a A schematic diagram of the structure of another data service system provided in an embodiment of the present application;
[0022] Figure 2b A schematic diagram of the structure of another data service system provided in an embodiment of the present application;
[0023] Figure 3 A flowchart of a data service method provided in an embodiment of the present application;
[0024] Figure 4 A schematic diagram of the structure of a server provided in an embodiment of the present application;
[0025] Figure 5 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0026] As mentioned above, common data query methods can be implemented using databases such as MySQL or Key Value. Taking MySQL as an example, the data collector can first upload the queryable data to a MySQL table, and then the data query party can query the MySQL table to obtain the required data.
[0027] However, in the above solution, when the queryable data in the MySQL data table is updated in the data source, the data collector needs to update the data in the MySQL data table accordingly. However, the update process may affect the data query party's query of data from the MySQL data table, resulting in poor query performance.
[0028] Based on the above problems, an embodiment of the present application provides a data service system, which includes a first data processing device, a second data processing device, a first memory, and a second memory, wherein the first memory and the second memory both include a first data set. Correspondingly, the first data processing device can be used to obtain a second data set and send it to the first memory for storage, wherein the second data set includes an update result of the first data set; the second data processing device can be used to call a background thread to obtain the second data set from the first memory and update the second memory based on the second data set; and in response to a query request for target data in the second data set, the main thread is called to obtain the target data from the second memory.
[0029] In this way, with the help of the first memory and the second data processing device, the second data processing device can independently complete the update and query of the second memory, that is, the background thread can be used to store the second data set in the second memory, and the main thread can be used to query the required target data from the second memory. Therefore, it is possible to avoid the mutual influence of different data processing devices when performing data updates and queries, thereby improving query performance. In addition, since the data update process of the second memory is completed by the second data processing device using the background thread, the data update process can be executed without affecting the main thread's task of executing data query, and the main thread can also perform data query services at any time without waiting for data update, which can further improve data query performance.
[0030] It should be noted that the embodiments of this application do not limit the specific implementation of the first data processing device and the second data processing device in the data service system. For example, the first data processing device or the second data processing device may be implemented by a user terminal or a server, or a user terminal and a server may be implemented in collaboration. As an example, user terminals include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. The server may be a standalone server, a cluster server, or a cloud server.
[0031] To facilitate understanding, the following first introduces the terms that may be involved in the embodiments of the present application.
[0032] Object storage refers to object-based storage. Data is stored as independent objects and accessed through unique identifiers. Each object contains the data itself and associated metadata, such as the file name, creation date, and file size. However, unlike files, objects do not have a hierarchical structure within a layered structure. Each object is at the same level in a flat address space called a storage pool, and one object does not belong to the next level of another object. Compared to traditional file systems or block storage methods, object storage can better adapt to large-scale, distributed data storage needs.
[0033] Cloud Object Storage (COS) is a cloud computing technology used to store and manage large-scale object data in the cloud. Based on the distributed architecture and storage technologies of cloud service providers, COS aims to provide scalable, secure, and highly available storage solutions. COS utilizes the object storage data model described above.
[0034] A background thread is a thread that runs within a program or operating system. Its primary purpose is to perform auxiliary tasks in the background without blocking or interfering with the main thread. Compared to the main thread, background threads typically perform tasks that are lower priority, take longer, or don't require immediate results. Background threads can improve multitasking capabilities and enable the main thread to respond more quickly to user actions.
[0035] An atomic operation is an indivisible, uninterruptible operation in concurrent programming. Simply put, an atomic operation is either fully executed or not executed at all; partial execution is not permitted. Atomic operations are generally used to ensure data consistency and thread safety. In a multithreaded environment, multiple threads may access and modify shared data simultaneously. Failure to ensure atomicity can lead to data inconsistency or race conditions. The use of atomic operations can avoid data errors and concurrency issues caused by race conditions, thereby improving program performance and reliability.
[0036] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0037] Figure 1A structural diagram of a data service system provided in an embodiment of the present application. Figure 1 As shown, the data service system provided in the embodiment of the present application may include a first data processing device, a second data processing device, a first memory and a second memory, wherein the first memory and the second memory both include a first data set.
[0038] The first data processing device is configured to obtain a second data set and send the data to the first memory for storage, where the second data set includes an update result of the first data set.
[0039] The second data processing device is used to call the background thread to obtain the second data set from the first memory and update the second memory based on the second data set; in response to a query request for target data in the second data set, call the main thread to obtain target data from the second memory.
[0040] In the embodiments of the present application, the first memory can be implemented as an object storage, such as COS. Using COS as the first memory effectively leverages COS's advantages, ensuring the reliability and high availability of data stored in COS. Furthermore, COS can also collaborate with the second data processing device, allowing the second data processing device to independently update and query the second memory. This prevents cross-interference between data updates and queries performed by different data processing devices, thereby improving query performance.
[0041] Furthermore, in the embodiments of the present application, in order to enable the second data processing device to retrieve the second data set from the first memory via a background thread, the COS may provide an interface for data retrieval, allowing the background thread of the second data processing device to retrieve the second data set via the interface. In this way, the COS enables the second data processing device to flexibly and quickly retrieve the second data set, thereby helping to improve the query performance of the second data processing device.
[0042] As mentioned earlier, common data query methods can be implemented through databases. However, the inventors found that building a database requires additional machine resources, and the database expansion process is relatively cumbersome, resulting in increased costs. In addition, when performing data queries, the data query party needs to access the database before the query results can be returned, which consumes network communication resources and may be affected by the network, resulting in reduced query performance. Especially in the scenario of high-concurrency queries, the access pressure on the database will be further aggravated. In addition, when data needs to be inserted into the database in batches, a large number of write operations may be concentrated in specific areas or specific keys of the database, which may cause hot write problems, create system bottlenecks, and even affect the availability of query services.
[0043] Based on this, in an embodiment of the present application, the second memory can be implemented using the internal memory of the second data processing device, that is, the memory of the second data processing device is used as the second memory. In this way, the second data processing device can query data directly through the memory without going through the database, thereby effectively reducing the resources and operation and maintenance costs for the database, and without consuming network communication resources. In addition, writing data to the memory can also avoid the hot write problem caused by the database performing a large number of write operations, which helps to further improve the query performance and ensure the availability of the data query service.
[0044] In the embodiments of the present application, the method by which the first data processing device obtains the second dataset is not specifically limited. For example, the first data processing device may be configured to, when the first dataset is updated in the data source, extract the updated first dataset from the data source and preprocess the updated first dataset to obtain the second dataset. In this way, extracting the updated first dataset and preprocessing it to obtain the second dataset helps make the second dataset more suitable for the second data processing device, thereby improving the query performance of subsequent data query services.
[0045] In practical applications, the data source is, for example, a database, an application programming interface, a cloud service platform, or a data file. The preprocessing of the updated first data set may include at least one of data cleaning, data deduplication, data normalization, and data aggregation. Specifically, data cleaning can remove invalid data and / or incomplete data in the updated first data set, data deduplication can remove redundant data in the updated first data set, data normalization can unify the field format and type of the data included in the updated first data set, and data aggregation can merge or count data from different data sources to provide a more comprehensive data analysis perspective.
[0046] It should be noted that the embodiments of the present application may not specifically limit the specific implementation methods of data cleaning, data deduplication, data normalization or data aggregation. For example, any existing or future method that can perform data cleaning, data deduplication, data normalization or data aggregation can be used for implementation.
[0047] In addition, the data in the second data set can be further compressed to obtain a form that is easier to transmit and deploy, so as to be sent to the first memory for storage. In actual applications, if the second data set includes a large amount of data, the large amount of data can be split into multiple data files before compression.
[0048] Furthermore, the background thread of the second data processing device can periodically detect whether the dataset stored in the first memory has been updated. If so, it can directly obtain the updated dataset and update the second memory based on the updated dataset, thereby improving the accuracy and real-time performance of subsequent data queries. For ease of understanding, the following describes the process of the second data processing device obtaining the second dataset, using the first and second datasets as examples and in conjunction with a possible implementation.
[0049] As a possible implementation, the second data processing device can be specifically used to call a background thread to obtain the time when the first data set is stored in the second memory, and to periodically query the time when the second data set is stored in the first memory; when the time when the second data set is stored in the first memory is later than the time when the first data set is stored in the second memory, call the background thread to obtain the second data set from the first memory. Here, the time when the second data set is stored in the first memory is later than the time when the first data set is stored in the second memory, which can indicate that the second data set in the first memory includes the update result of the first data set, and therefore needs to be updated to the second memory. In this way, the background thread can determine whether the second memory needs to be updated by comparing the time when the first data set is stored in the second memory with the time when the second data set is stored in the first memory, thereby helping to improve the accuracy and real-time performance of the data in the second memory.
[0050] Furthermore, in the embodiments of the present application, the frequency of the second data processing device's periodic data query is not specifically limited. For example, the second data processing device may query the time the second data set was stored in the first memory every 1 second, 3 seconds, 5 seconds, or 10 seconds. In this way, the shorter periodic query method helps further improve the accuracy and real-time performance of the data in the second memory.
[0051] Furthermore, after the background thread of the second data processing device obtains the second data set, it can use the second data set to update the second memory, thereby improving the accuracy and real-time performance of subsequent data queries from the second memory. For ease of understanding, the following describes the process of the second data processing device updating the second memory, still using the first and second data sets as examples, in conjunction with a possible implementation.
[0052] As a possible implementation, the second data processing device can be specifically configured to call a background thread to send the second data set to a second memory for storage, and to update the pointer in the second memory pointing to the first data set to point to the second data set using an atomic operation. In this way, the second memory can store both the first and second data sets. By retaining two copies of the data sets, the background thread can directly update the pointer in the second memory associated with the first data set, shifting it from the first data set to the second data set, thereby achieving smooth switching of data sets and ensuring that the data update process does not affect the data query service, thereby improving query performance. Furthermore, updating the pointer's pointing using an atomic operation helps improve data consistency and integrity through the use of the atomic operation mechanism, avoiding data inconsistencies when the pointer's pointing is switched. Additionally, after the second data set is stored in the second memory and before the pointer's pointing is updated, the first data set can serve as the data currently used by the data query service, while the second data set can serve as a backup of the first data set. By storing two copies of the data set in the second memory, data loss or corruption can be avoided during data updates, thereby improving the availability and fault tolerance of the data in the second memory.
[0053] In order to facilitate understanding of the updating process of the pointer, an exemplary description is given below with reference to the accompanying drawings, wherein the first memory takes COS as an example, and the second memory takes the memory of the second data processing device as an example.
[0054] Figure 2a A schematic diagram of another data service system provided in an embodiment of the present application is provided. Figure 2b This is a structural diagram of another data service system provided in the embodiment of the present application. As a possible implementation method, combined with Figure 2a As shown, in the data service system, before the second memory is updated, the pointer 1 in the second memory points to the first data set. Figure 2b As shown, in the data service system, after the second data is stored in the second memory, the pointer 1 in the second memory may be updated to point to the second data set.
[0055] Furthermore, in an embodiment of the present application, after the second data processing device obtains the second data set through a background thread, it can also process the second data set before updating the second memory to better adapt to subsequent data queries. Specifically, the second data processing device can also be used to call the background thread to parse the second data set to obtain a parsing result, and convert the format of the parsing result to a format that matches the second data processing device to obtain a processed second data set. In this way, by parsing and formatting the second data set, a data storage format suitable for the second data processing device can be obtained, facilitating subsequent updates to the second memory and data queries therefrom.
[0056] Correspondingly, with respect to the implementation process of updating the second memory described above, the second data processing device can be specifically configured to update the second memory based on the processed second data set. That is, the second data processing device can be specifically configured to call a background thread to send the processed second data set to the second memory for storage, and to update the pointer in the second memory pointing to the first data set to point to the processed second data set using an atomic operation. For the implementation of the second data processing device updating the second memory based on the processed second data set, reference can be made to the implementation of the second data processing device updating the second memory based on the second data set in the above-described embodiment, and will not be further elaborated here.
[0057] In actual applications, once the pointer to the first dataset is updated to point to the second dataset, that is, after the second memory is updated, the background thread of the second data processing device can further send a notification message to the main thread indicating the update completion. This allows the main thread to use the latest dataset from the second memory for data query services, thereby ensuring the accuracy and real-time nature of data queries.
[0058] In addition, in the embodiment of the present application, the second data processing device calls the main thread to implement the data query service, and the query can be performed with the help of the data index to quickly locate the relevant data. For ease of understanding, the following is an explanation in conjunction with a possible implementation method.
[0059] In one possible implementation, the first data processing device may also be configured to create index information corresponding to each of the multiple query data items included in the second data set, and transmit the index information to the second data processing device. The multiple query data items may include target data. In practical applications, the index corresponding to each of the multiple query data items in the second data set may be embodied as at least one of text content, keywords, and tags corresponding to each of the multiple query data items.
[0060] Correspondingly, the second data processing device can be specifically used to call the main thread to query the index information of the target data from the index information corresponding to the multiple data to be queried, and obtain the target data from the second data set stored in the second memory based on the index information of the target data.
[0061] In this way, by creating index information corresponding to multiple pieces of data to be queried, and performing data query with the help of the index information, the required data can be quickly located when the data in the second data set needs to be queried from the second memory, thereby improving query performance.
[0062] Furthermore, in practical applications, the query request may include query conditions corresponding to the target data, such as at least one of text content, tags, and keyword information corresponding to the target data. Thus, when a user enters query conditions in a query interface via the client, a corresponding query request is generated and sent to the second data processing device. The second data processing device then searches the index information corresponding to the plurality of data to be queried based on the query conditions, thereby obtaining index information for the target data and further returning the target data in the query interface.
[0063] Based on the relevant content of the above-mentioned data service system, it can be seen that in an embodiment of the present application, it can be implemented by a data service system including a first data processing device, a second data processing device, a first memory, and a second memory, wherein the first memory and the second memory both include a first data set; the first data processing device can be used to obtain the second data set and send it to the first memory for storage, and the second data set includes the update result of the first data set; the second data processing device can be used to call a background thread to obtain the second data set from the first memory and update the second memory based on the second data set; in response to a query request for target data in the second data set, the main thread is called to obtain the target data from the second memory. In this way, with the help of the first memory and the second data processing device, the second data processing device can independently complete the update and query of the second memory, that is, the background thread can be used to store the second data set in the second memory, and the main thread can be used to query the required target data from the second memory. Therefore, it is possible to avoid mutual influence when different data processing devices perform data updates and queries, thereby improving query performance. Moreover, since the data update process of the second memory is completed by the second data processing device using a background thread, the data update process can be executed without affecting the main thread's task of executing data query, and the main thread can also perform data query services at any time without waiting for data update, which can further improve data query performance.
[0064] Furthermore, in embodiments of the present application, the data service system can be applied to offline data update and query scenarios. Offline data refers to data from a specific day in the past, such as historical business data. In this context, in offline data scenarios, both the first and second memories can include offline datasets. Upon updating the offline dataset at the data source, the first data processing device can extract the updated offline dataset from the data source and preprocess the updated offline dataset to obtain a preprocessed offline dataset. The first data processing device can then send the preprocessed offline dataset to the first memory for storage. Simultaneously, the first data processing device can also create index information for each of the multiple offline data items included in the preprocessed offline dataset and send this index information to the second data processing device. The second data processing device can then, through the data acquisition interface provided by the first memory, invoke a background thread to retrieve the preprocessed offline dataset from the first memory and send the preprocessed offline dataset to the second memory for storage. The second data processing device can then continue to invoke the background thread and, through atomic operations, update the pointer to the offline dataset in the second memory to point to the preprocessed offline dataset. In this way, the offline data is updated in the second memory. To subsequently query the offline data, the user can send a query request to the second data processing device through the query interface. This query request can include the user-entered query criteria for a specific offline data item. Correspondingly, the second data processing device can invoke the main thread, retrieve the index information of the offline data item from the respective index information of the offline data item based on the query criteria, and retrieve the offline data item from the preprocessed offline dataset stored in the second memory based on the index information of the offline data item. Finally, the offline data item can be returned for the user to view in the query interface.
[0065] In this way, the second data processing device can update the offline data set in the second memory and query the offline data using the second memory, thereby avoiding the mutual influence between data updates and queries performed by different data processing devices, thereby improving query performance. Furthermore, because the update process of the offline data in the second memory is completed by the second data processing device using a background thread, the offline data update can be performed without affecting the main thread's execution of the data query task. The main thread can also perform data query services at any time without waiting for data updates, which can further improve data query performance.
[0066] Based on the data service system provided in the above embodiment, the present application embodiment can also provide a data service method. The data service method is described below in conjunction with the embodiments and drawings.
[0067] Figure 3 A flowchart of a data service method provided in an embodiment of the present application. Figure 3 As shown, the data service method provided in the embodiment of the present application can be applied to a data service system. The data service system includes a first data processing device, a second data processing device, a first memory, and a second memory, wherein the first memory and the second memory both include a first data set. Based on this, the embodiment of the present application uses the interaction between the first data processing device and the second data processing device as the execution body to illustrate the steps of the data service method. The data service method may include:
[0068] S301: The first data processing device obtains a second data set and sends it to the first memory for storage.
[0069] The second data set includes an updated result of the first data set.
[0070] S302: The second data processing device calls a background thread to obtain the second data set from the first memory, and updates the second memory based on the second data set.
[0071] S303: In response to a query request for target data in the second data set, the second data processing device calls a main thread to obtain the target data from the second memory.
[0072] Optionally, the second data processing device updates the second memory based on the second data set, including:
[0073] The second data processing device calls the background thread to send the second data set to the second memory for storage, and uses an atomic operation to update the pointer in the second memory pointing to the first data set to point to the second data set.
[0074] Optionally, the first memory is an object memory, and the second memory is an internal memory of the second data processing device.
[0075] Optionally, the data service method further includes:
[0076] The first data processing device creates index information corresponding to a plurality of to-be-queried data included in the second data set, and sends the index information corresponding to the plurality of to-be-queried data to the second data processing device, wherein the plurality of to-be-queried data includes the target data;
[0077] The second data processing device calls a main thread to obtain the target data from the second memory, including:
[0078] In response to the query request, the second data processing device calls the main thread to query the index information of the target data from the index information corresponding to the multiple data to be queried, and obtains the target data from the second data set stored in the second memory based on the index information of the target data.
[0079] Optionally, the first data processing device acquires the second data set, including:
[0080] When the first data set is updated in the data source, the first data processing device extracts the updated first data set from the data source and preprocesses the updated first data set to obtain the second data set, where the preprocessing includes at least one of data cleaning, data deduplication, data normalization, and data aggregation.
[0081] Optionally, the second data processing device calls a background thread to obtain the second data set from the first memory, including:
[0082] The second data processing device calls the background thread to obtain the time when the first data set is stored in the second memory, and periodically queries the time when the second data set is stored in the first memory;
[0083] When the time when the second data set is stored in the first memory is later than the time when the first data set is stored in the second memory, the second data processing device calls the background thread to obtain the second data set from the first memory.
[0084] Optionally, the data service method further includes:
[0085] The second data processing device calls the background thread to parse the second data set to obtain a parsing result, and converts the format of the parsing result into a format matching the second data processing device to obtain a processed second data set;
[0086] The second data processing device updates the second memory based on the second data set, comprising:
[0087] The second data processing device is specifically configured to update the second memory based on the processed second data set.
[0088] The structures of the control devices implementing the above data service method are introduced below in terms of server form and terminal device form respectively.
[0089] Figure 4A schematic diagram of a server structure provided for an embodiment of the present application, wherein the server 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 922 (e.g., one or more processors) and a memory 932, and one or more storage media 930 (e.g., one or more mass storage devices) storing application programs 942 or data 944. Among them, the memory 932 and the storage medium 930 may be temporary storage or permanent storage. The program stored in the storage medium 930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 922 may be configured to communicate with the storage medium 930 to execute a series of instruction operations in the storage medium 930 on the server 900.
[0090] The server 900 may also include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input and output interfaces 958, and / or one or more operating systems 941, such as Windows Server 2003 or Windows Server 2003R. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.
[0091] The steps performed by the server in the above embodiment can be based on the Figure 4 The server structure shown.
[0092] The CPU 922 is configured to execute the steps executed by the first data processing device and / or the second data processing device in the aforementioned embodiment, wherein the first data processing device is configured to execute the following steps:
[0093] The first data processing device acquires a second data set and sends the data to the first memory for storage, wherein the second data set includes an update result of the first data set;
[0094] The second data processing device is configured to perform the following steps:
[0095] The second data processing device calls a background thread to obtain the second data set from the first memory, and updates the second memory based on the second data set;
[0096] In response to a query request for target data in the second data set, the second data processing device calls a main thread to obtain the target data from the second memory.
[0097] The present application also provides another control device, such as Figure 5 For ease of explanation, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), a sales terminal (English full name: Point of Sales, English abbreviation: POS), a car computer, etc., taking the mobile phone as an example:
[0098] Figure 5 The block diagram shows a partial structure of a mobile phone related to the terminal provided in the embodiment of the present application. Figure 5 The mobile phone includes components such as a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art will appreciate that Figure 5 The mobile phone structure shown in the figure does not constitute a limitation to the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0099] The following combination Figure 5 A detailed introduction to the various components of a mobile phone:
[0100] RF circuitry 1010 can be used to receive and transmit signals during information transmission or calls. Specifically, it receives downlink information from the base station and transmits it to processor 1080 for processing. It also transmits uplink data to the base station. Typically, RF circuitry 1010 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, and more. RF circuitry 1010 can also communicate with the network and other devices via wireless communications. The above-mentioned wireless communications may use any communication standard or protocol, including but not limited to Global System of Mobile Communications (Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (English full name: General Packet Radio Service, GPRS), Code Division Multiple Access (English full name: Code Division Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (English full name: Wideband Code Division Multiple Access, English abbreviation: WCDMA), Long Term Evolution (English full name: Long Term Evolution, English abbreviation: LTE), email, Short Messaging Service (English full name: Short Messaging Service, SMS), etc.
[0101] The memory 1020 can be used to store software programs and modules. The processor 1080 executes the various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 may mainly include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory 1020 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state memory device.
[0102] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by the user using any suitable object or accessory such as a finger, stylus, etc. on or near the touch panel 1031) and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch direction and detects the signal caused by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor 1080. It can also receive commands sent by the processor 1080 and execute them. In addition, the touch panel 1031 can be implemented using various types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick.
[0103] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (English full name: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (English full name: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 5 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.
[0104] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that the mobile phone can also be configured with, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described here.
[0105] Audio circuit 1060, speaker 1061, and microphone 1062 provide an audio interface between the user and the phone. Audio circuit 1060 converts received audio data into electrical signals and transmits them to speaker 1061, which then converts them into sound signals for output. Microphone 1062, on the other hand, converts collected sound signals into electrical signals, which are then received by audio circuit 1060 and converted into audio data. The audio data is then processed by processor 1080 and transmitted to, for example, another phone via RF circuit 1010, or stored in memory 1020 for further processing.
[0106] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse the web and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access. Figure 5 A WiFi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.
[0107] Processor 1080 is the control center of the phone, connecting all parts of the phone using various interfaces and circuits. By running or executing software programs and / or modules stored in memory 1020 and accessing data stored in memory 1020, it executes various phone functions and processes data, thereby collecting data and information about the phone as a whole. Optionally, processor 1080 may include one or more processing units; preferably, processor 1080 may integrate an application processor and a modem processor, where the application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 1080.
[0108] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, thereby managing charging, discharging, and power consumption through the power management system.
[0109] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.
[0110] In the embodiment of the present application, the processor 1080 included in the terminal further has the functions of the first data processing device and / or the second data processing device in the aforementioned embodiment, wherein the first data processing device has the following functions:
[0111] The first data processing device is configured to obtain a second data set and send the second data set to the first memory for storage, wherein the second data set includes an update result of the first data set;
[0112] The second data processing device has the following functions:
[0113] The second data processing device is used to call a background thread to obtain the second data set from the first memory and update the second memory based on the second data set; in response to a query request for target data in the second data set, call the main thread to obtain the target data from the second memory.
[0114] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.
[0115] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0116] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0117] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.
[0118] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.
[0119] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English full name: Read-Only Memory, English abbreviation: ROM), random access memory (English full name: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program code.
[0120] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A data service system, characterized in that: include: a first data processing device, a second data processing device, a first memory and a second memory, wherein the first memory and the second memory each include a first data set; The first data processing device is configured to obtain a second data set and send the second data set to the first memory for storage, wherein the second data set includes an update result of the first data set; The second data processing device is configured to call a background thread to obtain the second data set from the first memory, and update the second memory based on the second data set; In response to a query request for target data in the second data set, the main thread is called to obtain the target data from the second memory.
2. The data service system according to claim 1, characterized in that: The second data processing device is specifically configured to call the background thread to send the second data set to a second memory for storage, and update the pointer in the second memory pointing to the first data set to point to the second data set using an atomic operation.
3. The data service system according to claim 1, wherein: The first memory is an object memory, and the second memory is an internal memory of the second data processing device.
4. The data service system according to any one of claims 1 to 3, characterized in that: The first data processing device is further configured to create index information corresponding to a plurality of to-be-queried data items included in the second data set, and send the index information corresponding to the plurality of to-be-queried data items to the second data processing device, wherein the plurality of to-be-queried data items include the target data; The second data processing device is specifically used to call the main thread to query the index information of the target data from the index information corresponding to the multiple data to be queried, and obtain the target data from the second data set stored in the second memory based on the index information of the target data.
5. The data service system according to any one of claims 1 to 3, characterized in that: The first data processing device is specifically used to extract the updated first data set from the data source when the first data set is updated in the data source, and preprocess the updated first data set to obtain the second data set, wherein the preprocessing includes at least one of data cleaning, data deduplication, data normalization and data aggregation.
6. The data service system according to any one of claims 1 to 3, characterized in that: The second data processing device is specifically configured to call the background thread to obtain the time when the first data set is stored in the second memory, and to periodically query the time when the second data set is stored in the first memory; When the time when the second data set is stored in the first memory is later than the time when the first data set is stored in the second memory, the background thread is called to obtain the second data set from the first memory.
7. The data service system according to any one of claims 1 to 3, characterized in that: The second data processing device is further configured to call the background thread to parse the second data set to obtain a parsing result, and convert the format of the parsing result into a format matching the second data processing device to obtain a processed second data set; The second data processing device is specifically configured to update the second memory based on the processed second data set.
8. A data service method, characterized in that: Applied to a data service system, the data service system includes a first data processing device, a second data processing device, a first memory and a second memory, the first memory and the second memory both including a first data set, the data service method including: The first data processing device acquires a second data set and sends the data to the first memory for storage, wherein the second data set includes an update result of the first data set; The second data processing device calls a background thread to obtain the second data set from the first memory, and updates the second memory based on the second data set; In response to a query request for target data in the second data set, the second data processing device calls a main thread to obtain the target data from the second memory.
9. A data service device, characterized in that: The device includes a processor and a memory: The memory is used to store a computer program and transmit the computer program to the processor; The processor is configured to execute the steps performed by the first data processing device or the second data processing device according to any one of claims 1 to 7 according to instructions in the computer program.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a computer program, which, when executed by a terminal device, implements the steps performed by the first data processing device or the second data processing device according to any one of claims 1 to 7.
11. A computer program product, characterized in that The computer program comprises a computer program which, when executed by a terminal device, implements the steps performed by the first data processing device or the steps performed by the second data processing device according to any one of claims 1 to 7.