List data processing method and system, electronic equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and provides a list data processing method and related equipment. The list data processing method comprises the steps of optimizing a storage structure of list data based on an optimization target of reducing the resource usage amount of a database and reducing the storage magnitude of the database; based on the optimized storage structure, an incremental data synchronization mechanism is introduced; the new list identifier is transparently transmitted in a full-link manner, so that the list in the list is matched with the list version of the application layer, and a shunting experiment is accessed to the business logic layer and the list in the list; and through real-time data acquisition and comparison, the new list data and the old list data are kept consistent. By optimizing the storage structure of the list data, improving the list making and application process and perfecting the data comparison and verification mechanism, the resource usage amount of the database is reduced, the storage magnitude of the database is reduced, the total data synchronization time is shortened, and the real-time performance and accuracy of service pushing list making are improved; and the overall performance and user experience of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a list data processing method, system, electronic device and storage medium. Background Art

[0002] In the field of list data processing, with the continuous development of business and the continuous growth of data volume, the original list data processing method faces many severe challenges.

[0003] First, the massive amount of list data placed immense pressure on the database's CPU (central processing unit), with CPU utilization peaking at 80%, severely impacting the system's overall performance and responsiveness. Second, due to the large number of changes to data tables, I / O (input and output) also faced bottlenecks, resulting in low data read and write efficiency. Furthermore, there were issues such as lengthy full data synchronization times and data backlogs.

[0004] These problems result in the existing list data processing methods being unable to meet the business needs for fast processing and real-time display of list data.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0006] In view of this, the present invention provides a list data processing method, system, electronic device and storage medium. By optimizing the storage structure of the list data, improving the list making and application process, and perfecting the data comparison and verification mechanism, it solves the problems existing in the existing list data processing methods, such as excessive database pressure, excessive storage level, slow data synchronization, and the inability to push business rankings online in real time.

[0007] According to one aspect of the present invention, a list data processing method is provided, comprising the following steps: optimizing the storage structure of list data based on the optimization goal of reducing database resource usage and lowering the storage level of the database; synchronizing incremental data based on the optimized storage structure; transparently transmitting new list identifiers throughout the entire link to match the list versions of the list middle platform with the application layer, and accessing diversion experiments at the business logic layer and the list middle platform; and ensuring consistency between new list data and old list data through real-time data collection and comparison.

[0008] In some embodiments, the storage structure of the list data is optimized, including: merging and deduplicating common business data of different lists; merging and deduplicating related data of different language environments of the same list; independently storing different language content of the same list; decoupling list dimension data from listed item dimension data; and storing the list dimension data and the listed item dimension data in a unified format.

[0009] In some embodiments, the list dimension data and the listed item dimension data are decoupled, including: converting the association relationship between the list dimension data and the listed item dimension data from associating the listed item dimension data with the list dimension data through an association ID to associating the list dimension data and the listed item dimension data through a list ID.

[0010] In some embodiments, the list dimension data and the listed item dimension data are stored in a unified format, including: storing the list dimension data and the listed item dimension data in a unified storage format using the list ID and the corresponding language environment.

[0011] In some embodiments, a diversion experiment is accessed at the business logic layer and the list middle platform, including: the list middle platform dynamically selects the data interface of the new list version and the old list version according to the identification parameters of the user request.

[0012] In some embodiments, the new list data is kept consistent with the old list data through the collection and comparison of real-time data, including: collecting real-time data from the production environment and storing it persistently; after processing the real-time data, comparing and verifying it with the real-time requests of the production environment to keep the new list data consistent with the old list data.

[0013] In some embodiments, the list data processing method further includes: consuming new list data and old list data in parallel during the migration process, and achieving a smooth transition of data versions through configurable rules.

[0014] According to another aspect of the present invention, a list data processing system is provided for implementing the list data processing method as described in any of the above embodiments, and the list data processing system includes the following modules: a storage structure optimization module, which is used to optimize the storage structure of the list data based on the optimization goal of reducing the resource usage of the database and lowering the storage level of the database; a data synchronization module, which is used to synchronize incremental data based on the optimized storage structure; a process management module, which is used to fully transmit the new list identifier to match the list version of the list middle platform with the application layer, and access the diversion experiment at the business logic layer and the list middle platform; a data verification module, which is used to keep the new list data consistent with the old list data through real-time data collection and comparison.

[0015] According to another aspect of the present invention, there is provided an electronic device, comprising: a processor; a memory, wherein the memory stores executable instructions; wherein when the executable instructions are executed by the processor, the list data processing method as described in any of the above embodiments is implemented.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided for storing a program, wherein when the program is executed by a processor, the method for processing list data as described in any of the above embodiments is implemented.

[0017] The beneficial effects of the present invention compared with the prior art include at least:

[0018] The list data processing solution provided by the present invention reduces database resource usage, reduces the database storage level, shortens the full data synchronization time, and improves the real-time and accuracy of business push list making by optimizing the storage structure of list data, improving the list making and application processes, and perfecting the data comparison and verification mechanism, thereby improving the overall performance and user experience of the system, meeting the growing business needs, and solving the problems of existing list data processing methods such as excessive database pressure, excessive storage level, slow data synchronization, and the inability of business push list making to be online in real time.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present invention, and together with the description, serve to explain the principles of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and it is clear that those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0021] Figure 1 A schematic diagram showing the steps of a method for processing list data according to an embodiment of the present invention;

[0022] Figure 2 A schematic diagram showing the existing storage structure of list dimension data;

[0023] Figure 3 A schematic diagram showing the existing storage structure of the dimension data of the listed items;

[0024] Figure 4 A schematic diagram showing the storage structure of list data in an embodiment of the present invention is shown;

[0025] Figure 5 A schematic diagram illustrating application layer transformation according to an embodiment of the present invention is shown;

[0026] Figure 6 A schematic diagram illustrating real-time data collection and comparison in an embodiment of the present invention is shown;

[0027] Figure 7 A schematic diagram showing the modules of a list data processing system according to an embodiment of the present invention is shown;

[0028] Figure 8 A schematic structural diagram of an electronic device in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in many forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to make this disclosure more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art.

[0030] The accompanying drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0031] In addition, the processes shown in the drawings are merely exemplary and do not necessarily include all steps. For example, some steps may be decomposed, some steps may be combined or partially combined, and the actual execution order may change according to actual circumstances.

[0032] It should be noted that, in the absence of conflict, the embodiments of the present invention and features in different embodiments may be combined with each other.

[0033] Figure 1 The main steps of the list data processing method are shown in Figure 2. Figure 1 As shown, the list data processing method provided by the embodiment of the present invention mainly includes steps S110 to S140.

[0034] S110 , based on the optimization goal of reducing database resource usage and lowering the database storage level, optimize the storage structure of the list data.

[0035] Database resource usage primarily includes CPU usage and I / O usage. Reducing the database's storage capacity means reducing the physical storage space occupied by the database while maintaining unimpeded business functions.

[0036] In some embodiments, the storage structure of the list data is optimized, including: merging and deduplicating common business data of different lists; merging and deduplicating related data of different language environments of the same list; and independently storing different language content of the same list.

[0037] Merging and deduplicating public business data from different lists enables the reuse of public data. Specifically, for multiple independent lists (such as parent-child lists, word-of-mouth lists, and so on), only one copy of the public business data (such as the list ID, creation time, basic rules, etc.) for each list in different languages or regions (locales, such as Chinese, English, and Japanese) is stored. This data is dynamically adapted to multilingual display using locale identifiers, rather than being stored repeatedly for each locale. This reduces redundant data storage, lowers database storage capacity, and improves read and write efficiency. For example, a word-of-mouth list's public business data (basic information, independent of language or region, such as the list name, effective time, sorting rules, etc.) is the same in both Chinese (zh-CN) and English (en-US) environments. Only one copy needs to be stored, and the display language is distinguished by the locale field. Language-related descriptive content (such as recommendation reasons and list descriptions) can be stored separately to meet multilingual display requirements.

[0038] Merging and deduplicating the associated data of different language environments of the same list can achieve the decoupling of associated data. Specifically, under the same list, the associated data of different language versions (such as the ranking, weight, and association rules of the listed items) are independent of the language, and only one public version is stored, which can decouple the language dependency and reduce duplicate data. In the original system, different language versions of the same list need to store the data associated with the listed items (such as ranking, weight), resulting in data redundancy, increasing the storage level and synchronization pressure. The optimization scheme of the present invention decouples language and associated data, and designs the associated data of the list and the listed items as language-independent public data, which is only stored in one copy; while the language content is stored independently, that is, the language-related descriptive content can be stored separately by locale and associated with the public associated data through a unique identifier (such as rankingId). In this way, the repeated storage of the association relationship between the list and the listed items can be avoided, the storage level can be significantly reduced, the database read and write pressure can be reduced, the CPU utilization rate can be reduced, and during incremental synchronization, only the public associated data changes need to be processed, and there is no need to synchronize separately for each locale, thereby improving synchronization efficiency.

[0039] Separately storing content in different languages for the same list enables language independence. Specifically, multilingual content for items on the same list can be stored separately as needed. Language-related descriptive content (such as recommendation reasons, item names, and list descriptions) is stored separately by locale identifier and linked to public data through a unique identifier (such as rankingId), enabling flexible multilingual support.

[0040] In some embodiments, optimizing the storage structure of the list data further includes: decoupling the list dimension data from the listed item dimension data. The list data includes the list dimension data and the listed item dimension data, and decoupling the list dimension data from the listed item dimension data specifically includes: converting the association relationship between the list dimension data and the listed item dimension data from associating the listed item dimension data with the list dimension data via the association ID (rankingDetailId) to associating the list dimension data with the listed item dimension data via the list ID (rankingId).

[0041] By converting the original rankingDetailId association into a rankingId association, the list dimension data is completely decoupled from the listed item dimension data. In the original system, the specific information of each listed item in the list (such as ranking, weight, multilingual description, etc.) is associated with the list through rankingDetailId. For example, in lists of different language versions (such as Chinese and English) of the same listed item, multiple rankingDetailIds are required to record the rankings and descriptions in different languages, resulting in data redundancy, and the list dimension data (such as list rules, effective time) is tightly coupled with the listed item dimension data (such as the ranking and description of each listed item), resulting in low storage and query efficiency. The present invention introduces rankingId, upgrades the universal identifier of the list to rankingId, and uses it as the unique identifier of the list to associate the public data of the list dimension (such as list name, rules, effective time, etc.). In this invention, list dimension data (such as rules and status) is stored solely via rankingId, ensuring independence from the specific ranking or description of the item. The ranking, multilingual description, and other data associated with the list are linked to the rankingId, ensuring independence from the item dimension data and eliminating reliance on rankingDetailId. Furthermore, in this invention, multilingual descriptions are stored separately as needed, associating rankingId with locale (language identifier) to achieve language-specific separation, rather than being tied to a specific rankingDetailId.

[0042] By decoupling list dimension data from item dimension data, the following benefits can be achieved. Storage optimization: Redundant rankingDetailId records are eliminated, and common data (such as list rules) is stored only once, significantly reducing storage requirements. Performance improvement: Database queries are simplified by directly linking lists and items via rankingId, reducing complex relational queries and lowering CPU utilization. Flexibility and scalability: List rules and item data can be modified independently without affecting each other. When adding a new language version, only the language description table needs to be expanded, without modifying the core list structure. By replacing rankingId with rankingDetailId, the list dimension data and item dimension data are loosely coupled instead of tightly coupled, achieving the de-duplication goals of reducing duplicate storage and database load, simplifying data association logic and improving read and write performance for efficient queries, and supporting rapid iteration of multiple languages and multiple lists, enabling flexible expansion to meet dynamic business needs. This reconstruction fundamentally resolves the performance bottlenecks and storage redundancy issues of the original system, ensuring the efficient operation of the list system.

[0043] Figure 2 It shows the current storage structure of the list dimension data. Figure 3 This diagram illustrates the current storage structure of the ranked item dimension data. It can be seen that the same rankingId is associated with multiple rankingDetailIds, and the list dimension data and the ranked item dimension data are severely coupled. Figure 4 The storage structure of the list data of the present invention is illustrated, which effectively decouples the list dimension data and the listed item dimension data.

[0044] Furthermore, in some embodiments, the storage structure of the list data is optimized, and the list dimension data and the listed item dimension data are stored in a unified format, which may specifically include: storing the list dimension data and the listed item dimension data in a unified storage format using the list ID and the corresponding language environment.

[0045] In the original solution, the ranking, recommendation reasons and other information of the same listed item in different language environments (such as Chinese and English) need to be stored as independent records, resulting in data redundancy. For example, if a listed item supports 10 languages, 10 duplicate data need to be stored (only the language tags are different). A large amount of duplicate data causes a surge in the storage level of the database, and the synchronization and query efficiency are low. After optimization of the present invention, all language configurations of the same listed item are merged into a single piece of data, and the ranking of the listed item in the list and multiple languages are converted from the original N to 1 piece of data, that is, compressed into a single piece of structured data, and the language range can be marked by a separator. Specific examples of data formatting processing performed by the present invention are as follows: 24100861|all; 24100777|zh-CN,zh-HK,zh-TW; 54344060|all. Among them, 24100861 can represent the ID of the item on the list, all indicates that the ranking / recommendation reason for this ID applies to all locales, and zh-CN, zh-HK, and zh-TW indicate that it applies only to Simplified Chinese and Traditional Chinese (Hong Kong / Taiwan). Furthermore, unified format storage can use the following separation rules: separate different items with a semicolon, separate the item ID and language tag with a vertical bar, and separate multiple language tags with a comma.

[0046] Using a unified storage format can achieve the following benefits: Storage requirements are reduced, significantly reducing database storage by eliminating redundant data; query efficiency is improved, as a single piece of structured data can read all language configurations with a single I / O operation, reducing database access pressure; business expansion is flexible, as adding a new language environment only requires modifying the language tag set, without adding new data records; and synchronization time is shortened, significantly reducing the time required to fully synchronize data to Elasticsearch / Redis due to the reduced data volume. This data format reconstruction and compression resolves storage redundancy and performance bottlenecks in multi-language ranking scenarios, reducing database pressure and improving synchronization efficiency.

[0047] S120: Synchronize incremental data based on the optimized storage structure.

[0048] In the original solution, the time for synchronizing the full amount of data to Elasticsearch and redis was too long, which seriously affected the real-time performance and availability of the data. For example, the synchronization time for the full amount of data of the Elasticsearch list was as long as 60 minutes, and the synchronization time for the full amount of data of the listed items was as long as 83 minutes. There are also problems with caching. The cache time for the full list is 9 minutes, the cache time for the full ranking is 20 minutes, and the cache time for the full amount of recommendation reasons is 67 minutes. The present invention fully compares the old and new versions of the list and the listed item data, introduces an incremental data synchronization mechanism when the amount of data is reduced, significantly shortens the time for data synchronization to Elasticsearch and Redis, improves the real-time performance and availability of the data, meets the business needs for fast data processing, and realizes efficient data synchronization. In addition, the present invention improves the cache mechanism, optimizes the cache strategy of the full list, ranking and recommendation reasons, reduces the cache time, improves the real-time display capability of data, and enhances the user experience.

[0049] S130: Transparently transmit the new list identifier throughout the entire chain, ensuring that the list version at the middle platform matches the list version at the application layer. A traffic diversion experiment is implemented at both the business logic layer and the middle platform. This involves the middle platform dynamically selecting the data interface for the new and old list versions based on the identifier parameters requested by the user.

[0050] Figure 5 For an example of application layer transformation, refer to Figure 5 As shown, through application layer transformation, it is possible to achieve full-link transparent transmission of the new version identifier, access of the BFF layer (business logic layer) to the AB diversion experiment, access of the list middle platform to the AB diversion experiment, transparent transmission of the new version identifier by the list middle platform, and unified data version switching of the search application.

[0051] S140: Ensure the new list data is consistent with the old list data by collecting and comparing real-time data. This may include: collecting real-time data from the production environment and storing it persistently; processing the real-time data and then comparing and verifying it with real-time requests from the production environment to ensure consistency between the new list data and the old list data.

[0052] Figure 6 This shows the real-time data collection and comparison process, refer to Figure 6 As shown, the data verification and comparison mechanism includes the following processes.

[0053] 1) Online traffic collection: 1a) Log table (ClickHouse Clog): used to store online real-time traffic data; 1b) Traffic storage: used to persistently store traffic data to ensure data integrity and traceability.

[0054] 2) Data processing and conversion: 2a) Configuration-based quantification: Quantify the collected online real-time traffic according to the preset configuration; 2b) Filtering, deduplication, and replacement: Preprocess the traffic data, including filtering invalid data, removing duplicate data, and replacing necessary data.

[0055] 3) Data comparison: 3a) Comparison module: used to compare the processed traffic data, and the comparison rules are based on the configuration; 3b) Obtaining traffic: extracting the traffic data to be compared from the traffic storage.

[0056] 4) Traffic Replay and Comparison: 4a) Forwarded Traffic: Forwards processed traffic data to the comparison system; 4b) Real-time Requests: Compares with real-time requests in the production environment; 4c) Mirror Environment: Used for traffic mirroring to ensure comparison accuracy; 4d) Production: Actual request data in the production environment is used for comparison with the mirrored traffic.

[0057] Figure 6 The data validation and comparison mechanism outlined here ensures data accuracy and consistency by collecting, processing, storing, and comparing traffic data. Configurable comparison rules allow the system to flexibly adapt to different business needs, ensuring consistency and reliability between new and old versions of the rankings in different environments.

[0058] Furthermore, in some embodiments, the list data processing method further includes: consuming new list data and old list data in parallel during the migration process, and achieving a smooth transition of data versions through configurable rules.

[0059] The consumption process of the QMQ message for ranking can specifically include: a) the old consumer group executes the old ranking process → DB table update → publish the ranking → DRC message → synchronize the old ES index; b) the new consumer group executes the new ranking process → new storage structure update → publish the ranking → synchronize the ranking QMQ → synchronize the new ES index.

[0060] In summary, the list data processing method of the present invention has the following advantages:

[0061] Database optimization: By redesigning the storage structure of the list data, we reduce the database's CPU and IO usage, lower the database's storage capacity, and thus alleviate database performance bottlenecks.

[0062] Efficient data synchronization: When the data volume decreases, an incremental data synchronization mechanism is introduced to significantly shorten the time it takes to synchronize data to Elasticsearch and Redis, improving data real-time performance and availability, and meeting business needs for fast data processing.

[0063] Improved caching mechanism: Optimized the caching strategy for the full list, rankings, and recommendation reasons to reduce caching time, improve the real-time display of data, and enhance the user experience;

[0064] Process improvement: Improve the ranking and application process to ensure that the ranking can be put online in real time;

[0065] Data verification and comparison: Through a comprehensive data comparison and verification mechanism, we ensure the accuracy and consistency of data during synchronization and display, and avoid data errors and duplications.

[0066] The comparison of the results of the list data processing method of the present invention and the existing list data processing method is shown in the following table.

[0067]

[0068]

[0069] It can be seen that the list data processing method of the present invention significantly improves the overall performance and user experience of the list data processing system, can effectively cope with the data processing challenges brought about by business development, and meet the growing business needs.

[0070] Embodiments of the present invention also provide a list data processing system that can be used to implement the list data processing method described in any of the above embodiments. The features and principles of the list data processing method described in any of the above embodiments can be applied to the following list data processing system embodiments. In the following list data processing system embodiments, the features and principles of list data processing that have already been explained will not be repeated.

[0071] Figure 7 The main modules of the list data processing system are shown in Figure 2. Figure 7 As shown, the list data processing system 400 includes the following modules: a storage structure optimization module 410, which is used to optimize the storage structure of the list data based on the optimization goal of reducing the resource usage of the database and lowering the storage level of the database; a data synchronization module 420, which is used to synchronize incremental data based on the optimized storage structure; a process management module 430, which is used to fully transmit the new list identifier to match the list version of the list middle platform with the application layer, and access the diversion experiment at the business logic layer and the list middle platform; a data verification module 440, which is used to keep the new list data consistent with the old list data through real-time data collection and comparison.

[0072] Furthermore, the list data processing system may also include modules for implementing other process steps of the above-mentioned list data processing method embodiments. The specific principles of each module can refer to the description of the above-mentioned list data processing method embodiments and will not be repeated here.

[0073] The list data processing system of the present invention reduces database resource usage, reduces the database storage level, shortens the full data synchronization time, and improves the real-time and accuracy of business push list making by optimizing the storage structure of list data, improving the list making and application process, and perfecting the data comparison and verification mechanism, thereby improving the overall performance of the system and user experience to meet the growing business needs.

[0074] An embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores executable instructions. When the executable instructions are executed by the processor, the list data processing method described in any of the above embodiments is implemented.

[0075] The electronic device of the present invention executes the above-mentioned list data processing method, which can reduce the resource usage of the database, reduce the storage level of the database, shorten the full data synchronization time, and improve the real-time and accuracy of business push list making by optimizing the storage structure of the list data, improving the list making and application process, and improving the data comparison and verification mechanism, thereby improving the overall performance of the system and user experience, and meeting the growing business needs.

[0076] Figure 8 The structure of the electronic equipment is shown in Figure 1. Figure 8 As shown, electronic device 600 is implemented as a general-purpose computing device. Components of electronic device 600 include, but are not limited to, at least one processing unit 610, at least one storage unit 620, and a bus 630 connecting different platform components (including storage unit 620 and processing unit 610).

[0077] The storage unit 620 stores program code, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of the list data processing method described in any of the above embodiments. The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) and / or a cache memory unit, and may further include a read-only memory unit (ROM). The storage unit 620 may also include a program / utility having one or more program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination thereof may include the implementation of a network environment.

[0078] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0079] The electronic device 600 can also communicate with one or more external devices, which can be one or more of a keyboard, a pointing device, a Bluetooth device, and the like. These external devices enable a user to interact with the electronic device 600. The electronic device 600 can also communicate with one or more other computing devices, including a router and a modem. This communication can be performed via an input / output (I / O) interface. Furthermore, the electronic device 600 can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter. The network adapter can communicate with other modules of the electronic device 600 via the bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0080] An embodiment of the present invention further provides a computer-readable storage medium for storing a program, which, when executed, implements the list data processing method described in any of the above embodiments.

[0081] When the storage medium of the present invention is executed by a processor, it can reduce the resource usage of the database, reduce the storage level of the database, shorten the full data synchronization time, and improve the real-time and accuracy of business push ranking by optimizing the storage structure of the ranking data, improving the ranking and application processes, and improving the data comparison and verification mechanism, thereby improving the overall performance of the system and user experience, and meeting the growing business needs.

[0082] The storage medium may be a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the storage medium of the present invention is not limited thereto, and can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0083] The storage medium can be any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples of readable storage media include, but are not limited to, an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0084] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. The data signal propagated may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable signal medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0085] The program code for performing the operations of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, and the like, as well as conventional procedural programming languages such as C or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device, such as via the Internet using an Internet service provider.

[0086] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the list data processing method described in any of the above embodiments.

[0087] When the computer program product of the present invention is run on a terminal device, it can reduce the resource usage of the database, reduce the storage level of the database, shorten the full data synchronization time, and improve the real-time and accuracy of business push ranking by optimizing the storage structure of the ranking data, improving the ranking and application process, and improving the data comparison and verification mechanism, thereby improving the overall performance of the system and user experience, and meeting the growing business needs.

[0088] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A list data processing method, characterized in that: The following steps are involved: Optimize the storage structure of list data based on the optimization goals of reducing database resource usage and lowering the database storage level; Synchronize incremental data based on the optimized storage structure; The new list identifier is transparently transmitted throughout the entire chain, so that the list version of the middle platform matches that of the application layer, and a diversion experiment is implemented in the business logic layer and the middle platform. By collecting and comparing real-time data, the new list data is kept consistent with the old list data.

2. The list data processing method according to claim 1, characterized in that: The optimization of the storage structure of the list data includes: Merge and remove duplicate public business data from different lists; Merge and remove duplicate related data from different language environments on the same list; Independent storage of content in different languages for the same list; Decouple the list dimension data from the listed item dimension data; The list dimension data and the listed item dimension data are stored in a unified format.

3. The list data processing method according to claim 2, characterized in that: Decouple the list dimension data from the item dimension data, including: The association relationship between the list dimension data and the listed item dimension data is converted from associating the listed item dimension data with the list dimension data through an association ID to associating the list dimension data with the listed item dimension data through a list ID.

4. The list data processing method according to claim 2, wherein: The list dimension data and the listed item dimension data are stored in a unified format, including: The list dimension data and the listed item dimension data are stored in a unified storage format using the list ID and the corresponding language environment.

5. The list data processing method according to claim 1, characterized in that: The diversion experiment was implemented in the business logic layer and the ranking platform, including: The list center dynamically selects the data interface of the new list version and the old list version based on the identification parameters requested by the user.

6. The list data processing method according to claim 1, wherein: Through real-time data collection and comparison, the new list data is kept consistent with the old list data, including: Collect real-time data from the production environment and store it persistently; After the real-time data is processed, it is compared and verified with the real-time request of the production environment to ensure that the new list data is consistent with the old list data.

7. The list data processing method according to claim 1, characterized in that: Also includes: During the migration process, new and old list data are consumed in parallel, and a smooth transition of data versions is achieved through configurable rules.

8. A list data processing system, characterized in that: For implementing the list data processing method according to any one of claims 1 to 7, the list data processing system includes the following modules: The storage structure optimization module is used to optimize the storage structure of the list data based on the optimization goals of reducing database resource usage and lowering the database storage level; Data synchronization module, used to synchronize incremental data based on the optimized storage structure; The process management module is used to transparently transmit the new list identifier throughout the entire chain, so that the list version of the middle platform and the application layer matches, and integrates the diversion experiment in the business logic layer and the middle platform; The data verification module is used to collect and compare real-time data to ensure that the new list data is consistent with the old list data.

9. An electronic device, characterized in that: include: processor; a memory, wherein executable instructions are stored in the memory; When the executable instructions are executed by the processor, the list data processing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the list data processing method according to any one of claims 1 to 7 is implemented.

Images