An automated delta collection and verification method, system, and storage medium

Abstract

The application provides an automatic incremental collection and verification method, system and storage medium. The method comprises the following steps: obtaining a site entry and a country category parameter in a configuration list, combining the parameters with a date parameter to generate an idempotent task identifier of a data collection task; obtaining list record data of a request page and encapsulating obtained standardized fields after respectively executing a static request collection mode and / or a dynamic page collection mode; performing identification analysis and version label annotation on a song name and outputting a multi-view field; storing the standardized field and the multi-view field in a target database to continuously check and repair a song ranking; storing unique identifiers in the target database in a memory collection based on a current date parameter, and repeatedly comparing through time complexity, and if the comparison is not repeated, writing the unique identifiers in the target database in batches to avoid data redundancy and save memory occupation space, and further improving the overall throughput capacity through batch writing.

Description

Technical Field

[0001] This invention relates to the field of data acquisition technology, and in particular to an automated incremental acquisition and verification method, system, and storage medium. Background Technology

[0002] With the rapid development of the digital music industry, music charts, as core data carriers reflecting music trends and user preferences, are increasingly demonstrating their data value. Music charts are widely used in various fields such as music market analysis, copyright management, and content recommendation. To unlock this data value, it is necessary to efficiently collect and integrate chart data from major music platforms, including Apple Music and Spotify.

[0003] Existing music chart data collection methods suffer from several problems: full, repetitive scraping leads to wasted bandwidth and storage; significant differences in field structures across different sites make alignment and analysis difficult; titles are affected by multilingual and version tags, making standardization challenging; deduplication often involves querying each record individually during writing, resulting in low efficiency; and some sites use dynamic rendering, causing frequent page structure changes and making parsing vulnerable. Therefore, a data collection method is needed that is compatible with multiple sites, multiple countries, and both dynamic and static pages. This method should reduce redundancy and improve efficiency while ensuring robust data parsing and rigorous validation, and possess engineering capabilities in terms of idempotency, high throughput, and traceability. Summary of the Invention

[0004] This invention provides an automated incremental data acquisition and verification method, system, and storage medium. Its main purpose is to solve the problem that existing technologies cannot accurately and efficiently acquire data from multiple versions of music charts.

[0005] To achieve the above objectives, the present invention provides an automated incremental acquisition and verification method, comprising:

[0006] Data collection tasks are triggered based on preset conditions. The site entry and country category parameters in the configuration list are obtained and combined with the date parameter to obtain the idempotent task identifier corresponding to the data collection task. The idempotent task identifier is used to determine whether the data collection task has been executed. If the data collection task has been executed, data collection is stopped; if it has not been executed, data collection continues.

[0007] After executing the static request collection mode and / or dynamic page collection mode based on the site entry, the ranking record data of the requested page is obtained;

[0008] The data recorded in the charts is encapsulated into standardized fields according to the following format: song title, artist name, album name, song ranking, song category, date parameter, data source platform, country of origin, and number of plays; the song titles are identified, analyzed, and labeled with version tags, and multi-view fields containing various forms of names are output;

[0009] The standardized fields and the multi-view fields are stored in the target database, and the song rankings in the chart data are continuously checked and repaired according to the song category; the standardized fields and the multi-view fields have unique identifiers;

[0010] After reading the unique identifier from the target database based on the date parameter, it is stored in the memory set and compared with the new data to be written to the target database in batches using time complexity. If they are duplicates, the data writing is canceled. If they are not duplicates, the new data is written to the target database in batches after the data writing threshold is reached, and a unique identifier is generated for the new data.

[0011] Optionally, after obtaining the idempotent task identifier corresponding to the data acquisition task by combining it with the date parameter, the method further includes:

[0012] After a data acquisition task is triggered according to preset conditions, the system determines whether the data acquisition task has been executed based on the idempotent task identifier. If the data acquisition task has been executed, the data acquisition is stopped; otherwise, the data acquisition continues.

[0013] Optionally, the process of executing the static request collection mode based on the site entry includes:

[0014] Access the first entry point and parse the first data source to obtain the song category, song access link, song name, and song number; if the first data source does not exist, parse the second data source to obtain the song category, song access link, song name, and song number.

[0015] Access the second entry point and parse the third data source to obtain the song category, song access link, song name, and song number.

[0016] Optionally, the process of executing the dynamic page collection mode based on the site entry includes:

[0017] Load the access page and close the pop-up notifications;

[0018] Get song ranking, song name, artist name, album, and number of plays using a preset selector.

[0019] Optionally, the method further includes:

[0020] The access URL, page text content, last usage time, and usage count of the requested page are cached in a preset storage device. When the same site entry is accessed repeatedly, the access results in the preset storage device are directly retrieved and the last usage time and usage count are updated.

[0021] Optionally, if the new data is not duplicated, it is written to the target database in batches after reaching the data writing threshold, and a unique identifier is generated for the new data.

[0022] The new data is accumulated until a preset data writing threshold is reached, and then it is stored in the target database in batches.

[0023] The new data is constructed using date parameters, song category, song ranking, song title, artist name, and data source platform as unique identifiers.

[0024] Optionally, the method further includes:

[0025] When there are abnormal records in the ranking list, access requests fail, or parsing fails, an error message is returned and the abnormal situation is stored in the log.

[0026] Optionally, performing continuity checks and repairs on song rankings based on the categories of the chart data includes:

[0027] The chart data is grouped according to the song category, and the song numbers in the chart data are checked to see if they are consecutive and without repetition.

[0028] If there are missing or duplicate serial numbers, the data in the ranking list will be supplemented or reordered according to the category sorting rules.

[0029] To address the aforementioned issues, this invention also provides an automated incremental acquisition and verification system, comprising a processor, a memory, and a computer program / instructions stored in the memory. The processor executes the computer program / instructions, and when the computer program / instructions are executed, the system implements the aforementioned automated incremental acquisition and verification method.

[0030] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the aforementioned automated incremental acquisition and verification method.

[0031] This invention provides an automated incremental data collection and verification method. It avoids duplicate data collection from leaderboards by using idempotent task identifiers, thus preventing data redundancy and saving memory. Static and dynamic page collection modes are specifically designed for different scenarios. Standardized fields eliminate field differences between different sites, providing a unified foundation for data alignment and comparative analysis and reducing the difficulty of data integration. Multi-view fields resolve formatting issues caused by multilingual titles and version tags, meeting the needs of different scenarios. Time complexity is used for duplicate comparison to improve the speed of duplicate detection and batch writing, ensuring stable database operation. Batch writing further enhances overall throughput, adapting to the needs of large-scale data collection. It ensures the sequential continuity of leaderboard records, avoiding the loss of logical data information due to collection oversights.

[0032] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0033] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating an embodiment of the automated incremental acquisition and verification method of the present invention.

[0035] Figure 2 This is a flowchart illustrating the process of executing a static request collection mode at a site entry point according to an embodiment of the present invention.

[0036] Figure 3 This is a schematic diagram illustrating the process of executing a dynamic page collection mode at the site entry point according to an embodiment of the present invention.

[0037] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0039] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0040] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0041] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0042] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0043] Figure 1 This is a flowchart illustrating an embodiment of the automated incremental acquisition and verification method of the present invention. This application provides an automated incremental acquisition and verification method, which includes the following steps S101-S105:

[0044] Step S101: Trigger a data collection task according to preset conditions, obtain the site entry and country category parameters in the configuration list, and combine them with the date parameter to obtain the idempotent task identifier corresponding to the data collection task. The idempotent task identifier is used to determine whether the data collection task has been executed. If the data collection task has been executed, stop data collection; otherwise, continue data collection.

[0045] Step S102: Obtain the ranking record data of the requested page by executing the static request collection mode and / or dynamic page collection mode according to the site entry.

[0046] Step S103: Encapsulate the chart record data into standardized fields according to the format of song name, artist name, album name, song ranking, song category, date parameter, data source platform, country, and number of plays; identify and analyze the song names and label them with version tags, and output multi-view fields containing various forms of names.

[0047] Step S104: Store the standardized fields and multi-view fields in the target database, and perform continuity checks and repairs on the song rankings in the chart data according to the song category; the standardized fields and multi-view fields have unique identifiers.

[0048] Step S105: Read the unique identifier in the target database based on the date parameter and store it in the memory set. Compare it with the new data to be written to the target database in batches using time complexity. If they are duplicates, cancel the data writing. If they are not duplicates, write the new data to the target database in batches after the new data reaches the data writing threshold, and generate a unique identifier for the new data.

[0049] In step S101, the data acquisition task is triggered via command line or timer. The configuration list is a table file containing a large amount of data, and the site entry is the URL to be accessed; the country category parameters include, but are not limited to, Thailand (TH), Australia (AU), United Kingdom (GB), Japan (JP), South Korea (KR), and Singapore (SG).

[0050] In this embodiment of the invention, after obtaining the idempotent task identifier corresponding to the data acquisition task by combining it with the date parameter, the method further includes: after triggering the data acquisition task according to preset conditions, determining whether the data acquisition task has been executed based on the idempotent task identifier; if the data acquisition task has been executed, stopping data acquisition; otherwise, continuing data acquisition.

[0051] Specifically, an idempotent task identifier is a unique identifier used to mark data acquisition tasks and ensure the idempotency of task operations. When a data acquisition task is executed, the idempotent task identifier of the task is recorded. When a new acquisition task request is received, the idempotent task identifier is used to determine whether a corresponding task execution record already exists. If it exists, the task execution is stopped and the existing request result is returned directly. If it does not exist, the task execution continues, avoiding redundant data caused by repeated execution and saving storage space.

[0052] In step S102, the static request capture mode (requests) is applicable to static pages, including but not limited to Apple Music, while the dynamic page capture mode (Selenium) is applicable to pages that need to be rendered, including but not limited to Spotify. Executing the static request capture mode and / or the dynamic page capture mode includes three scenarios: executing the static request capture mode, executing the dynamic page capture mode, and simultaneously executing both static request capture mode and dynamic page capture mode. The chart data is recorded from multiple music service platforms in different countries, including but not limited to Apple Music, Spotify, Deezer, Amazon Music, Joox, Zvuk Music, VK Music, and Melon.

[0053] Furthermore, Figure 2 This is a flowchart illustrating the process of executing a static request collection mode at a site entry point according to an embodiment of the present invention. Figure 3This is a flowchart illustrating the dynamic page collection mode executed at a site entry point according to an embodiment of the present invention. In this embodiment, the process of executing a static request collection mode based on the site entry point includes steps S1021-S1022:

[0054] Step S1021: Access the first entry point and parse the first data source to obtain the song category, song access link, song name and song number; if the first data source does not exist, parse the second data source to obtain the song category, song access link, song name and song number.

[0055] Step S1022: Access the second entry point and parse the third data source to obtain the song category, song access link, song name and song number.

[0056] Specifically, for static pages, the structured data of the playlist resource entry is parsed first to obtain the data written in Hypertext Markup Language (HTML), including but not limited to song category, song access link (URL), song name, and song number; the URL is a Uniform Resource Locator, which includes protocol, host, and path; if there is no structured data for the playlist resource entry, the second data source is parsed, which is stored in JSON text format based on key-value pairs; furthermore, the second entry is the room resource entry, and the two types of entry are compatible and improve data reading efficiency.

[0057] In this embodiment of the invention, the process of executing the dynamic page collection mode based on the site entry includes steps S1 to S2:

[0058] S1: Load the access page and close the pop-up window displaying interfering information.

[0059] S2: Obtain song ranking, song name, artist name, album, and number of plays through preset selectors.

[0060] Specifically, preset selectors include, but are not limited to, XML path language selectors (XPath selectors) and Cascading Style Sheets selectors (CSS selectors); furthermore, the information that selectors can read also includes producer information and lyricist / composer information.

[0061] In step S103, the song name, artist name, album name, song rank, song category, date parameter, data source platform, country, and play count are used as a unified output data structure. The song name contains multiple languages ​​and multiple version tags. The version tags contain version feature information of the song. Version tags include, but are not limited to, Live, Acoustic, Remix, Remaster, Speed ​​Up, Slowed + Reverb, Instrumental, Original Soundtrack, Featured, From, and With.

[0062] Furthermore, the song title is identified and structured using a title normalization algorithm (parse track name). First, the title is split and language is distinguished. Then, the split main title and version tag are obtained. Finally, multi-view fields are obtained. Specifically, the various title formats include normalized title (name), original title (name raw), normalized English title (name en), original English title (name en raw), alternative normalized title (alternative name), and alternative original title (alternative name raw).

[0063] In steps S104 and S105, the song category performs continuity repair on the song index and rank to ensure the integrity of the ordered list formed according to specific rules; in this embodiment of the invention, the continuity check and repair of song rankings based on the category of the chart data includes steps S1041~S1042:

[0064] Step S1041: Group the chart data according to song category and check whether the song numbers in the chart data are consecutive and without repetition.

[0065] Step S1042: When there are missing or duplicate serial numbers, the list data is supplemented or reordered according to the category sorting rules.

[0066] Furthermore, the unique identifiers for standardized fields and multi-view fields can adopt a structure of (song name, artist name, song category) or (original song name, artist name, song category, date parameter); after the unique identifier enters the memory set, it is determined whether the new data is duplicated in time complexity O(1). If it is duplicated, the writing is canceled; if it is not duplicated, incremental collection is achieved by batch writing (insert many). In this embodiment of the invention, if it is not duplicated, the new data is batch written to the target database after reaching the data writing threshold, and the process of generating unique identifiers for the new data includes steps S1051~S1052:

[0067] Step S1051: Accumulate new data until a preset data writing threshold is reached, then store the data in batches into the target database.

[0068] Step S1052: Construct a unique identifier for the new data based on the date parameter, song category, song ranking, song name, artist name, and data source platform.

[0069] In this embodiment of the invention, the method further includes: caching the access URL, page text content, last usage time and usage count of the requested page in a preset storage device; when the same site entry is accessed repeatedly, the access results in the preset storage device are directly called and the last usage time and usage count are updated.

[0070] In this embodiment of the invention, the method further includes: when there are abnormal ranking record data, access request failures, or parsing failures, returning an exception prompt and storing the exception status as a log. Specifically, when a request or parsing fails, a log is recorded and a retry is performed; when there are abnormal ranking changes, missing items, or missing fields, a re-collection and alarm prompts are triggered; and proxy or request header configuration is supported.

[0071] On the other hand, the present invention also provides an automated incremental acquisition and verification system, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions. When the computer program / instructions are executed, the system implements the above-mentioned automated incremental acquisition and verification method.

[0072] On the other hand, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the above-described automated incremental acquisition and verification method.

[0073] The present invention will now be described with reference to a specific embodiment:

[0074] This invention proposes an automated incremental data collection and verification method, system, and storage medium. It reads multi-site / multi-country ranking entry points from an external configuration list and generates collection tasks. For each entry point, it performs at least one of static request collection and dynamic page collection to obtain ranking records. It then performs field standardization and title normalization on the records, outputting multi-view fields including name, name en, and alternative name. Based on the daily dimension, it preloads an existing key set, combines a joint unique index and sample ID rules, and performs incremental judgment and idempotent writing. It employs batch writing and performs continuous repair on group rankings. Verified records are stored in a database, with optional recording of source page cache and processing logs. Static request collection prioritizes parsing schema:music-playlist, and if it fails, it falls back to parsing the serialized-server-data JSON data structure. Dynamic page collection, after page loading and pop-up closing, extracts ranking, song name, artist, album, and play count according to a preset selector, and optionally opens a side window to extract producer or lyricist / composer information. Title normalization includes parsing bracket content, English and non-English language discrimination, version tag recognition, and multi-view output. The preloaded key set includes (normalized title, artist, category) or (original title, artist, category, date) key combinations for the current day dimension, used for deduplication in O(1) time during the collection loop. Sample IDs are generated according to song {date} {country} {serial number}, where the sequence number is a zero-padded integer incrementing by country for the current day. The database uses batch insert many writes and pushes to disk when a threshold is reached. Grouping and consecutive number repair is performed for each list category to ensure continuous ranking. The database contains a composite unique index for (normalized title, artist, category, date, source) and a query index for (date, category, ranking). The method includes the following:

[0075] 1. Data Acquisition Task Configuration and Scheduling. The system reads site entry points and country parameters from an external list, combining them with the date to form an idempotent task identifier with a site, country, and date structure. Data acquisition tasks are triggered via command line or timer. The external list can be in Excel spreadsheet format. Country parameters include TH, AU, GB, JP, KR, and SG.

[0076] 2. Dual-channel data collection: static request mode (requests) and dynamic page collection mode (Selenium). Static request mode targets Apple Music's static pages, prioritizing the parsing of schema:music-playlist; if missing, it falls back to parsing serialized-server-data JSON; it is compatible with both playlist and room entry points, extracting category names, track access links (URLs), song names, and sequence numbers. Dynamic page collection mode (Selenium) targets Spotify pages that need to be rendered; after loading the page and closing the pop-up, it extracts rankings, song names, artists, albums, and play counts using stable XPath / CSS selectors, and optionally opens a side window to extract producer or songwriter information.

[0077] 3. URL-level source data caching. The source page is cached based on the requested page's URL, page text content (html), last used time (last used), and used count (used count). The caching strategy is to reuse the cached data directly if it is hit, and to initiate a network request if it is not hit, thereby reducing bandwidth and site pressure.

[0078] 4. Field standardization and multilingual title normalization. Establish a standard structure, which includes song name, artist name, album name, song rank, song category, date parameter, data source platform, country, and play count;

[0079] The title normalization algorithm (parse track name) parses the content in brackets within the title and the main title, distinguishing between English and non-English text; it identifies and removes or labels version tags, including Live, Acoustic, Remix, Remaster, Sped Up, Slowed + Reverb, Instrumental, OST, feat, from, and with; the output includes a multi-view field containing the normalized title (name), the original title (name raw), the normalized English title (name en), the original English title (name en raw), the alternative normalized title (alternative name), and the alternative original title (alternative name raw), adapting to multiple languages, bracket variations, and version differences.

[0080] 5. Grouping and Consecutive Number Repair and Consequence Enhancement. The sequence number (index) or rank is repaired by grouping by category. Missing or misaligned numbers are rearranged to ensure the integrity of the Top N sequence.

[0081] 6. Preload key set deduplication and idempotent writing. Existing keys (name, singer, category) or (name raw, singer, category, date) are preloaded from the target set according to the current day dimension and loaded into the memory set. Deduplication is performed in O(1) time within the loop. Sample IDs are in the form of song{YYYY-MM-DD} {country} {seq:06d}, incrementing by country and date to ensure readability and order. YYYY-MM-DD represents the collection date, country represents the country parameter, and seq:06d represents the sequence number incrementing by country and date. Writing is done in batches (insert many), for example, every 100 records are considered a batch, significantly improving throughput. A composite unique index is established at the database layer to ensure eventual consistency.

[0082] 7. Quality Control and Anomaly Handling. Request failures and parsing failures are logged separately and can be retried. Network request failures and page parsing failures can be logged separately. Ranking jump anomalies, missing items, or missing fields trigger re-sampling or alarms. Proxy / request header configuration is supported to control throttling and site friendliness.

[0083] 8. An automated incremental acquisition and verification method system, comprising a task scheduling module, a static acquisition module, a dynamic acquisition module, a normalization module, a deduplication and idempotent writing module, a batch writing module, a caching and monitoring module, and a storage module.

[0084] 9. The specific process of collecting and storing Apple Music data based on requests includes:

[0085] (1) Entry and country configuration: Read the list of links from Excel, and map the country code to the database name, collection name and file path. For example, TH maps to song and GB maps to gb.

[0086] (2) Page parsing. When accessing https: / / music.apple.com / {cc} / playlist / ..., the JSON with id="schema:music-playlist" is parsed first to extract category / track(url,name) / index; if missing, the JSON with id="serialized-server-data" is parsed and data.sections.items is traversed to construct the song URL; when accessing the room link, the corresponding JSON is parsed to extract the category and item lists.

[0087] (3) Basic data entry and deduplication. Successfully parsed basic data is written to Apple Update Base and deduplicated using the combination of {url, date, category}.

[0088] (4) Details and caching. Access the track URL one by one for the base record of the day; if the source cache col(url / html / last used / used count) is hit, the html cache is directly reused; otherwise, it is written to the cache after the request. (5) Field extraction and merging: The album, artist collection, lyricist / composer, etc. in serialized-server-data are parsed and merged to form the apple update detail document.

[0089] (6) Normalization and daily deduplication. The parse track name is used to output the name, name en, and alternativename fields, and deduplication is performed on the daily dimension using two-level keys: {category,url} and {category,song,artist name}.

[0090] (7) Batch writing and indexing. Accumulate data to the threshold and then insert many.

[0091] Indexing suggestions. Apple update detail: index(date, category, rank), combined unique(song, artist_name, category, date, source). Apple update source: index(url).

[0092] 10. The specific process of Spotify data collection and storage based on Selenium includes:

[0093] (1) Driver initialization. The browser starts, and the browser can be headless. The cookie pop-up is closed.

[0094] (2) TopN extraction and threshold control. Parse the category name of the list; if there are already records that have reached the threshold for the day, skip the collection of that category.

[0095] (3) Field extraction. Extract core fields such as ranking, song, artist, album and play count through stable selector XPath / CSS; optionally, view the producer side window to extract lyrics and music and producer information and write it back into the basic data.

[0096] (4) Deduplication and ranking repair. Deduplication is performed by {song name, artist name, album name, category, date, play count}, and grouping and consecutive number repair is combined to ensure continuous ranking.

[0097] (5) Batch import. Write to the spotify origindata xx collection.

[0098] 11. Target library sampler and idempotent write for Apple or Spotify.

[0099] (1) Preloaded key set. Read (name, singer, category) or (name raw, singer, category, date) combinations from the target set of song update at the same time according to the dimension of the day, and form the memory set existingkeys.

[0100] (2) Sample ID generation. The sample id = "song {YYYY-MM-DD} {country} {seq:06d}" is incremented by country and date to facilitate horizontal comparison and sequential traversal.

[0101] (3) Target structure (annotation encapsulation). Includes fields such as popularity (rank), name (multiple views), playnumber, category, singer / writer / album, last update, source, country, and other tags.

[0102] (4) Index suggestions. Sample ID unique index; query index (annotation.0.last update, annotation.0.category, annotation.0.popularity); composite unique index (annotation.0.name, annotation.0.singer name, annotation.0.category, annotation.0.last update, annotation.0.source).

[0103] (5) Batch Write and Throughput. After accumulating to the batch threshold, insert many to disk to reduce write amplification and lock contention. The example supports proxy, request headers, and throttling control; disabling SSL verification is only for debugging / specific environments, and it is recommended to retain verification and maintain a trusted root certificate in production. TTL or raw response hash can be optionally added for auditing and re-collection.

[0104] 12. Optional implementations and extensions include:

[0105] (1) Cache TTL and snapshots. Set the TTL for the cache collection or save the response hash for easy auditing and re-sampling.

[0106] (2) Abnormal rollback and retry. Network failure index backoff; DOM mismatch automatically switches to a backup selector.

[0107] (3) Multi-site expansion. The same framework can be extended to Deezer, Amazon Music, Joox, Zvuk, VK, Melon, etc.

[0108] (4) Statistics and dashboards. Indicators such as collection coverage, incremental ratio, reasons for failure, and TopN completeness rate.

[0109] (5) Data consistency alignment. If the ranking difference between multiple sources on the same day and the same song exceeds the threshold, arbitration is triggered. For example, majority voting or weighted voting is adopted.

[0110] In summary, this invention provides an automated incremental data acquisition and verification method, system, and storage medium. The method includes triggering a data acquisition task based on preset conditions, obtaining site entry and country category parameters from a configuration list, and combining them with date parameters to obtain an idempotent task identifier for generating the data acquisition task; obtaining the chart record data of the requested page after executing a static request acquisition mode and / or a dynamic page acquisition mode based on the site entry; encapsulating the chart record data into standardized fields in the form of song name, artist name, album name, song ranking, song category, date parameter, data source platform, country, and play count; identifying and analyzing the song names and labeling them with version tags, and outputting multi-view fields containing various name formats; storing the standardized fields and multi-view fields in a target database, and performing continuity checks and repairs on the song rankings in the chart data based on the song category; reading the unique identifier from the target database based on the date parameter and storing it in a memory set, comparing it with new data to be batch-written into the target database using time complexity; if duplicates are found, the data writing is canceled; if no duplicates are found, the new data is batch-written into the target database after reaching the data writing threshold, and a unique identifier is generated for the new data.

[0111] This invention also provides an electronic device, the electronic device comprising:

[0112] At least one processor; and,

[0113] A memory communicatively connected to the at least one processor; wherein,

[0114] The memory has a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the point cloud and camera fusion annotation method described above.

[0115] This invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the point cloud and camera fusion annotation method described above.

[0116] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the utility model patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims

1. An automated incremental acquisition and verification method, characterized in that, The method includes: Data collection tasks are triggered based on preset conditions. The site entry and country category parameters in the configuration list are obtained and combined with the date parameter to obtain the idempotent task identifier corresponding to the data collection task. The idempotent task identifier is used to determine whether the data collection task has been executed. If the data collection task has been executed, data collection is stopped; if it has not been executed, data collection continues. After executing the static request collection mode and / or dynamic page collection mode based on the site entry, the ranking record data of the requested page is obtained; The data recorded in the charts is encapsulated into standardized fields according to the following format: song title, artist name, album name, song ranking, song category, date parameter, data source platform, country of origin, and number of plays; the song titles are identified, analyzed, and labeled with version tags, and multi-view fields containing various forms of names are output; The standardized field and the multi-view field are stored in the target database, and the song rankings in the chart record data are checked and repaired according to the song category; the standardized field and the multi-view field have unique identifiers; After reading the unique identifier from the target database based on the date parameter, it is stored in the memory set and compared with the new data to be written to the target database in batches using time complexity. If they are duplicates, the data writing is canceled. If they are not duplicates, the new data is written to the target database in batches after the data writing threshold is reached, and a unique identifier is generated for the new data.

2. The automated incremental acquisition and verification method according to claim 1, characterized in that, The process of executing the static request collection mode based on the site entry includes: Access the first entry point and parse the first data source to obtain the song category, song access link, song name, and song number; if the first data source does not exist, parse the second data source to obtain the song category, song access link, song name, and song number. Access the second entry point and parse the third data source to obtain the song category, song access link, song name, and song number.

3. The automated incremental acquisition and verification method according to claim 1, characterized in that, The process of executing the dynamic page collection mode based on the site entry includes: Load the access page and close the pop-up notifications; Get song ranking, song name, artist name, album, and number of plays using a preset selector.

4. The automated incremental acquisition and verification method according to claim 1, characterized in that, The method further includes: The access URL, page text content, last usage time, and usage count of the requested page are cached in a preset storage device. When the same site entry is accessed repeatedly, the access results in the preset storage device are directly retrieved and the last usage time and usage count are updated.

5. The automated incremental acquisition and verification method according to claim 1, characterized in that, If no duplicates are found, the new data is written in batches to the target database after reaching the data writing threshold, and a unique identifier is generated for the new data. The process includes: The new data is accumulated until a preset data writing threshold is reached, and then it is stored in the target database in batches. The new data is constructed using date parameters, song category, song ranking, song title, artist name, and data source platform as unique identifiers.

6. The automated incremental acquisition and verification method according to claim 1, characterized in that, The method further includes: When there are abnormal records in the ranking list, access requests fail, or parsing fails, an error message is returned and the abnormal situation is stored in the log.

7. The automated incremental acquisition and verification method according to claim 1, characterized in that, The continuity check and repair of song rankings based on the categories of the data recorded in the charts includes: The chart record data is grouped according to the song category, and the song sequence numbers in the chart record data are checked to see if they are consecutive and without repetition. If there are missing or duplicate serial numbers, the data in the list will be supplemented or reordered according to the category sorting rules.

8. An automated incremental acquisition and verification system, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images