A mobile phone application uploading information acquisition method based on network search

Abstract

The application discloses a mobile phone application uploading information acquisition method based on network search, comprising the following steps: S1, constructing a basic data layer; S2, calling a search engine for search by an application market active discovery layer; S3, deeply analyzing a webpage after preliminary screening; S4, matching features obtained by analysis with preset rules in an application market judgment rule library; S5, supplementing application market information judged as valid into the initial application market library to update and form a full-amount application market library; S6, accessing application list pages of each application market in the updated full-amount application market library based on an application data collection layer, collecting structured information and APK download links of the applications; S7, verifying the effectiveness of the collected APK download links; and S8, an iterative optimization layer counting the search efficiency, search effectiveness, effect index of the application market active discovery layer and the collection success rate of the application data collection layer.

Description

Technical Field

[0001] This invention relates to the field of mobile application listing, and more specifically, to a method for obtaining mobile application listing information based on web retrieval. Background Technology

[0002] In the mobile internet sector, mobile application distribution channels have diversified. Besides the 20-30 mainstream app stores like Tencent App Store and Wandoujia, there are thousands of small and medium-sized app stores and personal app distribution platforms, becoming the listing options for some developers. App store retrieval and data collection technology is the core of discovering and monitoring mobile applications. This technology relies on web crawlers, webpage parsing, and data storage to obtain application information. Currently, mainstream technologies are only developed for top app stores and lack coverage of the massive number of niche distribution platforms, significantly increasing the difficulty of discovering non-mainstream applications.

[0003] Existing mobile application discovery technologies primarily employ a "manually compiled static app market database + targeted data collection" approach: technical personnel manually collect the URLs and API information of mainstream and a small number of niche app markets to build a fixed app market database; then, web crawling technology is used to selectively collect information from the app markets in this database, extracting and storing information such as app names, versions, and APK installation packages; when users search for apps, they can only access app data from this static database. This approach lacks a mechanism for proactively discovering new app markets, and it does not include search engine retrieval or intelligent app market determination modules. The scope of collection and retrieval relies entirely on the initially manually compiled content, making it unsuitable for distribution scenarios involving a massive number of niche and personal app distribution platforms.

[0004] In summary, the existing technologies described above have the following obvious drawbacks: (1) The application market has limited coverage, only covering manually curated application markets, and cannot reach thousands of niche and personal application distribution platforms, resulting in insufficient comprehensiveness of application discovery. (2) It lacks the ability to proactively discover new application markets. The application market library is a static library, and newly added application distribution platforms need to be manually supplemented, resulting in extremely low update efficiency. (3) There are many missing application search results. Because some applications are only available on niche / personal application markets, users cannot find relevant applications through this system, which cannot meet the search needs of all applications. (4) Manually organizing and maintaining the application market database is costly and time-consuming. It is difficult to achieve comprehensive collection and timely updates when facing scattered niche distribution platforms. Summary of the Invention

[0005] This invention addresses the problems of incomplete market coverage, lack of proactive discovery capabilities in static databases, missing application search results, and high manual maintenance costs in existing technologies. It provides a method for obtaining mobile application listing information based on network retrieval, enabling dynamic proactive discovery, full inclusion, and continuous collection of application data in application markets. This solves the problem of users being unable to find applications listed in niche / personal application markets, while also reducing the manual maintenance costs of application market databases.

[0006] To achieve the above objectives, the present invention provides a method for obtaining mobile application listing information based on network retrieval, comprising: S1: Construct the basic data layer, which includes an initial application market library, a search keyword library, an application market judgment rule library, and an application data storage library. S2: Based on keywords in the keyword database, the application market proactive discovery layer calls the search engine to perform a search and performs an initial screening of the returned web page links to remove invalid and interfering links. At the same time, it calculates the search effectiveness and keyword search efficiency to provide data for subsequent iterative optimization. S3: Perform in-depth analysis on the web pages after initial screening to extract their structural and content features; S4: Match the parsed features with the preset rules in the application market determination rule base, and automatically determine the webpage as a valid application market, a suspected application market, or a non-application market according to the preset determination threshold. S5: Supplement the valid app market information into the initial app market database to update and form the full app market database. , For the library update rate, That is, the number of app stores that are determined to be valid and newly entered in step S4. That is, the number of original app stores in the initial app store library; S6: The application data collection layer is based on the updated full application market library. It accesses the application list pages of each application market, collects the structured information of the applications and APK download links, and calculates the collection success rate. S7: Verify the validity of the collected APK download links, download the APK files that can be accessed normally, calculate the MD5 value of each APK file, compare it with the MD5 values ​​of APK files already stored in the application data repository to remove duplicates, and associate and store the deduplicated APK files with the corresponding application listing information to ensure file traceability. , To ensure the validity of the APK link, The number of links that can be accessed and downloaded normally. This represents the total number of APK download links obtained in this data collection task. , To improve the accuracy of MD5 deduplication. This represents the number of duplicate APK files successfully identified and deleted via MD5 comparison. This represents the actual number of duplicate files present in the downloaded APK file. S8: The iterative optimization layer analyzes the retrieval efficiency, retrieval effectiveness, and performance indicators of the application market proactive discovery layer, as well as the collection success rate of the application data collection layer. Based on the threshold values ​​of each indicator, it optimizes the retrieval keyword library and the application market judgment rule library. At the same time, it adjusts the collection layer parameters according to the collection success rate and feeds the optimized library and parameters back to the corresponding level to achieve continuous optimization throughout the entire process.

[0007] In one embodiment of the present invention, step S2 may optionally include: randomly selecting 5 to 10 keywords from the keyword database and concatenating them into a search statement; calling the public application programming interface of the search engine to obtain the first 500 search results; and performing a preliminary screening of the search results to remove duplicate Uniform Resource Locators and invalid links with status codes of 404 or 500.

[0008] In one embodiment of the present invention, optionally, in step S4, S = W + C S is the total score for rule matching, and its value ranges from [0, W-total + C-total]. W represents the number of successfully matched webpage feature rules, C represents the number of successfully matched content feature rules, W-total represents the total number of webpage feature rules, and C-total represents the total number of content feature rules. The app market type is determined based on the total rule matching score as follows: Effective application market: satisfying S≥3 and W≥2 and C≥1. Suspected app market: Satisfies S=1, Non-application market: satisfies S=0, , For rule matching accuracy, The number of cases where the result in step S4 matches the actual number. This represents the total number of web pages involved in the judgment process in S4.

[0009] In one embodiment of the present invention, optionally, the preset application market determination rule base in step S4 includes webpage feature rules and content feature rules; the determination threshold is set to determine a valid application market when at least two webpage feature rules and at least one content feature rule are met; if only one rule is met, it is marked as a suspected application market; if no rule is met, it is determined as a non-application market.

[0010] In one embodiment of the present invention, optionally, the webpage feature rules include one or more of the following: the source code contains application download keywords, there is an application category navigation bar, and there is an application details page; and / or Content feature rules include one or more of the following: ≥30% of links are APK download addresses and non-social / e-commerce domains.

[0011] In one embodiment of the present invention, step S6 may optionally include: classifying the application markets in the full application market library, setting different collection frequencies for mainstream application markets and niche application markets, allocating collection tasks using an asynchronous concurrent framework, performing differentiated timed collection for application markets of different levels, and cleaning and storing the collected application structured information in the application data storage repository.

[0012] In one embodiment of the present invention, step 7 may optionally include: verifying the validity of the collected APK download link, downloading the APK file after it passes the verification, calculating the MD5 value of the downloaded file, and comparing it with the MD5 value of the stored APK file; if they match, the file is determined to be a duplicate and deleted; if they do not match, the file is stored in a file server according to a preset hierarchical path, and the storage information is recorded in the application data repository.

[0013] In one embodiment of the present invention, optionally, the performance indicators statistically analyzed in step S8 include rule matching accuracy, keyword retrieval efficiency, number of new application markets discovered in a single batch, and collection success rate; the keyword database is optimized based on keyword retrieval efficiency, inefficient keywords with retrieval efficiency below a first threshold are removed, related words of efficient keywords with retrieval efficiency above a second threshold are derived, and high-frequency words are extracted from the webpage summaries of newly discovered application markets to supplement the keyword database; the application market judgment rule database is optimized based on rule matching accuracy, sub-rules are added for rules that cause misjudgments, new rules are added for new types of application markets not covered by existing rules, and redundant rules are merged; the number of concurrent coroutines, request interval, and timeout parameters of the collection layer are adjusted based on the collection success rate.

[0014] In one embodiment of the present invention, optionally, in step S2, , For search effectiveness, This refers to the number of valid webpage links in the search results after initial screening, i.e., the number of links after removing duplicates, 404 / 500 status codes, and interfering links from social media / e-commerce platforms. This represents the total number of original links returned by the search engine. , To improve keyword retrieval efficiency, This refers to the number of valid webpage links in the search results after initial screening, i.e., the number of links after removing invalid and interfering links. This represents the total time from calling the search engine API to completing the initial screening of all results for a single search.

[0015] In one embodiment of the present invention, optionally, in step S6, , To increase the success rate of data collection, This refers to the number of application information items that have been collected, cleaned, and are suitable for storage before being uploaded to the app store. The target number of applications is preset for this data collection task.

[0016] In one embodiment of the present invention, optionally, in step S8,

[0017] To optimize performance, , These represent the keyword retrieval efficiency before and after optimization, respectively. , The results show the search effectiveness before and after optimization. , The data collection success rates are shown for the optimized and unoptimized versions, respectively.

[0018] The method for obtaining mobile application listing information based on network retrieval provided by this invention has the following beneficial technical effects: (1) It has the ability to proactively discover application markets and automatically collect new application markets through search engine retrieval and intelligent judgment, without the need for manual collection; (2) It has a wide coverage, covering more than 3,000 application markets and enabling monitoring of more than 95% of application listing scenarios; (3) Construct a closed-loop system of "discovery-collection-optimization", dynamically update the application market library, and continuously adapt to new application distribution channels; (4) High degree of automation: The entire process from application market discovery to application data collection is automated, with only a small number of suspected markets requiring manual review, which greatly reduces labor costs; (5) The application search results are comprehensive, and can obtain application listing information and APK installation packages from the full range of application markets, solving the problem of difficulty in finding non-mainframe applications. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a logical block diagram of the overall technical architecture; Figure 2 Flowchart for proactive app market discovery; Figure 3 Flowchart for closed-loop iterative optimization; Figure 4 Flowchart for APK installation package download and storage. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Figure 1 This is a logical block diagram of the overall technical architecture. Figure 2 A flowchart for proactively discovering applications in the market. Figure 3 The flowchart for closed-loop iterative optimization is as follows: Figure 4 A flowchart of the APK installation package download and storage process, such as... Figures 1-4 As shown, this invention provides a method for obtaining mobile application listing information based on network retrieval, which includes: S1: Construct the basic data layer, which includes an initial application market library, a search keyword library, an application market judgment rule library, and an application data storage library. S2: Based on keywords in the keyword database, the application market proactive discovery layer calls the search engine to perform a search and performs an initial screening of the returned web page links to remove invalid and interfering links. At the same time, it calculates the search effectiveness and keyword search efficiency to provide data for subsequent iterative optimization. S3: Perform in-depth analysis on the web pages after initial screening to extract their structural and content features; The effectiveness of the parsing is quantified by "number of effective extracted features / total number of extracted features" to verify the parsing quality. The specific calculation formula is as follows: , This represents the parsing effectiveness (in %). A higher value indicates better parsing quality and more sufficient features available for judgment. This represents the number of valid features (such as application download entry, version information, etc.) that can be used for subsequent rule matching after parsing. , which represents the total number of features attempted to be extracted during the parsing process.

[0023] At the same time, a preset parsing validity threshold is set. ,when At that time, the webpage feature parsing is deemed qualified.

[0024] S4: Match the parsed features with the preset rules in the application market determination rule base, and automatically determine the webpage as a valid application market, a suspected application market, or a non-application market according to the preset determination threshold. S5: Supplement the valid app market information into the initial app market database to update and form the full app market database. , This refers to the library update rate, used to measure the effectiveness of updates. A higher value indicates a more significant update effect and more comprehensive market coverage. That is, the number of app stores that are determined to be valid and newly entered in step S4. That is, the number of original app stores in the initial app store library; S6: The application data collection layer is based on the updated full application market library. It accesses the application list pages of each application market, collects the structured information of the applications and APK download links, and calculates the collection success rate. S7: Verify the validity of the collected APK download links, download the APK files that can be accessed normally, calculate the MD5 value of each APK file, compare it with the MD5 values ​​of APK files already stored in the application data repository to remove duplicates, and associate and store the deduplicated APK files with the corresponding application listing information to ensure file traceability. , This value indicates the validity of the APK link and is used to verify the link quality. The higher the value, the more likely the link is to be downloadable. The number of links that can be accessed and downloaded normally. This represents the total number of APK download links obtained in this data collection task. , This represents the MD5 deduplication accuracy, used to evaluate the deduplication effect. The higher the value, the more accurate the deduplication. This represents the number of duplicate APK files successfully identified and deleted via MD5 comparison. This represents the actual number of duplicate files present in the downloaded APK file. S8: The iterative optimization layer analyzes the retrieval efficiency, retrieval effectiveness, and performance indicators of the application market proactive discovery layer, as well as the collection success rate of the application data collection layer. Based on the threshold values ​​of each indicator, it optimizes the retrieval keyword library and the application market judgment rule library. At the same time, it adjusts the collection layer parameters according to the collection success rate and feeds the optimized library and parameters back to the corresponding level to achieve continuous optimization throughout the entire process.

[0025] In one embodiment of the present invention, step S2 may optionally include: randomly selecting 5 to 10 keywords from the keyword database and concatenating them into a search statement; calling the public application programming interface of the search engine to obtain the first 500 search results; and performing a preliminary screening of the search results to remove duplicate Uniform Resource Locators and invalid links with status codes of 404 or 500.

[0026] In one embodiment of the present invention, optionally, in step S4, S = W + C S is the total score for rule matching, and its value ranges from [0, W-total + C-total]. W represents the number of successfully matched webpage feature rules, C represents the number of successfully matched content feature rules, W-total represents the total number of webpage feature rules, and C-total represents the total number of content feature rules. The app market type is determined based on the total rule matching score as follows: Effective application market: satisfying S≥3 and W≥2 and C≥1. Suspected app market: Satisfies S=1, Non-application market: satisfies S=0, , For rule matching accuracy, The number of cases where the result in step S4 matches the actual number. This represents the total number of web pages involved in the judgment process in S4.

[0027] In one embodiment of the present invention, optionally, the preset application market determination rule base in step S4 includes webpage feature rules and content feature rules; the determination threshold is set to determine a valid application market when at least two webpage feature rules and at least one content feature rule are met; if only one rule is met, it is marked as a suspected application market; if no rule is met, it is determined as a non-application market.

[0028] In one embodiment of the present invention, optionally, the webpage feature rules include one or more of the following: the source code contains application download keywords, there is an application category navigation bar, and there is an application details page; and / or Content feature rules include one or more of the following: ≥30% of links are APK download addresses and non-social / e-commerce domains.

[0029] In one embodiment of the present invention, step S6 may optionally include: classifying the application markets in the full application market library, setting different collection frequencies for mainstream application markets and niche application markets, allocating collection tasks using an asynchronous concurrent framework, performing differentiated timed collection for application markets of different levels, and cleaning and storing the collected application structured information in the application data storage repository.

[0030] In one embodiment of the present invention, step 7 may optionally include: verifying the validity of the collected APK download link, downloading the APK file after it passes the verification, calculating the MD5 value of the downloaded file, and comparing it with the MD5 value of the stored APK file; if they match, the file is determined to be a duplicate and deleted; if they do not match, the file is stored in a file server according to a preset hierarchical path, and the storage information is recorded in the application data repository.

[0031] In one embodiment of the present invention, optionally, the performance indicators statistically analyzed in step S8 include rule matching accuracy, keyword retrieval efficiency, number of new application markets discovered in a single batch, and collection success rate; the keyword database is optimized based on keyword retrieval efficiency, inefficient keywords with retrieval efficiency below a first threshold are removed, related words of efficient keywords with retrieval efficiency above a second threshold are derived, and high-frequency words are extracted from the webpage summaries of newly discovered application markets to supplement the keyword database; the application market judgment rule database is optimized based on rule matching accuracy, sub-rules are added for rules that cause misjudgments, new rules are added for new types of application markets not covered by existing rules, and redundant rules are merged; the number of concurrent coroutines, request interval, and timeout parameters of the collection layer are adjusted based on the collection success rate.

[0032] In one embodiment of the present invention, optionally, in step S2, , Search validity measures the quality of search results; a higher score indicates better results and a higher percentage of valid links. This refers to the number of valid webpage links in the search results after initial screening, i.e., the number of links after removing duplicates, 404 / 500 status codes, and interfering links from social media / e-commerce platforms. The total number of original links returned by the search engine ( In one embodiment of the present invention, the value is set to 500, that is, the first 500 search results are obtained and subsequent results are ignored. , Keyword retrieval efficiency is used for subsequent keyword database optimization. A higher value indicates stronger retrieval efficiency and shorter processing time. This refers to the number of valid webpage links in the search results after initial screening, i.e., the number of links after removing invalid and interfering links. This represents the total time from calling the search engine API to completing the initial screening of all results for a single search.

[0033] In this invention, a first threshold for keyword retrieval efficiency can be preset. (e.g., 0.5 messages / second), second threshold (e.g., 2 items / second), used for keyword library optimization and determination in subsequent step S8.

[0034] In one embodiment of the present invention, optionally, in step S6, , The success rate of data collection is used to evaluate the effectiveness of data collection tasks; a higher value indicates better task performance. This refers to the number of application information items that have been collected, cleaned, and are suitable for storage before being uploaded to the app store. The target number of applications is preset for this data collection task.

[0035] In one embodiment of the present invention, optionally, in step S8,

[0036] To optimize the results and evaluate the effectiveness of the optimization, , These represent the keyword retrieval efficiency before and after optimization, respectively. , The results show the search effectiveness before and after optimization. , The data collection success rates are shown for the optimized and unoptimized versions, respectively.

[0037] In this invention, for example, an optimization effect threshold can be preset. ,when If the optimization is successful, it is deemed effective; otherwise, the optimization strategy must be readjusted and the optimization steps executed again.

[0038] This invention provides a method for obtaining mobile application listing information based on network retrieval, developed using Python. It constructs a four-layer technical system: a basic data layer, an application market proactive discovery layer, an application data collection layer, and an iterative optimization layer. Through the collaborative work of these layers, it achieves dynamic inclusion of application markets and continuous collection of application data, forming a closed loop of "application collection - new application market discovery - new application collection - rule optimization - more application market discovery," ultimately solving the technical problems of incomplete application market coverage and difficulty in finding non-mainstream applications. The core of this solution is "search engine keyword retrieval + application market rule-based judgment + automated data collection + closed-loop iterative optimization," with over 95% automation throughout the process. The following details the structure, function, workflow, key parameters, and overall closed-loop logic of each layer. The basic data layer is the data source and storage core of the entire technical solution. It undertakes the functions of initial data storage, data collection and storage, and rule / keyword library management, providing data support for the other three layers and realizing the separation and storage of structured and unstructured data.

[0039] Storage medium for the basic data layer: This invention uses a MySQL relational database to store structured data (application market information, application data, keyword library, judgment rule library, etc.); and a FastDFS file server to store unstructured APK installation package files, improving data retrieval and calling efficiency.

[0040] The core data set on the basic data layer: (1) Initial application market database: pre-enter information on 2000+ known application markets (including mainstream platforms such as Tencent App Store and Wandoujia, and niche platforms such as AppChina and Dragonfly Download Station). Each record contains fields such as "name, URL, page features, collection interface, and collection frequency". (2) Keyword database: Initially contains core keywords such as "game download, application download, Android APK download, mobile APP acquisition", and supports dynamic addition and deletion; (3) Application Market Judgment Rule Base: Preset two types of rules: webpage feature rules and content feature rules; (4) Application data storage repository: Stores the collected application structured data, including fields such as "name, developer, version number, APK download link, storage path".

[0041] Data management rules at the basic data layer: MySQL is managed hierarchically by database / table; application data is stored in tables categorized by application market (each table ≤ 1 million records); APK files are stored hierarchically by "application market - application name - version number", and unique identification is achieved through MD5 values ​​to avoid redundancy.

[0042] The app market proactive discovery layer is the core module for dynamically including new apps in the app market. Developed based on Python web crawler technology, it connects to the public search interfaces of search engines such as Baidu, 360, and Bing. Through the process of "keyword search - result filtering - rule matching - library update", it realizes the intelligent identification, judgment and inclusion of new apps in the app market, solving the problem that existing technologies lack proactive discovery capabilities.

[0043] The application marketplace's proactive discovery layer is developed using the Scrapy (or PySpider) web crawling framework combined with the Requests + BeautifulSoup web page parsing library (or XPath / lxml), balancing crawling concurrency efficiency and web page parsing flexibility. The core workflow and key parameters are as follows: Keyword retrieval performed by the app market's proactive discovery layer: 5-10 keywords are randomly selected from the keyword database (the number can be adjusted according to the search engine's anti-scraping rules), and concatenated into a search statement in the form of "keyword 1 + keyword 2" (e.g., "game download + Android app market"). The search engine's public API is then called to obtain the first 500 search results. The search results include basic information such as webpage title, URL, webpage summary, and status code.

[0044] The application market proactive discovery layer performs initial screening of search results: it automatically filters the search results and removes three types of invalid results: duplicate URLs, invalid links with status codes of 404 / 500, and interfering links whose domains belong to mainstream social / e-commerce platforms such as Taobao, JD.com, and WeChat, and only retains unknown web page links that are not yet included and have normal status. The application market proactive discovery layer performs deep webpage parsing: it performs batch parsing of unknown webpage links after initial screening, obtains the webpage source code through Python's Requests library, extracts the core content of the webpage through the BeautifulSoup library, including navigation bar structure, link types, text keywords, domain information, etc., and temporarily stores the parsing results.

[0045] The application market proactive discovery layer uses rule-based intelligent judgment: It matches the parsed webpage results against rules in the application market judgment rule base one by one, setting judgment thresholds: If ≥2 webpage feature rules + ≥1 content feature rule are met, it is directly judged as a valid application market; if only 1 rule is met (regardless of webpage / content feature), it is marked as a suspected application market; if no rule is met, it is judged as a non-application market and directly filtered. 5. Application Market Database Update: The names, URLs, parsed page features / collection interfaces, etc., of the judged valid application markets are supplemented and entered according to the initial application market database field structure, and the inclusion time and source search engine are marked; suspected application markets are stored separately in a review database for manual review by technical personnel. If the review is valid, it is added to the application market database; if the review is invalid, it is removed. 6. Key Parameters: The crawler request interval is set to 1-3 seconds (different intervals are adapted to different search engines to avoid IP blocking); the maximum number of webpages parsed in a single batch is 1000; the judgment threshold for rule matching is "2 webpage features + 1...". Content features (tested to keep the false positive rate below 5%).

[0046] The application data acquisition layer automates the collection, cleaning, and storage of application information and APK installation packages from the full application marketplace (initial library + newly added library) in the basic data layer, addressing the limitation of existing technologies in application data collection scope. This module is developed based on Python's AsyncIO (or aiohttp) asynchronous framework, enabling concurrent collection from multiple application marketplaces while balancing collection efficiency and server load. It also includes scheduled collection tasks for continuous monitoring and updating of application data.

[0047] The application data acquisition layer allocates acquisition tasks: based on the access frequency limit and data volume of the application market, all markets in the application market library are classified into levels, with mainstream application markets being level A and small / medium-sized / personal application markets being level B; an asynchronous concurrent mode is adopted, splitting the acquisition tasks into 10 concurrent coroutines, with each coroutine responsible for the acquisition of 1-5 application markets, and level A markets being assigned independent coroutines to avoid acquisition conflicts.

[0048] Application data acquisition layer: For each app store, access its app list page / category page, and extract information such as app name, developer, version number, release date, app description, download count, and APK download link based on pre-parsed page features / collection interfaces. Automated cleaning of the extracted raw information is performed, removing special characters and completing missing fields (marking "unknown" for no version number and "0" for no download count). After cleaning, the data is entered into MySQL according to the field structure of the application data repository.

[0049] APK installation package download and storage in the application data acquisition layer: Parse the application's APK download link, first verify the link's validity (returning a 200 status code and the file format is .apk); after successful verification, download the APK package via the Requests library, setting the download timeout to 30 seconds; if the download fails, record the reason for the failure and retry twice; after downloading, perform MD5 verification on the APK package; if it matches the MD5 value of a file already stored on the file server, it is determined to be a duplicate package and deleted to avoid redundant storage; if it is a new package, store it on the file server according to the hierarchical path, and record the package storage path, MD5 value, file size, etc. in MySQL.

[0050] Application data collection layer continuous monitoring and updating: Differentiated timed collection tasks are set for different levels of application markets: Level A mainstream application markets collect data once every 6 hours, and Level B small and medium-sized / personal application markets collect data once every 24 hours; each time data is collected, the newly collected data is compared with historical data, and information such as application version, listing status, and download volume is updated, and new and delisted applications are marked to achieve real-time monitoring of application listing status.

[0051] The application data acquisition layer handles anomalies by setting up an anomaly log database to record the application market / application that failed to collect data and the reasons for the failure (such as broken links, anti-crawling interception, server failure). Technical personnel can view the anomaly information through the log database and take targeted measures (such as adjusting crawler parameters or changing the acquisition interface).

[0052] The iterative optimization layer is the core module for realizing the closed loop of this invention. By statistically analyzing the working effects of the application market discovery layer and the data collection layer, it continuously optimizes the search keyword database and the application market judgment rule database, improves the discovery accuracy and retrieval efficiency of new application markets, and realizes the self-optimization and self-adaptation of the technical solution.

[0053] Iterative optimization layer performance metrics statistics: Setting a weekly iteration cycle, the system automatically calculates core performance metrics and stores the results in an optimization statistics database, providing data support for optimization efforts. (1) Rule matching accuracy = (Number of apps manually verified as valid / Number of apps automatically determined as valid) × 100%; (2) Keyword retrieval efficiency = (Number of effective application markets discovered for each keyword / Total number of search results for that keyword) × 100%; (3) Number of new app stores discovered in a single batch = Total number of new valid app stores added each week; (4) Success rate of data collection = (Number of successfully collected applications / Total number of applications in the application library) × 100%; Iterative optimization of the keyword database: Optimization based on keyword retrieval efficiency metrics. (1) Eliminate inefficient keywords with a retrieval efficiency of <1%; (2) For high-efficiency keywords with a retrieval efficiency > 5%, generate related keywords (such as "Android game download" to generate "mobile game installation package download" and "popular Android game acquisition") and add them to the keyword database; (3) Extract high-frequency, unindexed keywords from the webpage summaries of newly discovered app stores and add them to the keyword database to increase the probability of discovery in niche app stores; The iterative optimization layer optimizes the application market judgment rule base: based on the rule matching accuracy metric. (1) If a rule leads to a large number of misjudgments (such as rule 4 matching a virus download site), a sub-rule will be added to that rule (such as "APK download links must correspond to a legitimate domain name suffix, excluding non-legitimate domain names") to improve the accuracy of rule matching; (2) If a new type of application market (such as a pure mini-program distribution platform) is found not to be identified by the existing rules, new judgment rules (such as "containing a list of mini-program applications and supporting the download of mini-program installation packages") will be added to expand the scope of rule coverage; (3) If some rules are redundant (e.g., multiple rules have the same judgment effect), then the redundant rules are merged to improve the rule matching efficiency; Iterative optimization of collection parameters: Based on the collection success rate metric, fine-tuning parameters such as the number of concurrent coroutines, request interval, and timeout time in the collection layer are performed to adapt to the anti-crawling rules and server status of different application markets.

[0054] In this invention, MySQL can be replaced with PostgreSQL, and the FastDFS file server can be replaced with MinIO object storage without affecting data storage and retrieval functions.

[0055] In addition, the threshold substitution in this invention: the application market determination threshold can be adjusted to "≥1 webpage feature + ≥2 content features" to adapt to different application market feature determination requirements.

[0056] In this invention, the search engines Baidu, 360, and Bing can be replaced with search engines such as Sogou and Shenma that have public search APIs to achieve the same search effect.

[0057] This product offers alternatives to the standard sampling frequency of 6 hours / time for the mainstream market and 24 hours / time for niche markets. The sampling frequency can be adjusted to 4 / 8 hours or 12 / 48 hours to meet different monitoring needs.

[0058] The development environment for this invention is Python 3.8 or later. The core third-party libraries used (Scrapy, Requests, BeautifulSoup4, etc.) are all mainstream open-source libraries that can be publicly obtained and used. The search engines that are connected are all public platforms that use public search APIs without proprietary interfaces or permission restrictions. The keyword library and the judgment rule library can be flexibly added, deleted, and modified, and can be completed by those skilled in the art without creative effort. In the method of this invention, automation accounts for more than 95% of the operation, and only suspected application markets require simple manual review, without complex technical operations.

[0059] In actual testing, this invention can stably collect data from over 2000 app stores, achieving over 95% app listing monitoring, with a collection success rate of ≥90% and a rule matching accuracy rate of ≥95%. The crawler program of this invention can be configured with anti-crawling adaptation strategies such as request intervals and concurrency control to avoid IP blocking and maintain stable program operation.

[0060] The four-layer system of this invention forms an irreversible closed-loop working logic, specifically as follows: The basic data layer provides initial keywords and judgment rules for the application market discovery layer. The discovery layer discovers new application markets through search engines and adds them to the basic data layer. The application data acquisition layer collects application information and APK packages from the full application market of the basic data layer and stores the collection results back to the basic data layer. The iterative optimization layer extracts performance statistics from the discovery and collection layers from the basic data layer, optimizes the keyword library and judgment rule library, and synchronizes the optimized library to the application market discovery layer. Based on the optimized keywords and rules, the discovery layer conducts search engine searches again to discover more new application markets. This cycle repeats, achieving a closed loop of "collecting applications → discovering new application markets → collecting new applications → optimizing rules → discovering more application markets", continuously improving the coverage of application markets and application discovery capabilities.

[0061] The method for obtaining mobile application listing information based on network retrieval provided by this invention has the following beneficial technical effects: (1) It has the ability to proactively discover application markets and automatically collect new application markets through search engine retrieval and intelligent judgment, without the need for manual collection; (2) It has a wide coverage, covering more than 3,000 application markets and enabling monitoring of more than 95% of application listing scenarios; (3) Construct a closed-loop system of "discovery-collection-optimization", dynamically update the application market library, and continuously adapt to new application distribution channels; (4) High degree of automation: The entire process from application market discovery to application data collection is automated, with only a small number of suspected markets requiring manual review, which greatly reduces labor costs; (5) The application search results are comprehensive, and can obtain application listing information and APK installation packages from the full range of application markets, solving the problem of difficulty in finding non-mainframe applications.

[0062] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0063] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for obtaining mobile application listing information based on network retrieval, characterized in that, include: S1: Construct the basic data layer, which includes an initial application market library, a search keyword library, an application market judgment rule library, and an application data storage library. S2: Based on keywords in the keyword database, the application market proactive discovery layer calls the search engine to perform a search and performs an initial screening of the returned web page links to remove invalid and interfering links. At the same time, it calculates the search effectiveness and keyword search efficiency to provide data for subsequent iterative optimization. S3: Perform in-depth analysis on the web pages after initial screening to extract their structural and content features; S4: Match the parsed features with the preset rules in the application market determination rule base, and automatically determine the webpage as a valid application market, a suspected application market, or a non-application market according to the preset determination threshold. S5: Supplement the application market information that is determined to be valid into the initial application market library to update and form the full application market library; S6: The application data collection layer is based on the updated full application market library. It accesses the application list pages of each application market, collects the structured information of the applications and APK download links, and calculates the collection success rate. S7: Verify the validity of the collected APK download links, download the APK files that can be accessed normally, calculate the MD5 value of each APK file, compare it with the MD5 values ​​of APK files already stored in the application data repository to remove duplicates, and associate the deduplicated APK files with the corresponding application listing information to ensure file traceability; S8: The iterative optimization layer analyzes the retrieval efficiency, retrieval effectiveness, and performance indicators of the application market proactive discovery layer, as well as the collection success rate of the application data collection layer. Based on the threshold values ​​of each indicator, it optimizes the retrieval keyword library and the application market judgment rule library. At the same time, it adjusts the collection layer parameters according to the collection success rate and feeds the optimized library and parameters back to the corresponding level to achieve continuous optimization throughout the entire process.

2. The method according to claim 1, characterized in that, Step S2 includes: randomly selecting 5 to 10 keywords from the keyword database and concatenating them into a search statement; calling the search engine's public application programming interface to obtain the first 500 search results; and performing initial screening on the search results to remove duplicate Uniform Resource Locators and invalid links with status codes 404 or 500.

3. The method according to claim 1, characterized in that, In step S4, S = W + C S is the total score for rule matching, and its value ranges from [0, W-total + C-total]. W represents the number of successfully matched webpage feature rules, C represents the number of successfully matched content feature rules, W-total represents the total number of webpage feature rules, and C-total represents the total number of content feature rules. The app market type is determined based on the total rule matching score as follows: Effective application market: satisfying S≥3 and W≥2 and C≥1. Suspected app market: Satisfies S=1, Non-application market: satisfies S=0, ， For rule matching accuracy, The number of cases where the result in step S4 matches the actual number. This represents the total number of web pages involved in the judgment process in S4.

4. The method according to claim 1, characterized in that, The preset application market determination rule base in step S4 includes webpage feature rules and content feature rules; the determination threshold is set to determine a valid application market when at least two webpage feature rules and at least one content feature rule are met; if only one rule is met, it is marked as a suspected application market; if no rule is met, it is determined as a non-application market.

5. The method according to claim 4, characterized in that, Webpage feature rules include one or more of the following: the source code contains application download keywords, there is an application category navigation bar, and there is an application details page; and / or Content feature rules include one or more of the following: ≥30% of links are APK download addresses and non-social / e-commerce domains.

6. The method according to claim 1, characterized in that, Step S6 includes: classifying the application markets in the full application market library, setting different collection frequencies for mainstream application markets and niche application markets, allocating collection tasks using an asynchronous concurrent framework, performing differentiated timed collection for application markets of different levels, and cleaning and storing the collected application structured information in the application data storage repository.

7. The method according to claim 1, characterized in that, Step 7 includes: validating the collected APK download links; downloading the APK file after passing the verification; calculating the MD5 value of the downloaded file and comparing it with the MD5 value of the stored APK file; if they match, the file is identified as a duplicate and deleted; if they do not match, the file is stored on the file server according to a preset hierarchical path, and the storage information is recorded in the application data repository.

8. The method according to claim 1, characterized in that, The performance metrics in step S8 include rule matching accuracy, keyword retrieval efficiency, the number of new app stores discovered in a single batch, and the collection success rate. The keyword database is optimized based on keyword retrieval efficiency, removing inefficient keywords with retrieval efficiency below a first threshold, deriving related words for efficient keywords with retrieval efficiency above a second threshold, and extracting high-frequency words from the webpage summaries of newly discovered app stores to supplement the keyword database. The app store judgment rule base is optimized based on rule matching accuracy, adding sub-rules for rules that lead to misjudgments, supplementing new rules for new types of app stores not covered by existing rules, and merging redundant rules. The number of concurrent coroutines, request intervals, and timeout parameters of the collection layer are adjusted based on the collection success rate.

9. The method according to claim 1, characterized in that, In step S2, ， For search effectiveness, This refers to the number of valid webpage links in the search results after initial screening, i.e., the number of links after removing duplicates, 404 / 500 status codes, and interfering links from social media / e-commerce platforms. This represents the total number of original links returned by the search engine. ， To improve keyword retrieval efficiency, This refers to the number of valid webpage links in the search results after initial screening, i.e., the number of links after removing invalid and interfering links. This represents the total time from calling the search engine API to completing the initial screening of all results for a single search.

10. The method according to claim 9, characterized in that, In step S6, ， To increase the success rate of data collection, This refers to the number of application information items that have been collected, cleaned, and are suitable for storage before being uploaded to the app store. The target number of applications is preset for this data collection task.

11. The method according to claim 10, characterized in that, In step S8, To optimize performance, , These are the keyword retrieval efficiencies before and after optimization. , The results show the search effectiveness before and after optimization. , The data collection success rates are shown for the optimized and unoptimized versions, respectively.

Images