Data acquisition method and device, equipment and medium

Abstract

The invention discloses a data acquisition method and device, equipment and a medium. The method comprises the following steps: in response to a target login verification code picture uploaded after a distributed data acquisition node fails to locally analyze a login verification code of a to-be-acquired page, analyzing the target login verification code picture to obtain an identification result of the target login verification code picture; issuing an identification result of the target login verification code picture to the data acquisition node; wherein the data acquisition node is locally deployed with a basic OCR (Optical Character Recognition) model. According to the embodiment of the invention, the financial industry public data acquisition efficiency on the Internet can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data acquisition technology, and in particular to a data acquisition method, device, equipment and medium. Background Art

[0002] Banks need to regularly collect public data from the financial industry on the Internet to conduct their own business, such as the interest rates of time deposits and current deposits disclosed by third-party platforms, financial market analysis and macroeconomic data disclosed by some institutions, basic information of enterprises disclosed by official channels, financial reports and major event announcements disclosed by listed companies on the stock market, etc.

[0003] These public data are distributed on different websites or web pages on the Internet. The collection of these public data currently mainly relies on banks to arrange special employees to manually log into various websites and web pages to collect public data, which has low collection efficiency. Summary of the Invention

[0004] The present invention provides a data collection method, device, equipment and medium to improve the efficiency of collecting public data of the financial industry on the Internet.

[0005] According to one aspect of the present invention, there is provided a data collection method, comprising:

[0006] In response to the target login verification code picture uploaded by the distributed data collection node after the local parsing of the login verification code of the collection page fails, parsing the target login verification code picture to obtain a recognition result of the target login verification code picture;

[0007] Sending the recognition result of the target login verification code image to the data collection node;

[0008] Wherein, the data collection node is locally deployed with a basic OCR recognition model.

[0009] According to another aspect of the present invention, there is provided a data acquisition device, comprising:

[0010] The image parsing module is configured to, in response to a target login verification code image uploaded by a distributed data collection node after a local parsing failure of the login verification code of the collection page, parse the target login verification code image to obtain a recognition result of the target login verification code image;

[0011] An identification and sending module, configured to send the identification result of the target login verification code image to the data collection node;

[0012] Wherein, the data collection node is locally deployed with a basic OCR recognition model.

[0013] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the data collection method according to any embodiment of the present invention.

[0014] According to another aspect of the present invention, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data acquisition method described in any embodiment of the present invention.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data acquisition method according to any embodiment of the present invention when executed.

[0016] The distributed architecture of the embodiment of the present invention distributes collection tasks to multiple nodes for parallel processing, significantly improving the speed and efficiency of large-scale data collection and avoiding the single-point bottleneck problem. At the same time, through the hierarchical analysis of login verification codes, it ensures that the data collection system can handle complex authentication processes stably, further improving data collection efficiency.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 is a flow chart of a data collection method provided according to one embodiment of the present invention;

[0020] Figure 2 is a flow chart of a data collection method provided according to another embodiment of the present invention;

[0021] Figure 3 is a structural diagram of a data acquisition device provided according to another embodiment of the present invention;

[0022] Figure 4It is a schematic structural diagram of an electronic device implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first," "second," and the like in the present invention are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0025] Figure 1 This is a flow chart of a data collection method provided by one embodiment of the present invention. This embodiment is applicable to the case where public data of the financial industry required for business operations is collected through a distributed data collection system. The method can be executed by a data collection device, which can be implemented in the form of hardware and / or software. The device can be configured in an electronic device with corresponding data processing capabilities, such as a central management system in a distributed data collection system. Figure 1 As shown, the method includes:

[0026] S110 , in response to the target login verification code picture uploaded by the distributed data collection node after the local parsing of the login verification code of the collection page fails, the target login verification code picture is parsed to obtain a recognition result of the target login verification code picture.

[0027] S120: Send the recognition result of the target login verification code image to the data collection node.

[0028] Among them, the distributed data acquisition system consists of a central management system and multiple distributed data acquisition nodes.

[0029] Specifically, the verification code parsing module of the distributed data collection system adopts a hybrid architecture design, deploying login verification code parsing capabilities in layers according to complexity. Basic OCR recognition engines (such as locally trained CNN models) are integrated locally on the data collection node to process simple login verification codes (such as pure number / letter combinations). For complex login verification codes (such as graphic login verification codes with interference lines, slider verification, and behavioral verification), the data collection node uploads the login verification code image to the central management system. The central management system integrates deep learning models and third-party cloud service APIs to achieve high-concurrency processing through load balancing.

[0030] The central management system determines the pages to be collected that have public data in the financial industry and allow data collection, generates collection tasks corresponding to the pages to be collected, and sends them to the data collection nodes, so that the data collection nodes execute the collection tasks to collect the public data on the web pages to be collected.

[0031] Before collecting public data on the web page to be collected, the data collection node needs to pass the identity authentication of the web page to be collected. During the authentication process, in addition to entering the corresponding account and password, it is sometimes necessary to provide the verification code content of the login verification code on the page to be collected. For the login verification code, the data collection node first tries to use the local OCR engine to parse its content. If the parsing is successful, the authentication is completed and data collection is carried out. If the local parsing fails, the data collection node uses the image of the login verification code as the target login verification code image, encrypts it and uploads it to the central management system. The central management system selects a processing strategy based on the type of login verification code (such as calling a cloud API or triggering a machine learning model), and sends the parsing results to the data collection node. The data collection node completes the authentication based on the recognition results sent by the central management system and continues to collect public data on the web page to be collected.

[0032] After successful collection, the central processing unit cleans, dedupes, and stores the public data uploaded by the nodes to ensure data accuracy and integrity. Data is deduplicated using hashing algorithms or content fingerprinting techniques to avoid duplicate collection and storage. After cleaning, the data is stored in a database or data warehouse for subsequent analysis and use.

[0033] The embodiment of the present invention uses a distributed architecture to distribute collection tasks to multiple nodes for parallel processing, significantly improving the speed and efficiency of large-scale data collection and avoiding single-point bottleneck problems. At the same time, through the hierarchical analysis of login verification codes, it ensures that the data collection system can handle complex authentication processes stably, further improving data collection efficiency.

[0034] Figure 2 This is a flow chart of a data acquisition method provided by another embodiment of the present invention. This embodiment is optimized and improved on the basis of the above embodiment. Figure 2 As shown, the method includes:

[0035] S210 : In response to the target login verification code picture uploaded after the distributed data collection node fails to locally parse the login verification code of the collection page, determine whether there is a duplicate login verification code picture identical to the target login verification code picture in the cache.

[0036] S220. If there is no duplicate login verification code picture, perform deep analysis or manual intervention on the target login verification code picture to obtain a recognition result of the target login verification code picture; if there is a duplicate login verification code picture, determine the recognition result of the duplicate login verification code picture as the recognition result of the target login verification code picture.

[0037] Specifically, a cache space for login verification code images is deployed in the central management system, and the cache space stores the login verification code images that have been recently parsed successfully and have clear recognition results. For any target login verification code image uploaded by a data collection node, the system will place the target login verification code image in a queue, and regularly compare it with the login verification code images in the cache space to determine whether there are duplicate login verification code images that are identical to the target login verification code image in the cache space. If so, the recognition result of the duplicate login verification code image is directly determined as the recognition result of the target login verification code image, and the target login verification code image is no longer actually parsed. If not, the target login verification code image is deeply parsed or manually intervened to obtain the recognition result of the target login verification code image. By using the existing recognition results to the greatest extent possible, the processing pressure of the central management system can be reduced, which is especially effective when the number of data collection nodes is large.

[0038] S230: Send the recognition result of the target login verification code image to the data collection node.

[0039] Based on the above embodiment, optionally, performing deep analysis or manual intervention on the target login verification code image to obtain a recognition result of the target login verification code image includes:

[0040] If the login verification code picture is a self-processed login verification code picture, calling the advanced OCR recognition model to process the login verification code picture to obtain a recognition result of the login verification code picture;

[0041] If the login verification code picture is a manually processed login verification code picture, the login verification code picture is uploaded to the manual review platform, and the recognition result of the login verification code picture returned by the manual review platform is accepted.

[0042] Among them, the self-processed login verification code picture refers to the verification code picture that only contains numbers or simple letter combinations in the picture, and the manually processed verification code picture refers to the verification code picture that contains distorted characters, background interference lines or noise.

[0043] Specifically, the processing strategy is selected according to the type of login verification code. For manually processed login verification code images (such as dynamic login verification codes), the images are uploaded to a manual review platform (such as an internal annotation system). After the reviewer processes them through the web interface, the central processing system sends the recognition results manually given by the reviewer to the data collection node. For self-processed login verification code images, the system calls a local or third-party OCR recognition model to recognize the image, obtains the recognition results, and sends them to the data collection node. By distinguishing and processing login verification code images of different difficulty levels, orderly processing of login verification code images can be achieved, improving parsing efficiency.

[0044] Based on the above embodiment, optionally, the method further includes:

[0045] Obtaining a collection plan developed by a scheduling system, and splitting the collection plan into at least two collection tasks;

[0046] Allocating the at least two acquisition tasks to a data acquisition node;

[0047] During the process of the data collection node executing the collection task, if it is monitored that the data collection node is overloaded or abnormal, the collection task is reallocated or the load of the data collection node is reallocated.

[0048] Specifically, the intelligent scheduling system uses machine learning and rule engines to analyze features such as web page update frequency and content importance, generates a dynamic collection plan, and prioritizes pushing it to the central management system as the execution agent of the scheduling system, responsible for breaking down the collection plan into executable task units. The specific process is as follows: the central management system first divides the URL list of the target website in the collection plan by domain name through the URL grouping strategy, allocates the task volume based on the node's historical collection success rate, and then encapsulates each URL task into a JSON object containing metadata such as the target address, request header configuration, and parsing template ID, and implements secondary distribution through the partition binding mechanism of the stream processing platform message queue. After executing the collection task based on the subscribed collection task, the data collection node reports the results (HTML content, status code) to the dead letter queue of the message queue middleware system as an abnormal task, and feedbacks the load status (CPU, memory, queue length) to the central management system through the heartbeat protocol. Inter-system communication relies on a hybrid communication architecture: the scheduling system and the central management system transmit plans via an interface. The central management system and data collection nodes utilize the stream processing platform for task distribution and result transmission. The monitoring system collects node performance metrics in real time. When a node is detected to be overloaded or abnormal, the central management system dynamically migrates tasks through the service mesh or redistributes the load through the stream processing platform's consumer group rebalancing mechanism. Furthermore, the scheduling system can configure centrally updated collection policies (such as frequency limits). The central management system pushes policy changes to all nodes via a full-duplex communication protocol, enabling real-time policy synchronization and dynamic adjustment of task priorities. Distributed locks are used throughout the entire process to prevent conflicts caused by concurrent updates from multiple nodes. Consistent hashing algorithms are used to bind high-frequency websites to specific user agent groups, minimizing performance losses caused by frequent switching. This distributed architecture distributes collection tasks to multiple nodes for parallel processing, significantly improving the speed and efficiency of large-scale data collection, avoiding single-point bottlenecks, and enhancing data collection efficiency. The intelligent scheduling system dynamically adjusts collection policies based on factors such as web page update frequency, content importance, and node load, rationally allocating computing resources and network bandwidth to minimize resource waste.

[0049] Based on the above embodiment, optionally, a task scheduling node and a data coordination node are further present in the distributed data acquisition system where the data acquisition node is located.

[0050] Specifically, the distributed data collection system consists of three types of nodes working together: task scheduling nodes, data collection nodes, and data coordination nodes. The task scheduling nodes are responsible for splitting the collection tasks into URL queues and issuing instructions through message queues; the data collection nodes execute specific collection tasks, obtain web page content through HTTP requests, and then encapsulate the original HTML data, parsed structured data, and collection metadata (including response status code, collection timestamp, target website fingerprint, etc.) into a message body, which is uploaded to the central management system through a full-duplex communication protocol message queue. The data coordination nodes use distributed storage systems (such as object storage) for data persistence and synchronize node status through distributed coordination services. During system communication, the central management system caches and issues target URL whitelists and task priority parameters; at the same time, by monitoring the CPU utilization, network latency, and other operating indicators of each node, when a node overload is detected, the task allocation strategy is dynamically adjusted through the data center perception service discovery mechanism. Message queues act as duplex channels for communication: they achieve load balancing for task distribution through the stream processing platform's partitioning mechanism, while the message queue middleware system's confirmation mechanism ensures the reliable delivery of collection results. The distributed storage system ensures data consistency through version control, and unprocessed collection tasks can be recovered from the storage system when a node fails. Through a distributed architecture, collection tasks are distributed to multiple nodes for parallel processing, significantly improving the speed and efficiency of large-scale data collection and avoiding single-point bottlenecks. Furthermore, the distributed architecture and modular design enhance the system's fault tolerance and stability, reducing the impact of single-point failures on the overall system and ensuring the continuity and reliability of data collection tasks.

[0051] Based on the above embodiment, in response to the distributed data collection node failing to locally parse the login verification code of the collection page, before parsing the target login verification code image to obtain the recognition result of the target login verification code image, the method further includes:

[0052] For each page containing public data on the financial industry, obtain the data collection agreement and terms of service for that page;

[0053] If there is a data collection permission statement in the data collection agreement and terms of service, the page is determined as a page to be collected;

[0054] If the data collection agreement and terms of service do not contain a data collection permission statement, the page will be determined as a non-collectible page.

[0055] Specifically, the central management system retrieves the data collection agreement (robots.txt) and terms of service for each page, one by one, based on a user-provided list of pages containing public financial industry data. For each page, the system performs a keyword search within the data collection agreement and terms of service to determine whether a statement permitting data collection by computer programs exists. If so, the corresponding page is identified as a page to be collected, and the public financial industry data on it is subsequently collected through the data collection node. If not, the corresponding page is identified as a page that cannot be collected and feedback is provided to a human operator, who then reviews the public financial industry data on it.

[0056] Based on the above embodiment, optionally, the method further includes:

[0057] In response to the page information reported by the data collection node, record the relevant information of each collected page; the relevant information includes an increment identifier;

[0058] For each collected page, if the incremental identifier of the collected page changes or the content summary comparison fails, a complete collection task for the collected page is established;

[0059] If the incremental identifier of the collected page does not change or the content summary comparison passes, an incremental collection task for the collected page is established.

[0060] Specifically, after each successful collection, the central management system stores the page's incremental identifier (ETag and Last-Modified headers), content summary (e.g., SHA-256 hash), and collection timestamp in a distributed database as metadata for incremental collection. The central management system periodically triggers incremental check tasks through a scheduled task framework and generates a list of URLs to be checked based on the page's update frequency policy (e.g., hourly checks for high-priority pages and daily checks for low-priority pages). The incremental check module checks the target page's ETag or Last-Modified header using a lightweight HTTP request (retrieving only the response header). If the ETag or Last-Modified header of an already collected page changes or the content summary fails the comparison, a full collection task for the collected page is established. If the ETag or Last-Modified header of an already collected page remains unchanged or the content summary passes the comparison, an incremental collection task for the collected page is established. The central management system uses a machine learning model to analyze historical update patterns (e.g., a website updates every 30 minutes) and dynamically adjusts the check interval. It also uses rapid filtering to reduce invalid requests by identifying pages that have been determined to be non-updated. Throughout the entire process, the incremental check module in the central management system communicates with the nodes through the service grid to ensure the reliability and low latency of task distribution.

[0061] The embodiment of the present invention can reduce the processing pressure of the central management system by making maximum use of the existing recognition results, which is particularly effective when the number of data collection nodes is huge.

[0062] Figure 3 FIG. 1 is a structural diagram of a data acquisition device provided by another embodiment of the present invention. Figure 3 As shown, the device includes:

[0063] The image parsing module 310 is configured to, in response to a distributed data collection node failing to locally parse the login verification code of the collection page, parse the target login verification code image to obtain a recognition result of the target login verification code image;

[0064] The recognition and sending module 320 is used to send the recognition result of the target login verification code image to the data collection node;

[0065] Wherein, the data collection node is locally deployed with a basic OCR recognition model.

[0066] The data acquisition device provided by the embodiment of the present invention can execute the data acquisition method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0067] Optionally, the image parsing module 310 includes:

[0068] A duplicate checking unit, configured to determine whether there is a duplicate login verification code image identical to the target login verification code image in the cache;

[0069] An image parsing unit is configured to perform in-depth analysis or manual intervention on the target login verification code image if there is no duplicate login verification code image, to obtain a recognition result of the target login verification code image;

[0070] The result determination unit is configured to determine, if there is a duplicate login verification code picture, the recognition result of the duplicate login verification code picture as the recognition result of the target login verification code picture.

[0071] Optionally, the image parsing unit is specifically used to: if the login verification code image is a self-processed login verification code image, then call the advanced OCR recognition model to process the login verification code image to obtain the recognition result of the login verification code image; if the login verification code image is a manually processed login verification code image, then upload the login verification code image to the manual review platform, and accept the recognition result of the login verification code image returned by the manual review platform.

[0072] Optionally, the device also includes an incremental collection module, which is used to record relevant information of each collected page in response to page information reported by the data collection node; the relevant information includes an incremental identifier; for each collected page, if the incremental identifier of the collected page changes or the content summary comparison fails, a complete collection task for the collected page is established; if the incremental identifier of the collected page does not change or the content summary comparison passes, an incremental collection task for the collected page is established.

[0073] Optionally, the device also includes a dynamic scheduling module, which is used to obtain the collection plan formulated by the scheduling system and split the collection plan into at least two collection tasks; allocate the at least two collection tasks to the data collection node; during the process of the data collection node executing the collection task, if it is monitored that the data collection node is overloaded or abnormal, then reallocate the collection task or reallocate the load of the data collection node.

[0074] Optionally, the device also includes a page classification module, which is used to obtain the data collection agreement and service terms of each page containing public data of the financial industry; if there is a data collection permission statement in the data collection agreement and service terms, the page is determined as a page to be collected; if there is no data collection permission statement in the data collection agreement and service terms, the page is determined as a non-collectible page.

[0075] Optionally, the distributed data acquisition system where the data acquisition node is located also includes a task scheduling node and a data coordination node.

[0076] The data acquisition device further described can also execute the data acquisition method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0077] Figure 4 A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0078] like Figure 4As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., which is communicatively connected to the at least one processor 41. The memory stores a computer program that can be executed by the at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 into the random access memory (RAM) 43. Various programs and data required for the operation of the electronic device 40 can also be stored in the RAM 43. The processor 41, ROM 42, and RAM 43 are connected to each other via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0079] Multiple components in the electronic device 40 are connected to the I / O interface 45, including an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a magnetic disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0080] The processor 41 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors for running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 41 executes the various methods and processes described above, such as the data acquisition method.

[0081] In some embodiments, the data collection method can be implemented as a computer program that is tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the data collection method described above can be performed. Alternatively, in other embodiments, processor 41 can be configured to perform the data collection method in any other suitable manner (e.g., by means of firmware).

[0082] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0083] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0084] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0085] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0086] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0087] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0088] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0089] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data collection method, characterized in that: The method comprises: In response to the target login verification code picture uploaded by the distributed data collection node after the local parsing of the login verification code of the collection page fails, parsing the target login verification code picture to obtain a recognition result of the target login verification code picture; Sending the recognition result of the target login verification code image to the data collection node; Wherein, the data collection node is locally deployed with a basic OCR recognition model.

2. The method according to claim 1, characterized in that The parsing process of the target login verification code image to obtain a recognition result of the target login verification code image includes: Determine whether there is a duplicate login verification code image that is the same as the target login verification code image in the cache; If there is no duplicate login verification code picture, perform deep analysis or manual intervention on the target login verification code picture to obtain the recognition result of the target login verification code picture; If there is a duplicate login verification code picture, the recognition result of the duplicate login verification code picture is determined as the recognition result of the target login verification code picture.

3. The method according to claim 2, characterized in that The performing of deep analysis or manual intervention on the target login verification code image to obtain a recognition result of the target login verification code image includes: If the login verification code picture is a self-processed login verification code picture, calling the advanced OCR recognition model to process the login verification code picture to obtain a recognition result of the login verification code picture; If the login verification code picture is a manually processed login verification code picture, the login verification code picture is uploaded to the manual review platform, and the recognition result of the login verification code picture returned by the manual review platform is accepted.

4. The method according to claim 1, wherein The method further includes: in response to the distributed data collection node failing to locally parse the login verification code of the collection page and uploading a target login verification code image, before parsing the target login verification code image to obtain a recognition result of the target login verification code image; For each page containing public data on the financial industry, obtain the data collection agreement and terms of service for that page; If there is a data collection permission statement in the data collection agreement and terms of service, the page is determined as a page to be collected; If the data collection agreement and terms of service do not contain a data collection permission statement, the page will be determined as a non-collectible page.

5. The method according to claim 1, wherein The method further comprises: In response to the page information reported by the data collection node, record the relevant information of each collected page; the relevant information includes an increment identifier; For each collected page, if the incremental identifier of the collected page changes or the content summary comparison fails, a complete collection task for the collected page is established; If the incremental identifier of the collected page does not change or the content summary comparison passes, an incremental collection task for the collected page is established.

6. The method according to claim 1, wherein The method further comprises: Obtaining a collection plan developed by a scheduling system, and splitting the collection plan into at least two collection tasks; Allocating the at least two acquisition tasks to a data acquisition node; During the process of the data collection node executing the collection task, if it is monitored that the data collection node is overloaded or abnormal, the collection task is reallocated or the load of the data collection node is reallocated.

7. The method according to claim 1, characterized in that The distributed data acquisition system where the data acquisition node is located also includes a task scheduling node and a data coordination node.

8. A data acquisition device, characterized in that: The device comprises: The image parsing module is configured to, in response to a target login verification code image uploaded by a distributed data collection node after a local parsing failure of the login verification code of the collection page, parse the target login verification code image to obtain a recognition result of the target login verification code image; An identification and sending module, configured to send the identification result of the target login verification code image to the data collection node; Wherein, the data collection node is locally deployed with a basic OCR recognition model.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the data collection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data collection method according to any one of claims 1 to 7 when executed.

Images