Data processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a data processing method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of computers, in particular to the technical field related to data processing. The method comprises the following steps: acquiring to-be-processed data streams in various preset data formats based on a preset data acquisition mode; wherein the preset data acquisition mode comprises a real-time data acquisition mode and an offline data acquisition mode; caching the data streams in various preset data formats into a target message queue; streaming the data streams cached in the target message queue, and analyzing the data streams in a plurality of preset data formats into data streams in a target data format to obtain streaming-processed data streams; and performing persistence processing on the data stream subjected to streaming processing to obtain data subjected to persistence processing. According to the method, the integrity and effectiveness of data processing can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of data processing related technology, and more particularly to a data processing method, a data processing device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of Internet business, the black industry chain is increasingly developing and spreading. It uses Internet technology to carry out network attacks, steal information and other network violations, which poses a challenge to the implementation of anti-fraud business. On the other hand, it also promotes the rapid development of the anti-fraud field. As the core asset of Internet business, the quality of data plays a vital role in the development of related businesses in the anti-fraud field (such as risk control-related businesses). Summary of the invention

[0003] The embodiments of the present disclosure provide a data processing method, a data processing device, an electronic device, a computer-readable storage medium, and a computer program product. The data processing solution provided by the embodiments of the present disclosure can improve the integrity and effectiveness of data processing.

[0004] In a first aspect, an embodiment of the present disclosure proposes a data processing method, comprising: based on a preset data collection method, obtaining data streams in multiple preset data formats to be processed; wherein the preset data collection method includes a real-time data collection method and an offline data collection method; caching the data streams in multiple preset data formats into a target message queue; streaming the data streams cached in the target message queue, parsing the data streams in multiple preset data formats into data streams in a target data format, and obtaining a streamed data stream; and persisting the streamed data stream to obtain persisted data.

[0005] In the second aspect, the embodiment of the present disclosure proposes a data processing device, including: a collection module, a storage module, a parsing module and a processing module. Among them, the collection module is configured to obtain data streams in multiple preset data formats to be processed based on a preset data collection method; and the preset data collection method includes a real-time data collection method and an offline data collection method; the storage module is configured to cache data streams in multiple preset data formats into a target message queue; the parsing module is configured to stream the data streams cached in the target message queue, parse the data streams in multiple preset data formats into a data stream in a target data format, and obtain a streamed data stream; the processing module is configured to perform persistence processing on the streamed data stream to obtain persistent data.

[0006] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the data processing method described in any of the above implementation methods when executing.

[0007] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the data processing method described in any of the above implementations when executed.

[0008] In a fifth aspect, an embodiment of the present disclosure provides a computer program product including a computer program, which, when executed by a processor, can implement the steps of the data processing method described in any of the above implementations.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0011] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0012] Figure 2 A flowchart of a data processing method provided by an embodiment of the present disclosure;

[0013] Figure 3 A flowchart of another data processing method provided by an embodiment of the present disclosure;

[0014] Figure 4 An overall system framework diagram for performing data processing in an application scenario provided by an embodiment of the present disclosure;

[0015] Figure 5 A structural block diagram of a data processing device provided in an embodiment of the present disclosure;

[0016] Figure 6 A schematic diagram of the structure of an electronic device suitable for executing a data processing method provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.

[0018] It should be pointed out that in the technical solution of the present invention, the collection, acquisition, storage, processing, transmission, provision, disclosure and application of user personal information (such as file sharing records, etc.) are all carried out with the user's knowledge and explicit authorization, comply with the relevant laws and regulations, and do not violate public order and good morals.

[0019] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the data processing scheme of the present disclosure may be applied.

[0020] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0021] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The terminal devices 101, 102, 103 and the server 105 can be installed with various applications for realizing information communication between the two, such as data processing applications, data engine applications, instant messaging applications, etc.

[0022] Terminal devices 101, 102, 103 and server 105 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.; when terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server; when the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.

[0023] The server 105 can provide various services through various built-in applications. Taking the data processing application as an example, the server 105 can achieve the following effects when running the data processing application: at least a plurality of data streams in different data formats to be processed can be obtained through real-time data collection and offline data collection, and the obtained data streams in various preset data formats need to be buffered in the corresponding message queue, i.e., the target message queue, so as to realize hybrid data transmission of real-time data streams and offline data streams synchronously. Furthermore, the real-time and offline data streams in various data formats cached in the message queue can be unified or normalized in data format by streaming processing, so as to parse the data streams in various preset data formats into data streams in the target data format required for subsequent persistence processing and other processes, thereby obtaining a stream-processed data stream for persistence processing, so as to facilitate data storage and call, and improve the integrity and effectiveness of data processing.

[0024] It should be noted that the data streams of various data formats acquired based on the corresponding data acquisition methods can be acquired from the terminal devices 101, 102, 103 through the network 104, can also be acquired from other servers (not shown in the figure) through the network 104, and can also be pre-stored in the server 105 in various ways. Therefore, when the server 105 detects that the data has been stored locally (for example, the data corresponding to the data processing task to be executed that is retained before the processing starts), it can choose to directly acquire the data locally. In this case, the exemplary system architecture 100 may also not include the terminal devices 101, 102, 103 and the network 104.

[0025] Since the data processing performed on the data streams of various data formats acquired based on the corresponding data collection method requires more computing resources and stronger computing power, the data processing methods provided in the subsequent embodiments of the present disclosure are generally performed by the server 105 with stronger computing power and more computing resources, and accordingly, the data processing device is generally also set in the server 105. However, it should also be pointed out that when the terminal devices 101, 102, and 103 also have computing power and computing resources that meet the requirements, the terminal devices 101, 102, and 103 can also complete the above-mentioned various operations that are handed over to the server 105 through the data processing applications installed thereon, and then output the same results as the server 105. In particular, when there are multiple terminal devices with different computing powers at the same time, but the data processing application determines that the terminal device where it is located has stronger computing power and more remaining computing resources, it can allow the terminal device to perform the above-mentioned operations, thereby appropriately reducing the computing pressure of the server 105, and accordingly, the data processing device can also be set in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104 .

[0026] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0027] Please refer to Figure 2 , Figure 2 This is a flow chart of a data processing method provided in an embodiment of the present disclosure. The data processing method is executed by an execution subject server, wherein process 200 includes the following steps:

[0028] Step 201: Based on a preset data collection method, a plurality of data streams in preset data formats to be processed are obtained; wherein the preset data collection method includes a real-time data collection method and an offline data collection method.

[0029] This step is intended to be performed by the above-mentioned execution subject of the data processing method (for example Figure 1 The server 105 shown in the figure can at least adopt a real-time data collection method and an offline data collection method to obtain the data stream to be processed, wherein the obtained data stream can correspond to a variety of different preset data formats.

[0030] In some optional implementations of the embodiments of the present disclosure, the above-mentioned multiple preset data formats may include but are not limited to at least two of the following: scripting language object notation JSON format (a lightweight data exchange format), protocol buffer (protobuf) format (a flexible, efficient, and automated structured data format) and text format. The specific settings of the multiple preset data formats can be flexibly configured according to actual needs, while ensuring the diversity of data formats that can be processed, and can meet the processing requirements for data in different data formats. Among them, the data streams of the above-mentioned multiple preset data formats to be processed can be obtained from at least one data source based on the above-mentioned predictive data collection method, and each data source corresponds to one or more of the multiple preset data formats.

[0031] Step 202: Cache data streams in multiple preset data formats into a target message queue.

[0032] On the basis of step 201, this step aims to cache the data streams to the corresponding message queues after the execution subject obtains the data streams in multiple preset data formats and before further processing, which helps to decouple the data collection and data cleaning related processes. Among them, the number of target message queues can be one or more, which can be flexibly set according to the correspondence between different data formats and message queues and the number of data streams.

[0033] Step 203: Stream-process the data stream cached in the target message queue, parse the data streams in multiple preset data formats into a data stream in the target data format, and obtain a stream-processed data stream.

[0034] Based on step 202, this step aims to consume the real-time data streams and offline data streams of various data formats cached in the target message queue by the above-mentioned execution subject through streaming processing, so as to realize the normalization or unified processing of the data format of the data stream, so as to provide a unified processing method for data sources of different types or formats. Among them, the above-mentioned target data format can include JSON format.

[0035] Step 204: Perform persistence processing on the stream-processed data flow to obtain persistently processed data.

[0036] On the basis of step 203, this step aims to perform persistence processing on the streaming data stream by the above-mentioned execution subject, so as to persist the corresponding data into persistent data for long-term storage, so as to facilitate further subsequent data applications.

[0037] The data processing method provided by the embodiment of the present disclosure can at least obtain data streams of multiple different data formats to be processed through real-time data collection methods and offline data collection methods, and needs to buffer the obtained data streams of multiple preset data formats into corresponding message queues, namely, target message queues, so as to realize hybrid data transmission of real-time data streams and offline data streams synchronously. Furthermore, the real-time and offline data streams of multiple data formats cached in the message queue can be unified or normalized in data formats by streaming processing, so as to parse the data streams of multiple preset data formats into data streams of the target data format required for subsequent persistence processing and other processes, thereby obtaining a data stream that has been streamed for persistence processing, so as to facilitate data storage and calling, and improve the integrity and effectiveness of data processing.

[0038] Please refer to Figure 3 , Figure 3 This is a flowchart of another data processing method provided by an embodiment of the present disclosure. The data processing method is executed by an execution subject server, wherein process 300 includes the following steps:

[0039] Step 301: Based on a preset data collection method, a plurality of data streams in preset data formats to be processed are obtained; wherein the preset data collection method includes a real-time data collection method and an offline data collection method.

[0040] Step 302: Cache data streams in multiple preset data formats into a target message queue.

[0041] Step 303: Stream-process the data stream cached in the target message queue, parse the data streams in multiple preset data formats into a data stream in the target data format, and obtain a stream-processed data stream.

[0042] Step 304: Perform persistence processing on the stream-processed data flow to obtain persistently processed data.

[0043] The above steps 301-304 are similar to Figure 2 Steps 201-204 shown are consistent, and for the same contents, please refer to the corresponding parts of the previous embodiment, which will not be repeated here.

[0044] Step 305: In response to receiving the risk assessment request, obtaining the to-be-assessed data corresponding to the risk assessment request from the persisted data.

[0045] Step 306: Based on the risk assessment strategy corresponding to the risk assessment request, determine the risk information corresponding to the data to be assessed.

[0046] In the embodiments of the present disclosure, for the acquired data streams in multiple preset data formats, the persisted data obtained after a series of processing processes such as the above-mentioned caching, format conversion and persistence processing can be used to perform corresponding risk assessments according to specific risk assessment needs. Specifically, the data to be evaluated that matches the current risk assessment request can be obtained from the persisted data, and then the corresponding risk information can be determined in combination with the corresponding risk assessment strategy, which can provide support for the formulation of strategies for anti-fraud fields, such as risk control-related businesses.

[0047] In some optional implementations of the embodiments of the present disclosure, the above step 306 can be further specifically performed as follows: extracting features related to risk assessment of the data to be assessed; processing the features related to risk assessment based on the risk assessment strategy to obtain risk information corresponding to the data to be assessed.

[0048] In this embodiment, features related to risk assessment can be extracted based on the data to be assessed, so that the extracted features can be processed based on the corresponding risk assessment strategy, and the risk information corresponding to the data to be assessed can be obtained to provide reliable support for the formulation of strategies for risk control-related businesses in the field of anti-fraud, such as Figure 4 The execution link 404 shown in the above can realize the execution of policy tasks, real-time interception of risks, corresponding model scoring, risk recall, and report output related to risk assessment results or risk recall results, etc., based on the risk information, so as to open up a set of risk control data engines such as feature calculation, real-time interception, and quasi-real-time recall in the field of anti-cheating. Further, risk identification and / or risk control can be performed based on the risk information obtained above. Risk identification can refer to the risk level used to identify the relevant behavior implemented by the user in the corresponding activity, and risk control can refer to the risk level adjusted according to the set indicators corresponding to the activity. Among them, the business related to risk control can correspond to a variety of scenarios, such as the risk control scenario for cheating behavior in the advertising push business, the risk control scenario of asset management, the risk control scenario of coal mine safety management, and the risk control scenario of credit. Further, the above risk assessment request can be associated with a specific business.

[0049] In some optional implementations of the embodiments of the present disclosure, in order to accurately and efficiently extract features related to risk assessment from the data to be assessed, feature extraction can be performed by inputting the data to be assessed into a pre-trained corresponding feature extraction model. The feature extraction model can be trained on a training set constructed based on sample historical data related to risk assessment of the corresponding business as input and corresponding real risk information as expected output.

[0050] In some optional implementations of the embodiments of the present disclosure, the above-mentioned risk assessment request, data to be assessed, risk assessment strategy, risk information, etc. may be associated with specific businesses, and may be set according to specific circumstances or needs, which are not specifically limited here; and the risk assessment strategy may be a risk assessment model, such as a classification model, used to obtain the risk type, or may be other forms, such as rules, etc., which may be set according to specific circumstances or needs, which are not specifically limited here. For example, the data to be assessed and the sample historical data may respectively include data related to the user's identification (such as IP address, login account information used, account registration time, etc.), and data related to the user's behavior (such as click behavior, sharing behavior (including cheating behavior based on the user's non-real intentions), behavior of stealing user information, etc.). Among them, the model to be trained used to train the feature extraction model may be a deep learning model, etc.

[0051] In order to deepen the understanding of the data processing method in any of the above disclosed embodiments, Figure 2 Corresponding disclosed embodiments and Figure 3 Based on the corresponding disclosed embodiments, the above step 201 or step 301 can be specifically performed as follows:

[0052] In response to the correspondence between data streams in multiple preset data formats and real-time incremental data, data streams in multiple preset data formats are obtained based on the real-time data collection method; in response to the correspondence between data streams in multiple preset data formats and historical static data, data streams in multiple preset data formats are obtained based on the offline data collection method.

[0053] In this embodiment, for different types of business data, corresponding data collection methods can be used to acquire data streams to ensure the integrity of the acquired data to be processed. Figure 4In the data collection link 401 shown, for data that needs to be transmitted in real time with incremental synchronization, the data streams of various preset data formats to be processed can be obtained from the storage data of data sources such as business instance logs and databases (DB). For example, the data stream can be obtained from the DB in an incremental subscription manner. Specifically, the master-slave replication can be realized by simulating the database slave interaction protocol, disguising as a slave node, sending a transfer or dump protocol to the master node, and incrementally monitoring the binlog (which can refer to a binary log that records all database table structure changes and table data modifications), and then converting it into a data stream; for the log files on the business instance (which can be divided into text format and proto format), the agent of the collection process buried in the business instance can be used to perform real-time incremental collection on the specified cluster, specified instance, and specified log file, such as through the corresponding log collection component, and then converted into a data stream after parsing. For historical static data that needs full synchronization, the data streams of various preset data formats to be processed can be obtained from the storage data of data sources such as data warehouses and file systems, and can be processed offline into data streams by parsing files.

[0054] In order to deepen the understanding of the data processing method in any of the above disclosed embodiments, Figure 2 Corresponding disclosed embodiments and Figure 3 On the basis of the corresponding disclosed embodiments, the process of caching data streams of multiple preset data formats into the target message queue in the above step 202 or step 302 can be specifically executed as follows: determining the target business topic corresponding to the data streams of multiple preset data formats; based on the correspondence between the pre-configured business topics and the message queues, determining the target message queue corresponding to the target business topic, and caching the data streams of multiple preset data formats into the target message queue.

[0055] In this embodiment, by pre-establishing or configuring the corresponding relationship between the business to which the data belongs and the message queue for storing the data, the data streams of each preset data format belonging to the corresponding business can be accurately allocated to the corresponding message queue for caching, such as Figure 4The data transmission link 402 shown, for example, the data stream cached in the message queue 1 can be the log data of the device fingerprint, and the data stream cached in the message queue 2 can be the record data of the online policy interception, etc. In this way, data temporary storage of different business and queues is realized, so as to support the data consumption mode of multiple message queues and different business themes, so as to meet the diversified consumption processing based on the business in the application scenario, and at the same time, the decoupling of data collection and data consumption can be realized, and the corresponding relationship can be flexibly and quickly expanded and reduced, and the number of message queues can be added or deleted for the corresponding data streams, so as to realize dynamic resource adjustment. Among them, the correspondence between the above-mentioned pre-configured business themes and message queues can be flexibly configured according to the specific situation, which is not specifically limited here, and the correspondence can be flexibly adjusted according to the increase or decrease of business themes to apply to more application scenarios.

[0056] In some optional implementations of the embodiments of the present disclosure, the correspondence between the above-mentioned pre-configured business topics and message queues includes a many-to-one relationship.

[0057] In this embodiment, data corresponding to different business topics can be temporarily stored in the same message queue, so that the storage resources of the message queue can be fully utilized. Figure 4 The data transmission link 402 shown can temporarily store the data stream collected in the data collection link into the corresponding message queue according to the corresponding business topic. For example, the corresponding relationship between the business topic and the message queue can be a 2-to-1 relationship, with topics 1 and 3 corresponding to message queue 1, and topic 2 and N (a positive integer greater than 3) corresponding to message queue 2. The data transmission link assumes the network transmission function from the data source to the downstream instance.

[0058] It should be noted that in some other optional implementation methods of the embodiments of the present disclosure, in order to achieve effective temporary storage of data for some business topics with large data volumes, the correspondence between the above-mentioned pre-configured business topics and message queues can also be a one-to-many relationship, that is, the data corresponding to the same business topic can be temporarily stored in different message queues.

[0059] In order to deepen the understanding of the data processing method in any of the above disclosed embodiments, Figure 2 Corresponding disclosed embodiments and Figure 3On the basis of the corresponding disclosed embodiments, the process of parsing data streams of multiple preset data formats into data streams of target data formats in the above step 203 or step 303 can be specifically executed as follows: based on the correspondence between the pre-configured data formats and the configuration information, determining the format conversion configuration information corresponding to the multiple preset data formats respectively; wherein the format conversion configuration information includes at least one of the following: type indication parameters of multiple preset data formats, message queue key values, and type indication parameters of the target data format; based on the format conversion configuration information, parsing the data streams of multiple preset data formats into data streams of the target data format.

[0060] In this embodiment, when streaming or consuming the data stream buffered in the message queue, the data formats of data streams in different data formats can be normalized or unified. Specifically, based on the correspondence between the pre-configured data format and the configuration information, the format conversion configuration information corresponding to the data format of the current data stream to be consumed can be first determined, and the data stream in the preset data format can be accurately parsed into a data stream in the target data format. In this way, it is possible to achieve normalization of different data formats through flexible configuration, achieve unification of encoding formats, and thus improve the efficiency of data processing. Among them, the corresponding configuration can be flexibly matched for different data formats, and the correspondence between the data format and the configuration information can be flexibly configured according to the specific situation, which is not specifically limited here.

[0061] Among them, the above-mentioned format conversion configuration information may include but is not limited to at least one of the following: type indication parameters of multiple preset data formats, message queue key values ​​and type indication parameters of the target data format; thus, based on the configuration, the message queue where the data stream of the preset data format indicated by the type indication parameters of the multiple preset data formats is located can be obtained (that is, it can be determined by the message queue key value), and it can be further known that the data stream of the preset data format temporarily stored in the message queue can be converted into a data stream of the target data format.

[0062] In some optional implementations of the embodiments of the present disclosure, in specific implementations, the data stream to be subjected to data format conversion may also be indicated by the above-mentioned configuration information, for example, the identifier (ID) and / or name of the data stream may be configured.

[0063] In some optional implementations of the embodiments of the present disclosure, the process of streaming the data stream cached in the target message queue may also include: before format conversion of the data streams in the above-mentioned multiple preset formats, service authentication and / or message filtering may be performed on the data streams in the multiple preset formats to reduce the probability of subsequent data format consistency processing failure or delay.

[0064] In some optional implementations of the embodiments of the present disclosure, the process of streaming the data stream cached in the target message queue may also include: mapping, extracting, and assembling the input data stream according to the pre-configured structure and expression (such as regular expression, etc.) after the format normalization process, so as to complete further processing of the data. Among them, the configuration involved in the process can be completed in the cloud, and can support minute-level hot updates.

[0065] In order to deepen the understanding of the data processing method in any of the above disclosed embodiments, Figure 2 Corresponding disclosed embodiments and Figure 3 Based on the corresponding disclosed embodiments, the process of persisting the streamed data stream in the above step 204 or step 304 can be specifically executed as follows: determining a storage agent instance corresponding to a preset storage type; and based on the storage agent instance, persisting the streamed data stream.

[0066] In this embodiment, the data stream that has at least been processed by format normalization can be persistently stored in the corresponding storage proxy instance, and the processed data stream is persisted in a streaming manner to facilitate subsequent data storage and call. Among them, the preset storage type of the corresponding storage proxy instance can be pre-configured, and the preset storage type can include but is not limited to: MySQL (a relational database management system that can store data in different tables), Bigquery (a large-scale enterprise data warehouse that can process and analyze TB-level data in seconds), Palo (a parallel-oriented, analysis-oriented database system), Remote Dictionary Server (Remote Dictionary Server, Redis; a log-type, Key-Value database that can be based on memory and can be persistent), Hive (a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading), files, etc.

[0067] In some optional implementations of the embodiments of the present disclosure, the configuration information in the correspondence between the pre-configured data format and the configuration information may include at least the above-mentioned format conversion configuration information, the identifier (and / or name) of the data stream, and the preset storage type. That is to say, a uniform correspondence between different data formats and the corresponding configuration information is established in advance. When it is necessary to process data in the corresponding data format, the correspondence can be directly called to obtain the configuration information matching the corresponding data format, so as to efficiently complete the corresponding operation.

[0068] Further, in some optional implementations of the embodiments of the present disclosure, in response to the correspondence between the pre-configured data format and the configuration information including the configuration information matching the above-mentioned multiple preset data formats, the above-mentioned corresponding caching to the target message queue, streaming processing, persistence processing and other related operations are performed. For details, please refer to the above-mentioned Figure 2 and Figure 3 The description of the same parts in the corresponding embodiments will not be repeated here; and in response to the correspondence between the above-mentioned pre-configured data format and configuration information not including configuration information matching the above-mentioned multiple preset data formats, the data processing method in any of the above-mentioned embodiments may also include: setting corresponding configuration information for the new data format, and establishing a corresponding step between the two, to achieve flexible processing between each data format and the corresponding configuration information, so that the corresponding configuration information can be directly called for the newly added data format for use in the future, which is efficient and convenient.

[0069] In the embodiments of the present disclosure, Figure 4 The data stream processing link 403 shown can be completed by multiple distributed CPU instance clusters to complete a series of processing such as service authentication, message filtering, data format normalization, processing, and persistence of the data stream stored in the message queue. Specifically, by subscribing to a specified message queue node, the association between each instance and the message queue can be established, and the data stream message can be consumed in a streaming manner to extract, clean, process, and persist the data. Furthermore, in the whole process, possible data loss, runtime anomalies, etc. can also be recorded and alarmed, which is convenient for manual backtracking. Among them, the cluster corresponding to each message queue is configured in advance.

[0070] In some optional implementations of the embodiments of the present disclosure, the streaming processing of the data stream stored in the target message queue can be deployed in different computer rooms and clusters, and the resources between different data streams are isolated from each other. The cluster can be deployed based on cloud native, so that resource warning, automatic circuit breaking, dynamic expansion and contraction, etc. can be achieved to ensure the high availability of the overall service. Figure 4 As shown, it is divided into computer rooms 1 and 2 to achieve physical isolation of resources. Corresponding instance clusters are deployed in different computer rooms. For example, computer room 1 deploys instances 1 and 2, and computer room 2 deploys instances 3-5.

[0071] In summary, the disclosed embodiments provide a high-throughput, high-availability, and scalable data engine architecture for anti-cheating scenarios. For new data stream transmission and risk control policy configuration, users can achieve low-cost configurable access, with a single message data collection and storage delay of seconds, and a single data stream daily throughput of tens of billions. In this way, it can be widely used in Internet risk control-related and other business fields.

[0072] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a data processing device 500. Figure 2-3 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0073] like Figure 5 As shown, the data processing device 500 of the embodiment of the present disclosure may include: a collection module 501, a storage module 502, a parsing module 503 and a processing module 504. The collection module 501 is configured to obtain data streams in multiple preset data formats to be processed based on a preset data collection method; and the preset data collection method includes a real-time data collection method and an offline data collection method; the storage module 502 is configured to cache the data streams in multiple preset data formats into a target message queue; the parsing module 503 is configured to stream the data streams cached in the target message queue, parse the data streams in multiple preset data formats into a data stream in a target data format, and obtain a streamed data stream; the processing module 504 is configured to perform persistence processing on the streamed data stream to obtain persistently processed data.

[0074] In the data processing device 500 of the present disclosure, the specific processing of the acquisition module 501, the storage module 502, the analysis module 503 and the processing module 504 and the technical effects thereof can be referred to in Figure 2 Steps 201-204 in the corresponding embodiment may refer to Figure 3 The relevant descriptions of steps 301 - 304 in the corresponding embodiment are not repeated here.

[0075] In some optional implementations of the embodiments of the present disclosure, the above-mentioned acquisition module 501 is further specifically configured as: in response to the correspondence between data streams in multiple preset data formats and real-time incremental data, obtaining data streams in multiple preset data formats based on a real-time data acquisition method; in response to the correspondence between data streams in multiple preset data formats and historical static data, obtaining data streams in multiple preset data formats based on an offline data acquisition method.

[0076] In some optional implementations of the embodiments of the present disclosure, the above-mentioned storage module 502 is further specifically configured to: determine the target business topic corresponding to the data streams in multiple preset data formats; based on the correspondence between the pre-configured business topics and the message queues, determine the target message queue corresponding to the target business topic, and cache the data streams in multiple preset data formats into the target message queue.

[0077] In some optional implementations of the embodiments of the present disclosure, the correspondence between the above-mentioned pre-configured business topics and message queues includes a many-to-one relationship.

[0078] In some optional implementations of the embodiments of the present disclosure, the above-mentioned parsing module 503 is further specifically configured to: determine format conversion configuration information corresponding to multiple preset data formats respectively based on the correspondence between pre-configured data formats and configuration information; wherein the format conversion configuration information includes at least one of the following: type indication parameters of multiple preset data formats, message queue key values ​​and type indication parameters of the target data format; based on the format conversion configuration information, parse the data streams of multiple preset data formats into data streams of the target data format.

[0079] In some optional implementations of the embodiments of the present disclosure, the processing module 504 is specifically configured to: determine a storage proxy instance corresponding to a preset storage type; and perform persistence processing on the streamed data stream based on the storage proxy instance.

[0080] In some optional implementations of the embodiments of the present disclosure, the above-mentioned multiple preset data formats include at least two of the following: script language object notation format, protocol buffer format and text format.

[0081] In some optional implementations of the embodiments of the present disclosure, the data processing device 500 in the embodiments of the present disclosure may further include an acquisition module 505 and an evaluation module 506. The acquisition module 505 is configured to, in response to receiving a risk assessment request, acquire the data to be evaluated corresponding to the risk assessment request from the persistently processed data; the evaluation module 506 is configured to determine the risk information corresponding to the data to be evaluated based on the risk assessment strategy corresponding to the risk assessment request.

[0082] In the data processing device 500 of the present disclosure, the specific processing of the acquisition module 505 and the evaluation module 506 and the technical effects thereof can be referred to in Figure 3 The relevant descriptions of steps 305-306 in the corresponding embodiment are not repeated here.

[0083] In some optional implementations of the embodiments of the present disclosure, the above-mentioned evaluation module 506 is further specifically configured to: extract features related to risk assessment of the data to be evaluated; process the features related to risk assessment based on the risk assessment strategy to obtain risk information corresponding to the data to be evaluated.

[0084] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment, and can at least obtain data streams of multiple different data formats to be processed through real-time data collection methods and offline data collection methods, and needs to buffer the obtained data streams of multiple preset data formats into corresponding message queues, namely, target message queues, so as to realize hybrid data transmission of real-time data streams and offline data streams synchronously. Further, the real-time and offline data streams of multiple data formats cached in the message queue can be unified or normalized in data formats by streaming processing, so as to parse the data streams of multiple preset data formats into data streams of the target data format required for subsequent persistence processing and other processes, thereby obtaining a data stream that has been streamed for persistence processing, so as to facilitate data storage and calling, and improve the integrity and effectiveness of data processing.

[0085] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the data processing method described in any of the above embodiments when executing.

[0086] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the data processing method described in any of the above embodiments when executed.

[0087] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, including a computer program, which can implement the data processing method described in any of the above embodiments when executed by a processor.

[0088] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device 600 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 600 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0089] like Figure 6As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0090] Multiple components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0091] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).

[0092] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0093] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0094] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0095] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0096] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0097] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain. Among them, a cloud server, also known as a cloud computing server or a cloud host, is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS, Virtual Private Server) services.

[0098] According to the data processing scheme of the embodiment of the present disclosure, at least a plurality of data streams in different data formats to be processed can be obtained through real-time data collection and offline data collection, and the obtained data streams in a plurality of preset data formats need to be buffered in a corresponding message queue, i.e., a target message queue, so as to realize hybrid data transmission of synchronization of real-time data streams and offline data streams. Furthermore, the real-time and offline data streams in a plurality of data formats cached in the message queue can be unified or normalized in data formats by using streaming processing, so as to parse the data streams in a plurality of preset data formats into data streams in a data format required for subsequent persistence processing and the like, i.e., a target data format, so as to obtain a data stream after streaming processing for persistence processing, so as to facilitate data storage and calling, and improve the integrity and effectiveness of data processing.

[0099] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0100] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data processing method, comprising: Based on a preset data collection method, a data stream in a plurality of preset data formats to be processed is obtained; wherein the preset data collection method includes a real-time data collection method and an offline data collection method; Cache the data streams in the plurality of preset data formats into a target message queue; Stream-processing the data stream cached in the target message queue, parsing the data streams in the plurality of preset data formats into data streams in the target data format, and obtaining a stream-processed data stream; The data stream that has undergone streaming processing is persistently processed to obtain persistently processed data.

2. The method according to claim 1, wherein: The method of acquiring data streams in multiple preset data formats to be processed based on the preset data collection method includes: In response to the data streams in the plurality of preset data formats corresponding to the real-time incremental data, acquiring the data streams in the plurality of preset data formats based on the real-time data acquisition method; In response to the data streams in the plurality of preset data formats corresponding to the historical static data, the data streams in the plurality of preset data formats are acquired based on the offline data collection method.

3. The method according to claim 1, wherein: The step of caching the data streams in the plurality of preset data formats into a target message queue includes: Determine target business topics corresponding to the data streams in the plurality of preset data formats; Based on the correspondence between the pre-configured business topics and message queues, the target message queue corresponding to the target business topic is determined, and the data streams in the plurality of preset data formats are cached in the target message queue.

4. The method according to claim 3, wherein: The correspondence between the pre-configured service topics and the message queues includes a many-to-one relationship.

5. The method according to claim 1, wherein: The step of parsing the data streams in the plurality of preset data formats into data streams in the target data format includes: Based on the correspondence between the pre-configured data formats and the configuration information, determine the format conversion configuration information corresponding to the multiple preset data formats respectively; wherein the format conversion configuration information includes at least one of the following: type indication parameters of the multiple preset data formats, message queue key values, and type indication parameters of the target data format; Based on the format conversion configuration information, the data streams in the plurality of preset data formats are parsed into data streams in the target data format.

6. The method according to claim 1, wherein: The step of performing persistent processing on the stream-processed data stream includes: Determine a storage proxy instance corresponding to a preset storage type; Based on the storage proxy instance, the stream-processed data stream is persisted.

7. The method according to claim 1, wherein: The multiple preset data formats include at least two of the following: script language object notation format, protocol buffer format and text format.

8. The method according to any one of claims 1 to 7, wherein: Also includes: In response to receiving a risk assessment request, obtaining to-be-assessed data corresponding to the risk assessment request from the persistently processed data; Based on the risk assessment strategy corresponding to the risk assessment request, risk information corresponding to the data to be assessed is determined.

9. The method according to claim 8, wherein: The determining, based on the risk assessment strategy corresponding to the risk assessment request, risk information corresponding to the data to be assessed includes: Extracting risk assessment-related features of the data to be assessed; The features related to risk assessment are processed based on the risk assessment strategy to obtain risk information corresponding to the data to be assessed.

10. A data processing device, comprising: The acquisition module is configured to acquire data streams in a plurality of preset data formats to be processed based on a preset data acquisition method; wherein the preset data acquisition method includes a real-time data acquisition method and an offline data acquisition method; A storage module, configured to cache the data streams in the plurality of preset data formats into a target message queue; A parsing module is configured to process the data stream cached in the target message queue in a streaming manner, parse the data streams in the plurality of preset data formats into a data stream in a target data format, and obtain a stream-processed data stream; The processing module is configured to perform persistence processing on the data stream that has undergone streaming processing to obtain data that has undergone persistence processing.

11. The device according to claim 10, wherein: The acquisition module is specifically configured as follows: In response to the data streams in the plurality of preset data formats corresponding to the real-time incremental data, acquiring the data streams in the plurality of preset data formats based on the real-time data acquisition method; In response to the data streams in the plurality of preset data formats corresponding to the historical static data, the data streams in the plurality of preset data formats are acquired based on the offline data collection method.

12. The device according to claim 10, wherein: The storage module is specifically configured as follows: Determine target business topics corresponding to the data streams in the plurality of preset data formats; Based on the correspondence between the pre-configured business topics and message queues, the target message queue corresponding to the target business topic is determined, and the data streams in the plurality of preset data formats are cached in the target message queue.

13. The device according to claim 12, wherein: The correspondence between the pre-configured service topics and the message queues includes a many-to-one relationship.

14. The device according to claim 10, wherein: The parsing module is specifically configured as follows: Based on the correspondence between the pre-configured data formats and the configuration information, determine the format conversion configuration information corresponding to the multiple preset data formats respectively; wherein the format conversion configuration information includes at least one of the following: type indication parameters of the multiple preset data formats, message queue key values, and type indication parameters of the target data format; Based on the format conversion configuration information, the data streams in the plurality of preset data formats are parsed into data streams in the target data format.

15. The device according to claim 10, wherein: The processing module is specifically configured as follows: Determine a storage proxy instance corresponding to a preset storage type; Based on the storage proxy instance, the stream-processed data stream is persisted.

16. The device according to claim 10, wherein: The multiple preset data formats include at least two of the following: script language object notation format, protocol buffer format and text format.

17. The apparatus according to any one of claims 10 to 16, further comprising: an acquisition module, configured to, in response to receiving a risk assessment request, acquire the to-be-assessed data corresponding to the risk assessment request from the persistently processed data; The assessment module is configured to determine the risk information corresponding to the data to be assessed based on the risk assessment strategy corresponding to the risk assessment request.

18. The device according to claim 17, wherein: The evaluation module is specifically configured as follows: Extracting risk assessment-related features of the data to be assessed; The features related to risk assessment are processed based on the risk assessment strategy to obtain risk information corresponding to the data to be assessed.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the data processing method according to any one of claims 1 to 9.

21. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 9 are implemented.

Images