Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium, relates to the field of data processing, and is suitable for the field of financial science and technology. The method comprises the steps of obtaining multiple pieces of original data in a character string format; based on a preset data mapping rule, performing mapping conversion on each piece of original data to obtain corresponding numeric data; mapping each piece of numeric data to a binary bit in a pre-constructed bitmap storage space to obtain bitmap data; in response to the acquired data processing instruction, determining a target analysis model in an analysis model cluster according to the data processing instruction; wherein the analysis model cluster comprises a plurality of candidate analysis models for different service analysis scenes; and calling the target analysis model to analyze and process the bitmap data to obtain a data processing result. The financial data processing efficiency can be improved while the storage cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing, is applicable to the field of financial technology, and in particular, to a data processing method and device, an electronic device, and a storage medium. Background Art

[0002] With the deepening of digital transformation, financial backend data has accumulated a huge amount of business data. This data is usually stored in the form of strings, such as user ID, order number, product code, etc., which takes up a lot of storage space in the big data cluster. Taking UTF-8-encoded Chinese characters as an example, each character takes up 3 bytes. When the data volume reaches TB or even PB, the storage and computing costs rise sharply.

[0003] In related technologies, two solutions are usually used to solve the above problems: one is to use data compression algorithms to reduce storage space, but this method requires decompression when querying data, which affects query performance; the other is to use a distributed computing framework to improve computing power, but it requires a lot of hardware resources and is costly. Therefore, how to improve the processing efficiency of financial data at a lower storage cost is still a problem that needs to be solved in the industry. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a data processing method and device, an electronic device and a storage medium, aiming to reduce the storage space occupied by financial data while improving data computing efficiency.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides a data processing method, the method comprising: acquiring a plurality of original data in a string format;

[0006] Based on the preset data mapping rules, each of the original data is mapped and converted to obtain corresponding numerical data;

[0007] Mapping each of the numerical data to a binary bit in a pre-constructed bitmap storage space to obtain bitmap data;

[0008] In response to obtaining the data processing instruction, determining a target analysis model in an analysis model cluster according to the data processing instruction; wherein the analysis model cluster includes a plurality of candidate analysis models for different business analysis scenarios;

[0009] The target analysis model is called to analyze and process the bitmap data to obtain a data processing result.

[0010] In some embodiments, the mapping conversion is performed on each of the original data based on a preset data mapping rule to obtain corresponding numerical data, including:

[0011] Acquire the data scale of the plurality of original data;

[0012] If the data size is smaller than a preset data size threshold, the first digit hash algorithm is selected as the target hash algorithm;

[0013] If the data size is greater than or equal to the data size threshold, selecting the second digit hash algorithm as the target hash algorithm;

[0014] Calculate each of the original data using the target hash algorithm to obtain a corresponding hash value;

[0015] The hash value is determined as the numerical data.

[0016] In some embodiments, the pre-construction process of the bitmap storage space includes:

[0017] Obtaining the maximum value of the multiple numerical data;

[0018] Determine the total length of the bitmap storage space according to the maximum value of the value;

[0019] Initialize all binary bits in the bitmap storage space to 0;

[0020] The step of mapping each of the numerical data to a binary bit in a pre-constructed bitmap storage space to obtain bitmap data includes:

[0021] Using each of the numerical data as a position index in the bitmap storage space;

[0022] The binary bit corresponding to the position index is set to 1, so that a mapping is established between the binary bit and the corresponding numerical data.

[0023] In some embodiments, in response to obtaining the data processing instruction, determining the target analysis model in the analysis model cluster according to the data processing instruction includes:

[0024] Obtaining an analysis type according to the data processing instruction analysis;

[0025] Selecting a corresponding target analysis model from the analysis model cluster according to the analysis type;

[0026] The analysis model cluster includes:

[0027] Funnel analysis model, which is used to analyze multiple continuous user behaviors to obtain the number of executions of each step and the conversion rate between adjacent steps;

[0028] Interval analysis models, which calculate the distribution of time differences between two related events;

[0029] Distribution analysis model, used to count the distribution of user behavior or business data in different dimensions;

[0030] Attribution analysis model, used to analyze the degree of correlation between the target event and the preceding events;

[0031] The process analysis model is used to analyze the object retention in different time periods and obtain the changing trend of object loss.

[0032] In some embodiments, when the target analysis model is a funnel analysis model, calling the target analysis model to analyze and process the bitmap data to obtain a data processing result includes:

[0033] Acquire multiple analysis steps of the funnel analysis model;

[0034] Analyze the bitmap data to obtain the execution times of each analysis step;

[0035] Calculating the conversion rate between adjacent steps based on the execution times of each of the analysis steps;

[0036] The data processing result is generated according to the conversion rate.

[0037] In some embodiments, before calling the target analysis model to analyze and process the bitmap data to obtain the data processing result, the method further includes:

[0038] Obtaining the original event type corresponding to the target analysis model;

[0039] When the original event type is a statistical event, obtaining a target time range;

[0040] Counting the number of times the event occurs within the target time range;

[0041] The event occurrence count is added to the bitmap data as merged data.

[0042] In some embodiments, the mapping conversion is performed on each of the original data based on a preset data mapping rule to obtain corresponding numerical data, including:

[0043] Get and initialize the integer value counter;

[0044] For each of the original data, checking whether the original data is assigned a corresponding integer value;

[0045] If the string is not assigned the integer value, assign the current value of the integer value counter to the string, and increase the value of the counter by 1;

[0046] If the string has been assigned the integer value, then obtaining the assigned integer value;

[0047] Establishing a mapping relationship between each of the original data and the corresponding integer value;

[0048] Based on the mapping relationship, the integer value corresponding to the original data is determined as the numerical data.

[0049] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a data processing device, the device comprising:

[0050] The acquisition module is used to obtain multiple raw data in string format;

[0051] A conversion module, used to perform mapping conversion on each of the original data based on a preset data mapping rule to obtain corresponding numerical data;

[0052] A mapping module, used for mapping each of the numerical data to a binary bit in a pre-constructed bitmap storage space to obtain bitmap data;

[0053] A response module, configured to respond to the acquisition of a data processing instruction and determine a target analysis model in an analysis model cluster according to the data processing instruction; wherein the analysis model cluster includes a plurality of candidate analysis models for different business analysis scenarios;

[0054] The analysis module is used to call the target analysis model to analyze and process the bitmap data to obtain a data processing result.

[0055] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0056] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0057] The data processing method and device, electronic device and storage medium proposed in the present application include: obtaining multiple original data in string format; mapping and converting each original data based on preset data mapping rules to obtain corresponding numerical data; mapping each numerical data to a binary bit in a pre-built bitmap storage space to obtain bitmap data; in response to obtaining a data processing instruction, determining a target analysis model in an analysis model cluster according to the data processing instruction; wherein the analysis model cluster includes multiple candidate analysis models for different business analysis scenarios; calling the target analysis model to analyze and process the bitmap data to obtain a data processing result.

[0058] According to the data processing method provided by the present invention, the original data in the string format is first obtained and converted into numerical data based on the preset data mapping rules; then, by mapping the numerical data to the bitmap storage space, the traditional string storage is converted into binary bit storage, so that data compression and management are more efficient; at the same time, by combining the bitmap data with the pre-built analysis model cluster, the scenario adaptability of data analysis is achieved; then, based on the data processing instructions, the target analysis model is determined from the analysis model cluster, and the analysis plan is determined through the model matching process; finally, the bitmap data is analyzed and processed by calling the target analysis model, so as to ensure the efficiency of the calculation process. Through the above method, the present invention can improve the processing efficiency of financial data while reducing storage costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flow chart of a data processing method provided in an embodiment of the present application;

[0060] Figure 2 is another flow chart of the data processing method provided in the embodiment of the present application;

[0061] Figure 3 is another flow chart of the data processing method provided in the embodiment of the present application;

[0062] Figure 4 is another flow chart of the data processing method provided in the embodiment of the present application;

[0063] Figure 5 is another flow chart of the data processing method provided in the embodiment of the present application;

[0064] Figure 6 is another flow chart of the data processing method provided in the embodiment of the present application;

[0065] Figure 7 is another flow chart of the data processing method provided in the embodiment of the present application;

[0066] Figure 8is another flow chart of the data processing method provided in the embodiment of the present application;

[0067] Fig. 9 is a structural diagram of a data processing device provided in an embodiment of the present application;

[0068] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0070] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0072] With the rapid development of financial technology, the digital transformation of various financial institutions has continued to deepen, accumulating a huge amount of business data. These data usually include but are not limited to: user ID, transaction serial number, product number, device ID and other string type data. In actual applications, the above data is usually stored in UTF-8 encoding, and each Chinese character requires 3 bytes of storage space. When the data scale reaches TB or even PB, the storage system faces huge storage and computing pressure.

[0073] At present, the industry mainly adopts the following two types of technical solutions to the above problems:

[0074] The first type of solution is to use data compression algorithms to reduce storage space usage. This type of solution compresses and stores the original data using general compression algorithms such as gzip and snappy. Although it can effectively reduce storage space usage, the data needs to be decompressed before data query and analysis. The decompression process consumes a lot of CPU resources, resulting in a significant decrease in query performance. At the same time, the compressed data cannot be directly filtered and analyzed by conditions, and needs to be decompressed before subsequent processing, which further reduces data processing efficiency.

[0075] The second type of solution is to improve computing power through a distributed computing framework. This type of solution improves the system's parallel processing capabilities by adding computing nodes. Although it can improve data processing efficiency to a certain extent, it requires a large amount of hardware resources, resulting in a sharp increase in system construction and operation and maintenance costs. In addition, when performing multi-table association analysis, since data needs to be transmitted and calculated between different nodes, data skew problems are prone to occur, affecting computing performance.

[0076] In financial business scenarios, historical data often needs to be quickly analyzed and queried in real time. For example, in anti-fraud scenarios, users' historical transaction behaviors need to be quickly analyzed, and in marketing scenarios, users' consumption characteristics need to be analyzed in real time. When dealing with such requirements, existing technical solutions either have high query latency due to decompression or have an unsatisfactory input-output ratio due to high hardware costs, and cannot meet business needs well.

[0077] In addition, financial data has the characteristics of large data volume, fast growth, and high real-time requirements. When processing incremental data updates, existing technical solutions often need to rebuild indexes or re-compress data, which not only increases the system burden, but may also affect the normal operation of online business. At the same time, since financial data involves multiple business scenarios, different scenarios have different performance requirements for data processing. Existing technical solutions lack the ability to optimize for different scenarios, making it difficult to achieve efficient use of resources.

[0078] Based on this, the embodiments of the present application provide a data processing method and device, an electronic device and a storage medium, which aim to reduce the storage space occupied by financial data while improving data computing efficiency.

[0079] The data processing method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the data processing method in the embodiments of the present application is described.

[0080] The data processing method provided in the embodiment of the present application is applicable to the field of financial technology in the field of data processing. The data processing method provided in the embodiment of the present application can be applied to the terminal, can be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the data processing method, etc., but is not limited to the above forms.

[0081] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0082] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0083] Figure 1 is an optional flowchart of the data processing method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0084] Step S101, obtaining a plurality of original data in a string format.

[0085] Step S102: Based on a preset data mapping rule, each original data is mapped and converted to obtain corresponding numerical data.

[0086] Step S103: Map each numerical data to a binary bit in the pre-constructed bitmap storage space to obtain bitmap data.

[0087] Step S104, in response to obtaining the data processing instruction, determining a target analysis model in the analysis model cluster according to the data processing instruction.

[0088] Step S105, calling the target analysis model to analyze and process the bitmap data to obtain a data processing result.

[0089] Steps S101 to S105 shown in the embodiment of the present application first obtain the original data in string format and convert it into numerical data based on the preset data mapping rules; then, by mapping the numerical data to the bitmap storage space, the traditional string storage is converted into binary bit storage, so that data compression and management are more efficient; at the same time, by combining the bitmap data with the pre-built analysis model cluster, the scenario adaptability of data analysis is achieved; then, based on the data processing instructions, the target analysis model is determined from the analysis model cluster, and the analysis plan is determined through the model matching process; finally, the bitmap data is analyzed and processed by calling the target analysis model, so as to ensure the efficiency of the calculation process. Through the above method, the present invention can improve the processing efficiency of financial data while reducing storage costs.

[0090] In step S101 of some embodiments, multiple raw data in string format are obtained. Specifically, the raw data may include, but are not limited to: transaction serial numbers, device identifiers, product codes, account IDs, etc. generated by users in financial services. These data are usually stored in a database or data warehouse using UTF-8 encoding. For example, a user's transaction serial number "TX202311210001" is a 12-bit string, which requires 12 bytes to store using UTF-8 encoding; for another example, a product description containing Chinese characters "gold financial products" consists of 5 Chinese characters, which requires 15 bytes to store using UTF-8 encoding. These raw data can be obtained through a variety of methods such as database query, message queue subscription, and file reading.

[0091] In step S102 of some embodiments, based on the preset data mapping rules, each raw data is mapped and converted to obtain the corresponding numerical data. Specifically, the data mapping rules may include a variety of hash algorithms, and a suitable hash algorithm may be selected according to the scale characteristics and business requirements of the raw data. For example, when historical data analysis shows that the total amount of different raw data does not exceed 400 million, a 32-bit hash algorithm (such as murmurHash2_32, xxHash32, etc.) can be selected to map the string to 4-byte integer data; when the amount of data exceeds 400 million, a 64-bit hash algorithm (such as farmHash64, cityHash64, etc.) can be selected to map the string to 8-byte integer data. This mapping conversion can compress the string data that originally occupies 12-15 bytes to 4-8 bytes.

[0092] See also Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S205:

[0093] Step S201, obtaining the data scale of multiple original data.

[0094] Step S202: If the data size is smaller than a preset data size threshold, the first digit hash algorithm is selected as the target hash algorithm.

[0095] Step S203: If the data size is greater than or equal to the data size threshold, the second digit hash algorithm is selected as the target hash algorithm.

[0096] Step S204, calculating each original data by using the target hash algorithm to obtain a corresponding hash value.

[0097] Step S205: determine the hash value as numerical data.

[0098] In step S201 of some embodiments, in step S201 of some embodiments, the data scale of multiple raw data is obtained. Specifically, the data scale can be determined by counting the total number of different raw data. For example, by de-duplicating the transaction serial numbers in the database, the total number of non-repeated transaction records is obtained; or by performing cardinality statistics on device identifiers, the number of independent devices is obtained. In order to accurately evaluate the data scale, a variety of statistical methods such as precise counting or approximate counting can be used to improve statistical efficiency while ensuring accuracy.

[0099] In step S202 of some embodiments, if the data scale is less than a preset data scale threshold, the first digit hash algorithm is selected as the target hash algorithm. Specifically, the first digit hash algorithm refers to a hash algorithm that can generate a 32-bit integer, including but not limited to: murmurHash2_32, xxHash32, hiveHash, etc. For example, when the preset data scale threshold is set to 400 million records, if the current data scale is 300 million, murmurHash2_32 is selected as the target hash algorithm. The integer range generated by this 32-bit hash algorithm is 0 to 2^32-1, that is, it can represent up to about 4.29 billion different values, which can meet the mapping requirements of small and medium-sized data sets.

[0100] In step S203 of some embodiments, if the data scale is greater than or equal to the data scale threshold, the second-digit hash algorithm is selected as the target hash algorithm. Specifically, the second-digit hash algorithm refers to a hash algorithm that can generate a 64-bit integer, including but not limited to: farmHash64, metroHash64, cityHash64, sipHash64, etc. For example, when the data scale reaches 500 million records, exceeding the preset threshold of 400 million, farmHash64 is selected as the target hash algorithm. The integer range generated by this 64-bit hash algorithm is 0 to 2^64-1, which can represent approximately 1.8×10^19 different values, which can meet the mapping requirements of large-scale data sets.

[0101] In step S204 of some embodiments, each raw data is calculated by the target hash algorithm to obtain a corresponding hash value. Specifically, the raw data is passed into the target hash algorithm as an input string, and the corresponding integer value is obtained through hash calculation. For example, for the transaction serial number "TX202311210001", if the murmurHash2_32 algorithm is used for calculation, the hash value is 2147483647; if the farmHash64 algorithm is used for calculation, the hash value is 9223372036854775807. The hash calculation process ensures that the same input string always obtains the same hash value, and different input strings obtain different hash values ​​with a very high probability.

[0102] In step S205 of some embodiments, the hash value is determined as numerical data. Specifically, the hash value calculated in step S204 is directly used as the numerical representation of the corresponding original data. For example, the hash value 2147483647 obtained by mapping the transaction serial number "TX202311210001" is directly used as a 32-bit integer to represent the transaction; or the hash value 9223372036854775807 obtained by mapping the device identifier is directly used as a 64-bit integer to represent the device. This mapping method not only realizes the conversion from string to integer, but also maintains the uniqueness of the original data, laying the foundation for subsequent bitmap storage and bit operation analysis.

[0103] Through step S201 to step S205, the embodiment of the present application provides a hash mapping scheme based on data scale adaptation. By evaluating the data scale, a 32-bit or 64-bit hash algorithm is intelligently selected to ensure sufficient mapping space while avoiding unnecessary waste of storage space. For example, for small and medium-sized data sets, the use of a 32-bit hash algorithm can provide sufficient mapping space (supporting up to about 4.29 billion different values), while saving 50% of storage space compared to a 64-bit algorithm; for large-scale data sets, it automatically switches to a 64-bit hash algorithm to ensure that there is sufficient mapping space (supporting up to about 1.8×10^19 different values) to meet business needs. In addition, the scheme directly calculates the mapping value through a hash algorithm, without the need to maintain a mapping relationship table, which greatly reduces the complexity of the system implementation and the operation and maintenance cost. Overall, the scheme not only ensures the reliability of the mapping, but also optimizes the storage space, providing a technical basis for the efficient processing of large-scale financial data.

[0104] See also Figure 3 In some embodiments, step S102 may include but is not limited to steps S301 to S306:

[0105] Step S301, obtaining and initializing an integer value counter.

[0106] Step S302: for each original data, check whether the original data is assigned a corresponding integer value.

[0107] Step S303: If the character string is not assigned an integer value, the current value of the integer value counter is assigned to the character string, and the value of the integer value counter is increased by 1.

[0108] Step S304: if the string has been assigned an integer value, the assigned integer value is obtained.

[0109] Step S305, establishing a mapping relationship between each original data and the corresponding integer value.

[0110] Step S306: Based on the mapping relationship, determine the integer value corresponding to the original data as numerical data.

[0111] In step S301 of some embodiments, an integer value counter is obtained and initialized. Specifically, the integer value counter is a counter for generating consecutive integers, and the initial value can be set to 1 or other predetermined starting values. For example, the counter can be initialized to 1, and then used to assign a unique integer identifier to each raw data that is not assigned an integer value; the counter can also be initialized to 1000000.

[0112] In step S302 of some embodiments, for each original data, it is checked whether the original data is assigned a corresponding integer value. Specifically, it is possible to determine whether the current original data has been assigned an integer value by querying the mapping relationship storage module. For example, for the transaction serial number "TX202311210001", it is queried whether the string has a corresponding integer value record in the mapping relationship table; or for the device identifier "DEVICE_001", it is checked whether a corresponding relationship with the integer value has been established.

[0113] In step S303 of some embodiments, if the string is not assigned an integer value, the current value of the integer value counter is assigned to the string, and the value of the integer value counter is increased by 1. Specifically, when it is found that a certain original data has not been assigned an integer value, the current counter value is used as the unique identifier of the data, and the counter is ensured to be incremented for the next allocation. For example, when the counter value is 1000, the value is assigned to a new transaction serial number, and then the counter value is increased to 1001 for use by the next unassigned data.

[0114] In step S304 of some embodiments, if the string has been assigned an integer value, the assigned integer value is obtained. Specifically, the integer value corresponding to the original data is directly obtained by querying the mapping relationship storage module. For example, when it is found that the transaction serial number "TX202311210001" has been assigned an integer value of 1000, the assigned value is directly returned to avoid repeated allocation and ensure that the same original data always corresponds to the same integer value.

[0115] In step S305 of some embodiments, a mapping relationship between each original data and the corresponding integer value is established. Specifically, the original data is used as the key and the corresponding integer value is used as the value to construct a key-value pair and store it in the mapping relationship storage module. For example, the mapping relationship "TX202311210001→1000" can be stored in a mapping table of a relational database; or "DEVI CE_001→1001" can be stored in a distributed key-value storage system. This storage of mapping relationships supports both fast query and subsequent maintenance and update.

[0116] In step S306 of some embodiments, based on the mapping relationship, the integer value corresponding to the original data is determined as numerical data. Specifically, by querying the mapping relationship established in step S305, the integer value corresponding to the original data is obtained, and the integer value is used as the numerical representation of the original data. For example, when the transaction serial number "TX202311210001" needs to be processed, the corresponding integer value 1000 is found through the mapping relationship, and 1000 is directly used as the numerical identifier of the transaction record. This conversion method based on the mapping relationship ensures a one-to-one correspondence between the original data and the numerical representation, and provides a reliable data basis for subsequent data processing.

[0117] Through step S301 to step S306, the embodiment of the present application ensures that each original data can obtain a unique integer identifier by maintaining an integer value counter and a mapping relationship table, while ensuring that the same original data is always mapped to the same integer value. For example, when processing a transaction serial number, it is possible to quickly determine whether a new integer value needs to be assigned by querying the mapping relationship, thereby avoiding the problem of repeated assignment and data inconsistency. This solution is particularly suitable for scenarios where the stability of data mapping needs to be maintained, such as data analysis across time periods, multi-system data interaction, etc. By establishing and maintaining a mapping relationship, it is possible to query the corresponding integer value according to the original data at any time, and it is also possible to reversely check the original data according to the integer value, providing a basis for two-way tracing of data. In addition, the continuous allocation characteristics of integer values ​​enable data to be more compact and efficient when stored and calculated. Overall, while ensuring the accuracy of data mapping, the solution provides a reliable data basis for subsequent data analysis and processing, and is particularly suitable for financial business scenarios with high requirements for data consistency.

[0118] In step S103 of some embodiments, each numerical data is mapped to a binary bit in a pre-constructed bitmap storage space to obtain bitmap data. Specifically, the pre-constructed bitmap storage space is a continuous storage area composed of binary bits, where each binary bit can represent a unique numerical data. For example, for numerical data mapped using a 32-bit hash algorithm, 4GB (2^32bit) of bitmap space needs to be pre-allocated; for data mapped using a 64-bit hash algorithm, 2EB (2^64bit) of bitmap space needs to be pre-allocated. When a certain numerical data appears, the binary bit at the corresponding position in the bitmap is set to 1, so that the storage space of 8 numerical values ​​can be compressed to 1 byte, further improving storage efficiency.

[0119] See also Figure 4 In some embodiments, before step S103, the following steps may also be performed, including but not limited to steps S401 to S403:

[0120] Step S401, obtaining the maximum value of multiple numerical data.

[0121] Step S402: determining the total length of the bitmap storage space according to the maximum value.

[0122] Step S403: Initialize all binary bits in the bitmap storage space to 0.

[0123] In step S401 of some embodiments, the maximum value of multiple numerical data is obtained. Specifically, by traversing all numerical data or using the maximum value statistical function of the database, the maximum integer value in the current data set is found. For example, assuming that the numerical data is generated by a 32-bit hash algorithm, it is found through statistics that the current maximum value is 3758096384, which will be used to determine the specific size of the bitmap storage space; for example, for data generated using a 64-bit hash algorithm, the maximum value may be found to be 9223372036854775807, and this maximum value is also used for subsequent space allocation.

[0124] In step S402 of some embodiments, the total length of the bitmap storage space is determined according to the maximum value of the value. Specifically, the total length of the required bitmap storage space is calculated based on the maximum value of the value. For example, when the maximum value is 3758096384, it is necessary to allocate at least 3758096384 binary bits (about 445MB) of continuous storage space to ensure that all possible values ​​from 0 to the maximum value can be represented; when the maximum value is 9223372036854775807, it is necessary to allocate storage space of corresponding size. The total length of the bitmap storage space is usually allocated in bytes, so the number of bits will be rounded up to a multiple of 8 during actual allocation.

[0125] In step S403 of some embodiments, all binary bits in the bitmap storage space are initialized to 0. Specifically, after the storage space allocation is completed, the initial values ​​of all binary bits need to be set to 0, indicating that no numerical data is currently mapped into the bitmap. For example, for the allocated 445MB bitmap space, all bytes need to be initialized to 0x00; or the memory clear function is used to directly initialize the entire storage area to 0. This initialization operation ensures that the initial state of the bitmap is determined and avoids possible interference caused by residual data in the memory.

[0126] Through the implementation process of step S401 to step S403, the present application provides a dynamic bitmap space allocation scheme based on data characteristics. The scheme first accurately evaluates the required storage space size through the actual distribution characteristics of statistical data, avoiding the waste of resources or insufficient space problems that may be caused by pre-allocating fixed-size space. For example, when the maximum value in the data set is much smaller than the theoretical maximum value (such as the maximum value of a 32-bit integer 4294967295), only the actual required space needs to be allocated, which greatly saves storage resources. At the same time, by uniformly initializing the allocated space, a clear starting state is provided for subsequent bitmap operations, ensuring the accuracy and reliability of data processing. Overall, the scheme realizes the on-demand allocation and efficient use of storage space, which not only meets the needs of data processing, but also avoids excessive occupation of resources.

[0127] See also Figure 5 In some embodiments, step S103 may include but is not limited to steps S501 to S502:

[0128] Step S501: Use each numerical data as a position index in the bitmap storage space.

[0129] Step S502, setting the binary bit corresponding to the position index to 1, so as to establish a mapping between the binary bit and the corresponding numerical data.

[0130] In step S501 of some embodiments, each numerical data is used as a position index in the bitmap storage space. Specifically, the numerical data is directly used as an index value for locating a binary bit in the bitmap storage space. For example, for the numerical data 12345, it means that the data will be mapped to the 12345th binary bit in the bitmap storage space.

[0131] In step S502 of some embodiments, the binary bit corresponding to the position index is set to 1 so that a mapping is established between the binary bit and the corresponding numerical data. Specifically, according to the position index determined in step S501, the binary bit of the corresponding position is modified to 1 in the bitmap storage space. For example, to set the binary position corresponding to the position index 12345 to 1, first calculate its corresponding byte offset as 1543 (12345 / 8) and bit offset as 1 (12345%8), and then set the first bit in the byte to 1 through bit operation (since the bitmap is physically stored in bytes, it is usually necessary to convert the position index into a byte offset and a bit offset during actual positioning, where the byte offset is equal to the position index divided by 8, and the bit offset is equal to the result of taking the position index modulo 8). In this way, a one-to-one mapping relationship is established between the original data, the numerical data, and the binary bits in the bitmap.

[0132] Through the implementation process of step S501 and step S502, the present application provides an efficient data-to-bitmap mapping mechanism. This mechanism avoids additional mapping calculation overhead by directly using numerical data as the bitmap index, and realizes the positioning operation with O(1) time complexity. At the same time, mapping the data to a single binary bit in the bitmap greatly compresses the storage space. For example, an integer value that originally requires 4 bytes or 8 bytes to store only occupies 1 binary bit in the bitmap. In addition, this mapping mechanism supports fast set operations, such as intersection and union, etc., and only needs to perform corresponding bit operations on the bitmap. Overall, this solution not only realizes efficient mapping and storage of data, but also provides a convenient operation method for subsequent data analysis, which is particularly suitable for high-performance processing scenarios of large-scale financial data.

[0133] In step S104 of some embodiments, in response to obtaining a data processing instruction, a target analysis model is determined in the analysis model cluster according to the data processing instruction. Specifically, the analysis model cluster contains a plurality of candidate analysis models optimized for different financial business scenarios, and these models may include but are not limited to: a behavioral feature model for anti-fraud analysis, a user portrait model for marketing analysis, a credit assessment model for risk control analysis, etc. By parsing information such as business scenario identification, analysis objectives, and timeliness requirements in the data processing instruction, a target model that best suits the current analysis needs is selected from the candidate models. For example, when the data processing instruction indicates that real-time anti-fraud analysis is required, a lightweight behavioral feature model optimized for real-time scenarios will be selected.

[0134] See also Figure 6 In some embodiments, step S104 may include but is not limited to steps S601 to S602:

[0135] Step S601, obtaining an analysis type according to the data processing instruction.

[0136] Step S602: Select a corresponding target analysis model from the analysis model cluster according to the analysis type.

[0137] In step S601 of some embodiments, the analysis type is obtained by parsing the data processing instruction. Specifically, the data processing instruction contains specific requirements for business analysis, and the type of analysis that needs to be performed can be determined by parsing the instruction content. For example, when the data processing instruction is "analyze the conversion of users from browsing products to final purchases", it is parsed to obtain that this is a funnel analysis type; for another example, when the instruction is "analyze the time distribution from when users submit orders to when payments are completed", it is parsed to obtain that this is an interval analysis type. The instruction parsing process can accurately identify the business analysis type based on keyword matching, semantic understanding, and other methods.

[0138] In step S602 of some embodiments, a corresponding target analysis model is selected from the analysis model cluster according to the analysis type. Specifically, the analysis model cluster pre-sets a plurality of professional models optimized for different analysis scenarios, including:

[0139] Funnel analysis model: The funnel analysis model is used to analyze multiple continuous user behaviors. Specifically, first obtain the complete user behavior link and clarify the behavior identification of each key node. For example, in the scenario of purchasing financial products, the behavior link includes but is not limited to: browsing product details, clicking the purchase button, completing risk assessment, confirming purchase intention, completing payment and other steps. Among them, the data of each behavior node has been mapped to a specific area in the bitmap storage space. Then, the number of users at each node is obtained through bitmap statistical operations, such as 10,000 initial browsing users, 3,000 click-to-buy users, 2,000 complete risk assessment users, 1,500 confirm purchase users, and 1,000 final payment users. Finally, by calculating the ratio of the number of users of adjacent nodes, the conversion rate indicators of each link are obtained. For example, the conversion rate from browsing to clicking to purchase is 30%, the conversion rate from clicking to purchase to completing risk assessment is 66.7%, and so on. Based on these data, the link with the lowest conversion rate can be identified to provide direction for business optimization.

[0140] Interval analysis model: The interval analysis model is used to calculate the distribution of time differences between two related events. Specifically, first locate the occurrence time of two related events in the bitmap data. For example, to analyze the time interval from order placement to payment, it is necessary to obtain the user's order timestamp and payment timestamp. Then, calculate the difference between the two timestamps and map the time difference to a preset time interval. For example, divide the time difference into intervals such as less than 1 minute, 1-5 minutes, 5-15 minutes, and more than 15 minutes. By counting the number of users in each interval, the distribution of time differences is obtained. For example, statistics show that 40% of users complete payment within 1 minute, 30% of users complete payment within 1-5 minutes, and calculate statistical indicators such as the average payment time of 6 minutes and the median payment time of 4 minutes. These analysis results can be used to evaluate the timeliness of business processes and identify abnormal situations.

[0141] Distribution analysis model: The distribution analysis model is used to count the distribution of user behavior or business data in different dimensions. Specifically, first determine the dimensions and category division criteria that need to be analyzed. For example, in the regional distribution analysis, it can be divided into Class A cities, Class B cities, and other cities; in the product type distribution analysis, it can be divided into Class C, Type D, Type E, etc. Then, the number of data in each category is counted through bitmap operations. For example, in the regional distribution, it is found that Class A cities account for 45% and Class B cities account for 35%.

[0142] Attribution analysis model: The attribution analysis model is used to analyze the degree of correlation between the target event and the preceding event. Specifically, first mark the user group where the target event (such as product purchase) occurs in the bitmap. Then, retrospectively analyze all the behavior records of these users within a certain time window. For example, for 100 purchasing users, count their key behaviors before purchase, such as checking the yield trend (influence 80%), reading product introduction (influence 60%), consulting customer service (influence 40%), etc. By calculating the correlation between these preceding events and purchasing behavior, the most influential factors are identified. For example, it is found that the behavior link of "checking the yield -> reading the introduction -> purchasing" has the highest conversion rate, and the user has the highest probability of purchasing within 2 hours after triggering the key behavior.

[0143] Process analysis model: The process analysis model is used to analyze the retention of objects in different time periods. Specifically, first determine the time period for analysis (such as 7 days, 14 days, 30 days) and the criteria for retention. Then, use a bitmap to record the user activity status at each time point. For example, by analyzing the usage of 100 new users for 30 days, it was found that 80 people were retained on the 7th day (retention rate 80%), 60 people were retained on the 14th day (retention rate 60%), and 40 people were retained on the 30th day (retention rate 40%). By analyzing the changing trend of the retention rate, the key churn points can be identified. For example, it was found that the first week was a high incidence period of user churn, while the churn rate stabilized after two weeks.

[0144] Through the implementation process of step S601 and step S602, the present application provides an intelligent model selection mechanism. The mechanism first accurately understands the business analysis needs by parsing the data processing instructions; then selects the target model that best suits the current analysis scenario from the professional analysis model cluster. This mechanism not only improves the accuracy and efficiency of the analysis, but also can deeply explore different types of business insights through the specialized division of labor of the model. For example, for user behavior analysis, in-depth research can be conducted from multiple dimensions such as funnel conversion, time interval, and behavior distribution at the same time to provide comprehensive data support for business decisions.

[0145] In step S105 of some embodiments, the target analysis model is called to analyze and process the bitmap data to obtain the data processing results. Specifically, since the data has been stored in the form of a bitmap, efficient data analysis can be achieved directly through bit operations. For example, when performing user group intersection analysis, it is only necessary to perform an "and" operation on the bitmap data corresponding to different user groups; when performing user group union analysis, it is only necessary to perform an "or" operation. This analysis method based on bit operations is not only computationally efficient, but also has a small memory footprint, and can support the real-time analysis needs of large-scale data. Finally, the results of the analysis and processing are returned to the requester to complete the entire data processing flow.

[0146] See also Figure 7 In some embodiments, when the target analysis model is a funnel analysis model, step S105 may include but is not limited to steps S701 to S704:

[0147] Step S701, obtaining multiple analysis steps of the funnel analysis model.

[0148] Step S702: Analyze the bitmap data to obtain the execution times of each analysis step.

[0149] Step S703, calculating the conversion rate between adjacent steps based on the number of executions of each analysis step.

[0150] Step S704: Generate data processing results according to the conversion rate.

[0151] In step S701 of some embodiments, multiple analysis steps of the funnel analysis model are obtained. Specifically, the analysis steps refer to the operation steps that the user needs to perform in sequence in the process of completing a certain business goal. For example, for the purchase scenario of financial products, the analysis steps may include: browsing the product details page, clicking the purchase button, confirming the purchase amount, completing the risk assessment, confirming the purchase, etc.; for another example, for the user account opening scenario, the analysis steps may include: registering an account, completing real-name authentication, binding a bank card, setting a transaction password, etc. These steps form a complete funnel conversion link according to the business logic.

[0152] In step S702 of some embodiments, the bitmap data is analyzed to obtain the number of executions of each analysis step. Specifically, since the execution records of each step have been mapped to the corresponding binary bits in the bitmap storage space, the number of executions of each step can be obtained by counting the number of binary bits with a value of 1 in the bitmap area corresponding to each step. For example, through bitmap operations, it is found that the number of users browsing the product details page is 10,000, the number of users clicking the purchase button is 3,000, and the number of users completing the purchase is 1,000; for another example, the number of users who have registered accounts is 5,000, the number of users who have completed real-name authentication is 3,500, and the number of users who have successfully opened an account is 2,000.

[0153] In step S703 of some embodiments, the conversion rate between adjacent steps is calculated based on the number of executions of each analysis step. Specifically, the conversion rate of each link is obtained by calculating the ratio of the number of executions of adjacent steps. For example, in the financial product purchase scenario, the conversion rate of clicking the purchase button relative to browsing the details page is 30% (3000 / 10000), and the conversion rate of finally completing the purchase relative to clicking to purchase is 33.3% (1000 / 3000); for another example, in the account opening scenario, the conversion rate of real-name authentication relative to registration is 70% (3500 / 5000), and the conversion rate of successful account opening relative to real-name authentication is 57.1% (2000 / 3500).

[0154] In step S704 of some embodiments, data processing results are generated according to the conversion rate. Specifically, the execution times and conversion rates of each step are sorted and displayed according to business analysis requirements. For example, a funnel chart can be generated to intuitively display the user's conversion process from the initial step to the final step, and the specific conversion rate is marked between each step; for another example, a trend chart can be generated to display the changes in the conversion rate of each step in different time periods to help discover the fluctuation pattern of the conversion rate.

[0155] Through the implementation process of step S701 to step S704, the present application provides an efficient funnel analysis solution based on bitmap operations. The solution first determines the key steps by analyzing the business process, then uses the characteristics of the bitmap data structure to quickly count the execution of each step, and finally deeply analyzes the user behavior characteristics through conversion rate calculation. Compared with the traditional relational database analysis method, this bitmap-based analysis method has the advantages of high computing efficiency and low storage cost. For example, when analyzing millions of user data, the traditional method requires a large number of table joins and aggregation operations, while the bitmap method only requires simple bit operations to complete.

[0156] See also Figure 8 In some embodiments, before step S105, the following steps may also be performed, but are not limited to, steps S801 to S804:

[0157] Step S801, obtaining the original event type corresponding to the target analysis model.

[0158] Step S802: when the original event type is a statistical event, obtain the target time range.

[0159] Step S803, counting the number of times the event occurs within the target time range.

[0160] Step S804: Add the event occurrence count as merged data to the bitmap data.

[0161] In step S801 of some embodiments, the original event type corresponding to the target analysis model is obtained. Specifically, the original event types include state events and statistical events. Among them, state events focus on the state value at a certain point in time, such as the risk level and account status of the user at a certain moment. Such events usually do not need to be aggregated in the time dimension; statistical events focus on the accumulation of events over a period of time, such as the number of user logins, transaction amounts, etc. Such events often record a large amount of millisecond-level detailed data, and need to be aggregated in the time dimension according to analysis requirements. For example, a user payment event may record a timestamp that is accurate to milliseconds, such as "2024-01-01 10:15:23.456", but in actual analysis, it is usually only necessary to count the total amount of payments per hour or day.

[0162] In step S802 of some embodiments, when the original event type is a statistical event, the target time range is obtained. Specifically, according to the actual needs of the business analysis scenario, determine the appropriate time aggregation granularity. For example, when analyzing the user's daily behavior distribution, it is necessary to aggregate at the hourly granularity, and merge the millisecond-level original behavior data into statistical values ​​at 24 hour points; when analyzing the monthly sales trend of the product, it is necessary to aggregate at the daily granularity, and merge all transaction records within each day into a daily summary value. By analyzing historical data, it is found that more than 99% of business analysis scenarios only require hourly or even daily time accuracy, and only a very small number of scenarios require finer-grained data.

[0163] In step S803 of some embodiments, the number of occurrences of the event within the target time range is counted. Specifically, the event data within the same time span is aggregated and counted to obtain the statistical features of the event within the time unit. For example, a user may generate hundreds of millisecond-level click records within the hour from 10:00 to 11:00. Through time aggregation, an hourly statistical value such as "the number of clicks of user A during 10 o'clock is 500 times" can be obtained; for another example, a product may generate tens of thousands of millisecond-level transaction records in a day. Through time aggregation, a daily statistical value such as "the total sales of product B on that day is 1 million yuan" can be obtained. This aggregation operation can compress the original trillion-level detailed data to hundreds of billions or even tens of billions, significantly reducing the data scale for subsequent analysis and processing.

[0164] In step S804 of some embodiments, the number of event occurrences is added to the bitmap data as merged data. Specifically, the corresponding position in the bitmap storage is updated according to the statistically obtained event aggregation value. For example, the number of user logins per hour is converted into binary representation and stored in the corresponding area of ​​the bitmap; the daily sales volume of the product is converted into binary code and added to the corresponding position of the bitmap data. This storage method not only maintains the efficient characteristics of the bitmap structure, but also realizes the compression of the time dimension of the data, providing an optimized data basis for subsequent model analysis.

[0165] Through the implementation process of step S801 to step S804, the present application provides a data optimization solution based on time dimension aggregation. This solution achieves dual optimization of storage space and computing performance by merging the original millisecond data at the time granularity. For example, for an average of 10 billion original detailed data per day, it can be compressed to 1 billion through hourly aggregation, and further compressed to 100 million through daily aggregation, which significantly improves system performance while meeting business analysis needs. This solution not only reduces storage costs and computing pressure, but also improves query response speed, and is particularly suitable for efficient analysis and processing of massive data in the financial field.

[0166] The data processing method and device, electronic device and storage medium proposed in the present application include: obtaining multiple original data in string format; mapping and converting each original data based on preset data mapping rules to obtain corresponding numerical data; mapping each numerical data to a binary bit in a pre-built bitmap storage space to obtain bitmap data; in response to obtaining a data processing instruction, determining a target analysis model in an analysis model cluster according to the data processing instruction; wherein the analysis model cluster includes multiple candidate analysis models for different business analysis scenarios; calling the target analysis model to analyze and process the bitmap data to obtain a data processing result.

[0167] According to the data processing method provided by the present invention, the original data in the string format is first obtained and converted into numerical data based on the preset data mapping rules; then, by mapping the numerical data to the bitmap storage space, the traditional string storage is converted into binary bit storage, so that data compression and management are more efficient; at the same time, by combining the bitmap data with the pre-built analysis model cluster, the scenario adaptability of data analysis is achieved; then, based on the data processing instructions, the target analysis model is determined from the analysis model cluster, and the analysis plan is determined through the model matching process; finally, the bitmap data is analyzed and processed by calling the target analysis model, so as to ensure the efficiency of the calculation process. Through the above method, the present invention can improve the processing efficiency of financial data while reducing storage costs.

[0168] See also Fig. 9 The present application also provides a data processing device that can implement the above data processing method. The device includes:

[0169] The acquisition module is used to obtain multiple raw data in string format;

[0170] A conversion module, used to perform mapping conversion on each of the original data based on a preset data mapping rule to obtain corresponding numerical data;

[0171] A mapping module, used for mapping each of the numerical data to a binary bit in a pre-constructed bitmap storage space to obtain bitmap data;

[0172] A response module, configured to respond to the acquisition of a data processing instruction and determine a target analysis model in an analysis model cluster according to the data processing instruction; wherein the analysis model cluster includes a plurality of candidate analysis models for different business analysis scenarios;

[0173] The analysis module is used to call the target analysis model to analyze and process the bitmap data to obtain a data processing result.

[0174] The specific implementation of the data processing device is substantially the same as the specific implementation of the above-mentioned data processing method, and will not be described in detail here.

[0175] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above data processing method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0176] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0177] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0178] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the data processing method of the embodiments of this application;

[0179] Input / output interface 1003, used to implement information input and output;

[0180] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0181] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0182] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0183] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above-mentioned data processing method when executed by a processor.

[0184] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0185] The data processing method and device, electronic device and storage medium proposed in the present application include: obtaining multiple original data in string format; mapping and converting each original data based on preset data mapping rules to obtain corresponding numerical data; mapping each numerical data to a binary bit in a pre-built bitmap storage space to obtain bitmap data; in response to obtaining a data processing instruction, determining a target analysis model in an analysis model cluster according to the data processing instruction; wherein the analysis model cluster includes multiple candidate analysis models for different business analysis scenarios; calling the target analysis model to analyze and process the bitmap data to obtain a data processing result.

[0186] According to the data processing method provided by the present invention, the original data in the string format is first obtained and converted into numerical data based on the preset data mapping rules; then, by mapping the numerical data to the bitmap storage space, the traditional string storage is converted into binary bit storage, so that data compression and management are more efficient; at the same time, by combining the bitmap data with the pre-built analysis model cluster, the scenario adaptability of data analysis is achieved; then, based on the data processing instructions, the target analysis model is determined from the analysis model cluster, and the analysis plan is determined through the model matching process; finally, the bitmap data is analyzed and processed by calling the target analysis model, so as to ensure the efficiency of the calculation process. Through the above method, the present invention can improve the processing efficiency of financial data while reducing storage costs.

[0187] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0188] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0189] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0191] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0192] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0193] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0194] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0196] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0197] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Get multiple raw data in string format; Based on the preset data mapping rules, each of the original data is mapped and converted to obtain corresponding numerical data; Mapping each of the numerical data to a binary bit in a pre-constructed bitmap storage space to obtain bitmap data; In response to obtaining the data processing instruction, determining a target analysis model in an analysis model cluster according to the data processing instruction; wherein the analysis model cluster includes a plurality of candidate analysis models for different business analysis scenarios; The target analysis model is called to analyze and process the bitmap data to obtain a data processing result.

2. The data processing method according to claim 1, characterized in that: The mapping conversion is performed on each of the original data based on the preset data mapping rules to obtain corresponding numerical data, including: Acquire the data scale of the plurality of original data; If the data size is smaller than a preset data size threshold, the first digit hash algorithm is selected as the target hash algorithm; If the data size is greater than or equal to the data size threshold, selecting the second digit hash algorithm as the target hash algorithm; Calculate each of the original data using the target hash algorithm to obtain a corresponding hash value; The hash value is determined as the numerical data.

3. The data processing method according to claim 1, characterized in that: The pre-construction process of the bitmap storage space includes: Obtaining the maximum value of the multiple numerical data; Determine the total length of the bitmap storage space according to the maximum value of the value; Initialize all binary bits in the bitmap storage space to 0; The step of mapping each of the numerical data to a binary bit in a pre-constructed bitmap storage space to obtain bitmap data includes: Using each of the numerical data as a position index in the bitmap storage space; The binary bit corresponding to the position index is set to 1, so that a mapping is established between the binary bit and the corresponding numerical data.

4. The data processing method according to claim 1, characterized in that: In response to acquiring the data processing instruction, determining the target analysis model in the analysis model cluster according to the data processing instruction includes: Obtaining an analysis type according to the data processing instruction analysis; Selecting a corresponding target analysis model from the analysis model cluster according to the analysis type; The analysis model cluster includes: Funnel analysis model, which is used to analyze multiple continuous user behaviors to obtain the number of executions of each step and the conversion rate between adjacent steps; Interval analysis models, which calculate the distribution of time differences between two related events; Distribution analysis model, used to count the distribution of user behavior or business data in different dimensions; Attribution analysis model, used to analyze the degree of correlation between the target event and the preceding events; The process analysis model is used to analyze the object retention in different time periods and obtain the changing trend of object loss.

5. The data processing method according to claim 4, characterized in that: When the target analysis model is a funnel analysis model, the calling of the target analysis model to analyze and process the bitmap data to obtain a data processing result includes: Acquire multiple analysis steps of the funnel analysis model; Analyze the bitmap data to obtain the execution times of each analysis step; Calculating the conversion rate between adjacent steps based on the execution times of each of the analysis steps; The data processing result is generated according to the conversion rate.

6. The data processing method according to claim 1, characterized in that: Before calling the target analysis model to analyze and process the bitmap data to obtain the data processing result, the method further includes: Obtaining the original event type corresponding to the target analysis model; When the original event type is a statistical event, obtaining a target time range; Counting the number of times the event occurs within the target time range; The event occurrence count is added to the bitmap data as merged data.

7. The data processing method according to claim 1, characterized in that: The mapping conversion is performed on each of the original data based on the preset data mapping rules to obtain corresponding numerical data, including: Get and initialize the integer value counter; For each of the original data, checking whether the original data is assigned a corresponding integer value; If the string is not assigned the integer value, assigning the current value of the integer value counter to the string, and increasing the value of the integer value counter by 1; If the string has been assigned the integer value, then obtaining the assigned integer value; Establishing a mapping relationship between each of the original data and the corresponding integer value; Based on the mapping relationship, the integer value corresponding to the original data is determined as the numerical data.

8. A data processing device, characterized in that: The device comprises: The acquisition module is used to obtain multiple raw data in string format; A conversion module, used to perform mapping conversion on each of the original data based on a preset data mapping rule to obtain corresponding numerical data; A mapping module, used for mapping each of the numerical data to a binary bit in a pre-constructed bitmap storage space to obtain bitmap data; A response module, configured to respond to the acquisition of a data processing instruction and determine a target analysis model in an analysis model cluster according to the data processing instruction; wherein the analysis model cluster includes a plurality of candidate analysis models for different business analysis scenarios; The analysis module is used to call the target analysis model to analyze and process the bitmap data to obtain a data processing result.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the data processing method according to any one of claims 1 to 7 when executed by a processor.

Images