Data processing method and device, equipment, storage medium and product

Abstract

The invention belongs to the technical field of international roaming, and discloses a data processing method and device, equipment, a storage medium and a product. According to the method, the SIOT protocol file is determined according to the international roaming mail, the roaming charge data is determined according to the SIOT protocol file, then the key atomic information is determined according to the roaming charge data, the discount model is configured according to the key atomic information, and the target amount is calculated according to the discount model and the roaming charge data. According to the method, the SIOT protocol file is automatically identified and extracted according to the international roaming mail, the roaming tariff data is determined according to the SIOT protocol file, the roaming tariff data is further abstracted and defined, the key atomic information is generated, and then the discount model is configured according to the key atomic information. According to the invention, the rapid settlement of the roaming tariff data is realized through the seam allowance model, and the target amount is obtained, so that the international roaming special tariff can be automatically and effectively settled.

Description

Technical Field

[0001] The present application relates to the field of international roaming technology, and in particular to a data processing method, device, equipment, storage medium and product. Background Art

[0002] At present, the Special International Roaming Tariff (SIOT) agreement involves a large number of tariff settlements. The solution is that the operator inspects the public mailbox every day, extracts the SIOT agreement and archives it. The tariff information in the SIOT agreement is manually identified and manually entered into the settlement front desk. The settlement front desk settles according to the tariff information. This method is inefficient. Due to the sharp increase in the volume of special international roaming call bills, it can no longer meet the requirements of special international roaming tariff settlement. Therefore, how to automatically and effectively settle special international roaming tariffs has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of this application is to provide a data processing method, device, equipment, storage medium and product, aiming to solve the technical problem of how to automatically and effectively settle special international roaming charges.

[0004] To achieve the above purpose, the present application provides a data processing method, which comprises the following steps:

[0005] Determine a SIOT protocol document according to the international roaming email, and determine roaming tariff data according to the SIOT protocol document;

[0006] Determining key atomic information according to the roaming tariff data, and configuring a discount model according to the key atomic information;

[0007] A target amount is calculated according to the discount model and the roaming tariff data.

[0008] Optionally, the step of determining a SIOT protocol file according to an international roaming email, and determining roaming tariff data according to the SIOT protocol file specifically includes:

[0009] Determine a SIOT protocol file according to the international roaming mail, and construct an initial execl file according to the SIOT protocol file;

[0010] Correcting the misidentified information in the initial execl file to obtain a target execl file;

[0011] The data corresponding to the target execl file is used as roaming charge data.

[0012] Optionally, the step of constructing an initial execl file according to the SIOT protocol file specifically includes:

[0013] Determine text box coordinates, text content, cell coordinates and table structure information according to the SIOT protocol file;

[0014] Determine the cell coordinates according to the text box coordinates and the cell coordinates;

[0015] Determine the cell text according to the text content and the cell coordinates;

[0016] Construct an initial execl file according to the cell text and the table structure information.

[0017] Optionally, the step of determining text box coordinates, text content, cell coordinates and table structure information according to the SIOT protocol file specifically includes:

[0018] Preprocessing the SIOT protocol file to obtain a preprocessed protocol file;

[0019] Performing text detection on the preprocessed protocol file to obtain text box coordinates;

[0020] Performing text recognition on the preprocessed protocol file to obtain text content;

[0021] The table structure of the SIOT protocol file is predicted to obtain cell coordinates and table structure information.

[0022] Optionally, the step of calculating the target amount according to the discount model and the roaming tariff data specifically includes:

[0023] Determine the charging model and the agreed tariff information according to the key atomic information;

[0024] Aggregating the roaming tariff data according to the agreed tariff information to obtain aggregated data of different granularities;

[0025] A target amount is calculated according to the aggregated data, the agreed tariff information and the discount model, wherein the target amount includes a discount amount and a refund amount.

[0026] Optionally, after the step of calculating the target amount according to the discount model and the roaming tariff data, the method further includes:

[0027] Process historical business data to obtain processed business data;

[0028] Training the initial prediction model according to the processed business data to obtain a target prediction model;

[0029] The target amount is predicted and analyzed by the target prediction model to obtain a prediction result.

[0030] In addition, to achieve the above-mentioned purpose, the present application also provides a data processing device, the data processing device comprising:

[0031] A data acquisition module, used to determine a SIOT protocol file according to an international roaming email, and determine roaming tariff data according to the SIOT protocol file;

[0032] A model configuration module, used to determine key atomic information according to the roaming tariff data, and configure a discount model according to the key atomic information;

[0033] The amount calculation module is used to calculate the target amount according to the discount model and the roaming fee data.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the data processing method described above.

[0035] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0037] The present application determines the SIOT protocol file according to the international roaming mail, determines the roaming tariff data according to the SIOT protocol file, then determines the key atomic information according to the roaming tariff data, configures the discount model according to the key atomic information, and then calculates the target amount according to the discount model and the roaming tariff data. The present application first automatically identifies and extracts the SIOT protocol file according to the international roaming mail, determines the roaming tariff data according to the SIOT protocol file, further abstracts and defines the roaming tariff data, generates the key atomic information, and then configures the discount model according to the key atomic information, realizes the rapid settlement of the roaming tariff data through the stop model, and obtains the target amount, so that the special international roaming tariff can be automatically and effectively settled. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 This is a flow chart of the first embodiment of the data processing method of the present application;

[0041] Figure 2 A schematic diagram of the particle level of an embodiment of the data processing method of the present application;

[0042] Figure 3 This is a flow chart of the second embodiment of the data processing method of the present application;

[0043] Figure 4 This is a general architecture diagram of an embodiment of the data processing method of the present application;

[0044] Figure 5 This is a schematic diagram of cross-domain collaborative processing of RAP robots in accordance with an embodiment of a data processing method of the present application;

[0045] Figure 6 A flowchart of an OCR module of an embodiment of a data processing method of the present application;

[0046] Figure 7 This is a flow chart of the third embodiment of the data processing method of the present application;

[0047] Figure 8 This is a schematic diagram of the overall process of calculating the target amount in an embodiment of the data processing method of this application;

[0048] Fig. 9 A detailed flowchart of calculating a target amount according to an embodiment of a data processing method of the present application;

[0049] Fig.10 This is a schematic diagram of an initial prediction model according to an embodiment of the data processing method of the present application;

[0050] Fig.11 A flow chart of a flow-batch integrated data processing mechanism of an embodiment of a data processing method of the present application;

[0051] Fig.12 This is a structural block diagram of the first embodiment of the data processing device of the present application;

[0052] Fig.13 It is a structural diagram of a data processing device of a hardware operating environment involved in an embodiment of the present application.

[0053] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0054] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0055] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0056] The main solution of the embodiment of the present application is: determine the SIOT protocol file according to the international roaming email, and determine the roaming tariff data according to the SIOT protocol file; determine the key atomic information according to the roaming tariff data, and configure the discount model according to the key atomic information; calculate the target amount according to the discount model and the roaming tariff data.

[0057] At present, the Special International Roaming Rate (SIOT) agreement involves a large number of rate settlements. The solution is for operators to inspect public mailboxes every day, extract SIOT agreements and archive them. Manually identify the rate information in the SIOT agreement and manually enter the rate information into the settlement front desk. The settlement front desk settles according to the rate information. This method is inefficient and can no longer meet the requirements of international roaming special rate settlement due to the sharp increase in the volume of international roaming special call bills.

[0058] The present application determines the SIOT protocol file according to the international roaming mail, determines the roaming tariff data according to the SIOT protocol file, then determines the key atomic information according to the roaming tariff data, configures the discount model according to the key atomic information, and then calculates the target amount according to the discount model and the roaming tariff data. The present application first automatically identifies and extracts the SIOT protocol file according to the international roaming mail, determines the roaming tariff data according to the SIOT protocol file, further abstracts and defines the roaming tariff data, generates the key atomic information, and then configures the discount model according to the key atomic information, realizes the rapid settlement of the roaming tariff data through the stop model, and obtains the target amount, so that the special international roaming tariff can be automatically and effectively settled.

[0059] It should be noted that the execution subject of the present application may be a computing service device with data processing, network communication and program running functions, such as a computer, or an electronic device capable of implementing the above functions, a data processing device, etc. The following takes a data processing device as an example to illustrate this embodiment and the following embodiments.

[0060] Based on this, the present application embodiment provides a data processing method, referring to Figure 1 , Figure 1This is a flowchart of the first embodiment of the data processing method of the present application.

[0061] In this embodiment, the data processing method includes the following steps:

[0062] Step S10: determining a SIOT protocol file according to the international roaming email, and determining roaming tariff data according to the SIOT protocol file.

[0063] It is understandable that robotic process automation (RPA) technology can be used to implement international roaming mail inspection, automatically extract mail attachments, download Special Inter Operator Tariff (SIOT) protocol files sent by international operators to the local computer, and decompress and classify the files, thereby realizing automatic downloading, decompression, and classification of international roaming SIOT protocol files.

[0064] It should be understood that, in this embodiment, the SIOT protocol file may be processed by optical character recognition (OCR) technology and neuro-linguistic programming (NLP) technology to obtain roaming tariff data.

[0065] Step S20: determining key atomic information according to the roaming tariff data, and configuring a discount model according to the key atomic information.

[0066] It is understandable that the roaming charge data can be further abstracted and defined to generate key atomic information, which may include call type, minimum billing unit, statistical dimension, calculation rule, billing model, etc. Specifically, call type: including voice caller local Voice Local, voice caller home Voice Home, voice caller third party Voice Other, voice caller Voice MT, SMS caller SMS MT, and data traffic Data. (2) Minimum billing unit: The minimum unit used in the billing process. Regardless of whether the actual usage reaches the minimum billing unit, the minimum billing unit is used for billing. For example, if the minimum billing unit is 1 minute, if the voice call duration is less than 1 minute, it will be billed as 1 minute.

[0067] Statistical granularity dimension: The statistical granularity dimension is based on the original call detail information (Call Detail Record) recorded in the TAP file, and the service usage and amount are counted and summarized at different granularity levels. For example, the call details of each International Mobile Subscriber Identity (IMSI) and the call summary of each roaming partner (RP). Service usage can be call business usage, number of users, number of devices, etc., collectively referred to as TAP usage below. There are usually the following granularity levels: ①CDR Level: Service usage is counted per call per RP per IMSI; ②IMSIDaily Level: Service usage is counted per day per RP per IMSI; ③IMSIMonthly Level: Service usage is counted per month per RP per IMSI; ④RP Level: Service usage is counted per RP, refer to Figure 2 , Figure 2 This is a schematic diagram of the particle level of an embodiment of the data processing method of this application. Calculation rules: The calculation rules define the calculation logic of the IOT discount model. Common calculation rules are: ① Unit Charge: Charge according to the unit price. ② Cap: Within a certain threshold range, regardless of the usage, the cap is the CAP value. ③ Fixed Fee: Charge a fixed amount.

[0068] Billing model: As the service usage increases, the unit rate gradually decreases or increases. This is a special manifestation of the threshold model. By defining thresholds, the usage is divided into different billing interval levels. Common service usage threshold units are: (1) Traffic: usage; (2) Amount: amount; (3) Count: count. For the sake of example, the following uniform definition is the usage threshold. Common threshold-based charging models include the paragraph model and the ladder model, which have the following different billing modes.

[0069] ① Section model: All usage is charged according to the tariff of the interval layer where the usage falls.

[0070] Section interval layer (MOC, unit min) IOT calculation rules (unit: SDR / min) [0,5000] 0.02 (5000,7000] 0.015 >7000 0.01

[0071] The above interval stratification also applies to SMS and Data. Assuming that the voice usage in the month is 10,000 minutes, the settlement amount = 10,000*0.01 = 100 SDR.

[0072] ②Tier model: When the usage reaches a given threshold of a certain interval layer, the interval usage of the interval layer is calculated according to the tariff of the interval layer, and finally the calculation fees of all previous interval layers are accumulated.

[0073] Tier interval layer (MOC, unit min) IOT calculation rules (SDR / min) [0,5000] 0.02 (5000,7000] 0.015 >7000 0.01

[0074] The above interval stratification also applies to SMS and Data. Assuming that the voice usage in the month is 10,000 minutes, the settlement amount = 5,000*0.02+2,000*0.015+3,000*0.01=160SDR.

[0075] ③Balanced model: Different from the threshold model, it is necessary to introduce the concept of balance factor. The balance factor can set a balance benchmark based on the roaming usage exchanged between operators to adjust and compensate the charges. The roaming usage of both parties needs to be compared with the balance benchmark. For the part equal to the balance benchmark, both parties are charged at the balance rate, and for the part exceeding the balance benchmark, they are charged at the imbalance rate. If the balance factor is n, the balance benchmark = min{incoming usage, outgoing usage}*n. Assuming the balance factor is 2, the balance rate for voice calls is 0.2SDR / min, and the imbalance rate is 0.3SDR / min. The usage of mutual visits between operators is as follows:

[0076] Usage of B visiting A (A as VPMN) A's usage for visiting B (A as HPMN party) 5000minutes 7000minutes

[0077] Obviously, the balance basis = min{5000,7000}*2 = 10000min, then, if A acts as the VPMN, A charges B: 5000*0.2 = 1000SDR; if A acts as the HPMN, B charges A: 7000*0.2 = 1400SDR.

[0078] ④ Commitment model: The commitment volume refers to a fixed commitment value set for voice, SMS, data and other call types within a certain period. If the commitment value is not reached, the service will be charged according to the committed value. The commitment can be made for the service usage or the amount.

[0079] In a specific implementation, since the discount model exists in the key atom information, the discount model can be configured according to the key atom information.

[0080] Step S30: Calculate the target amount according to the discount model and the roaming tariff data.

[0081] It is understandable that the target amount can be calculated based on the above discount model and roaming tariff data, and the target amount may include the discount amount, refund amount, etc. Specifically, RPA technology is used to enter key atomic information into the settlement system to realize the automatic import of SIOT protocol file tariffs and reduce the workload of SIOT protocol tariff entry.

[0082] This embodiment determines the SIOT protocol file according to the international roaming mail, determines the roaming tariff data according to the SIOT protocol file, then determines the key atomic information according to the roaming tariff data, configures the discount model according to the key atomic information, and then calculates the target amount according to the discount model and the roaming tariff data. This embodiment first automatically identifies and extracts the SIOT protocol file according to the international roaming mail, determines the roaming tariff data according to the SIOT protocol file, further abstracts and defines the roaming tariff data, generates the key atomic information, and then configures the discount model according to the key atomic information, and realizes the rapid settlement of the roaming tariff data through the stop model to obtain the target amount, so that the special international roaming tariff can be automatically and effectively settled.

[0083] refer to Figure 3 , Figure 3 This is a flow chart of the second embodiment of the data processing method of the present application.

[0084] Based on the above first embodiment, in this embodiment, step S10 includes:

[0085] Step S101: Determine the SIOT protocol file according to the international roaming mail, and construct an initial execl file according to the SIOT protocol file.

[0086] Understandably, referring to Figure 4 , Figure 4 This is a general architecture diagram of an embodiment of the data processing method of this application. Figure 4 As shown in the figure, the architecture is divided into three layers: application layer, analysis layer, and data layer. Application layer: mainly includes RPA client and settlement system front desk. By developing RPA robots, periodic execution tasks are set on the RPA client. The RPA client will issue instructions to the RPA robot. The robot completes the job processing by calling RPA capabilities and feeds back the execution results to the RPA client. RPA technology can realize data communication between mailboxes and settlement systems. The settlement system front desk is mainly used for international roaming settlement tariff data configuration and settlement report data query and display.

[0087] Analysis layer: mainly includes RPA automation services, intelligent document processing services, settlement and audit services, and completes data extraction, identification, input, settlement, audit and other functions. RPA automation services are implemented by calling RPA capabilities. Based on the RPA designer, a robot queue of the "inspection robot + input robot" mode is developed to realize the collaborative processing of data between different systems. The robot automatically extracts the SIOT protocol files sent by the peer operator or international company from the mailbox system (M domain) according to the email subject every day, and stores the protocols in different directories of the local server according to the operator name. The extracted SIOT protocol files will be identified locally, and the atomic attributes will be entered into the settlement system, that is, the application front end (B domain), to realize the full process of cross-domain special tariff data automatic export and input, and solve the problem of cross-domain data interaction. RPA automation services such as Figure 5 As shown, Figure 5 This is a schematic diagram of the cross-domain collaborative processing of the RAP robot according to an embodiment of the data processing method of the present application. The key atomic information can be determined according to the above process.

[0088] In addition, the intelligent document processing service includes an OCR module and a text error correction module. The OCR module uses advanced text recognition, table recovery and other technologies, based on large-scale data set training, to identify and extract elements such as tables and texts for documents in common scenarios such as office, politics and law, and bills, and perform full-text typesetting, which can achieve accurate recognition of document content. The SIOT protocol file sent by the operator is in an embedded image format and cannot be directly recognized and entered. Therefore, OCR technology is used to extract the tariff table data information in the SIOT protocol file and export it to Excel for subsequent entry into the system. The main principles are as follows: Figure 6 As shown, Figure 6 This is a flowchart of the OCR module of an embodiment of the data processing method of the present application, which mainly includes 6 steps, namely text detection, text recognition, table structure prediction, Cell coordinate aggregation, Cell text aggregation, and Excel export.

[0089] Furthermore, in order to accurately construct the initial execl file, in the present embodiment, the step S101 includes: determining the text box coordinates, text content, cell coordinates and table structure information according to the SIOT protocol file; determining the cell coordinates according to the text box coordinates and the cell coordinates; determining the cell text according to the text content and the cell coordinates; and constructing the initial execl file according to the cell text and the table structure information.

[0090] It can be understood that cell coordinate aggregation is to obtain cell coordinates by calculating the vertex distance between the text box coordinates obtained in text detection and the cell coordinates obtained by table structure prediction, and performing single-row and multi-row aggregation. Cell text aggregation is to splice text content and cell coordinates through coordinate aggregation information. Excel export combines table structure information and cell text, and finally exports the table content in the SIOT protocol into an initial excel file.

[0091] Furthermore, in order to accurately determine the text box coordinates, text content, cell coordinates and table structure information, in the present embodiment, the step of determining the text box coordinates, text content, cell coordinates and table structure information according to the SIOT protocol file specifically includes: preprocessing the SIOT protocol file to obtain a preprocessed protocol file; performing text detection on the preprocessed protocol file to obtain text box coordinates; performing text recognition on the preprocessed protocol file to obtain text content; performing table structure prediction on the SIOT protocol file to obtain cell coordinates and table structure information.

[0092] It is understandable that the SIOT protocol file can be preprocessed, including cropping, scaling, denoising, etc. In the table OCR, the SIOT protocol file is first converted into an image, and the converted image is cropped to identify the various areas of the table, such as horizontal and vertical fields such as titles, column names, and data. Commonly used image cropping methods include edge detection-based, threshold segmentation-based, and region segmentation-based methods. Through these methods, the table image can be cropped into different areas to obtain the preprocessed protocol file. Then, the preprocessed protocol file is subjected to text detection, that is, the text area in the image of the preprocessed protocol file is detected, which is used to confirm the position and boundary information of the text, and obtain the text box coordinates, that is, the 4-point coordinates of the text box.

[0093] It should be understood that text recognition can also be performed on the preprocessed protocol file, that is, the text area in the preprocessed protocol file is identified and converted into text content. The table structure prediction of the SIOT protocol file can also be performed. The table structure prediction obtains the table structure information in the SIOT protocol and the four-point coordinates of each EXCEL cell, that is, the cell coordinates. Specifically, PP-Structure can be used to recognize tables from pictures. The pictures will first be analyzed by Layout-Parser. In the layout analysis, the areas in the pictures will be classified into 5 categories, including text, titles, pictures, lists and tables. For the first 4 types of areas, PaddleOCR is used directly to complete the text detection and recognition of the corresponding areas. For table-type areas, after table structuring processing, the table image is converted into a structured file of the same table style.

[0094] In the specific implementation, you can use PaddleOCR based on deep learning to perform batch recognition of PDF files. Deep learning OCR uses neural networks to perform deep feature extraction on images, and can handle complex image backgrounds, lighting changes, font changes, etc., improving recognition accuracy. The process of PaddleOCR recognizing PDF is to first convert PDF into a picture, then recognize the picture, and use OCR to recognize the table text in the picture and convert it into csv, json, excel files, etc. The PaddleOCR kit includes: the base paddlepaddle to implement the general framework of deep learning, PaddleOCR to implement image recognition, PP-Structure, paddleclas, among which paddleclas is the component used for classification.

[0095] Step S102: Correct the misidentified information in the initial execl file to obtain the target execl file.

[0096] It is understandable that the text correction module can correct the misidentified information in the initial execl file by implementing tariff extraction based on NLP. The misidentified information can be the garbled information in the initial execl file. And the text content and table information can be accurately extracted. Specifically, the CRASpell model can be used to train the general word and settlement field text correction model to realize the revision of text grammatical errors such as OCR misrecognition, and to a certain extent solve the impact of image noise on OCR recognition. Deep fusion rule matching and deep learning technology mine the information contained in the text, and realize entity extraction, relationship extraction, tariff extraction and other capabilities based on named entity recognition, reference resolution, entity disambiguation, dependency syntax analysis and other technologies. The selected technical architecture includes but is not limited to "Bert+CRF". After correcting the misidentified information in the initial execl file in the above manner, the target execl file can be obtained.

[0097] Step S103: Use the data corresponding to the target execl file as roaming charge data.

[0098] It should be understood that the data corresponding to the target execl file can be used as roaming charge data and stored in the data layer.

[0099] The present embodiment determines the SIOT protocol file according to the international roaming mail, and constructs an initial execl file according to the SIOT protocol file, then corrects the misidentification information in the initial execl file, obtains the target execl file, and then uses the data corresponding to the target execl file as the roaming tariff data. The present embodiment constructs the initial execl file according to the SIOT protocol file, and then corrects the misidentification information in the initial execl file, can obtain a more accurate target execl file, and then uses the data corresponding to the target execl file as the roaming tariff data.

[0100] refer to Figure 7 , Figure 7 This is a flowchart of the third embodiment of the data processing method of the present application.

[0101] Based on the above embodiments, in this embodiment, step S30 includes:

[0102] Step S301: Determine the charging model and the agreement tariff information according to the key atomic information.

[0103] It should be noted that the key atomic information may include a calculation model, and the agreement tariff information may be specific information corresponding to the key atomic information, such as customer code, roaming partner code, expiration time, call type, minimum billing unit, etc.

[0104] Understandably, Figure 4 The data layer in: including StarRocks distributed database and data warehouse, which are used to store special international roaming tariff data information. The introduction of StarRocks for data storage and computing provides excellent performance and speed for data query, easily copes with the growing data scale, and can add new nodes when the data load increases. It is scalable. At the same time, StarRocks has strong real-time analysis capabilities, supports diversified real-time analysis functions for different dimensions, and can quickly process large-scale data. Since the tariff information of the agreement is mostly based on the batch price of the business volume within the entire year according to different billing models, this leads to the need for a database that can effectively store and process large-scale historical data, and the pre-processing of the data will also solve the system pressure to a certain extent.

[0105] It should be understood that the main steps are as follows: Step 1: Real-time synchronization of MySQL source data and archiving and storage; Step 2: Obtaining the agreed tariff information; Step 3: Automatically triggering statistical calculation tasks; Step 4: Preprocessing roaming tariff data to generate aggregated data of different granular dimensions and storing them in separate tables. Step 5: Obtaining aggregated data, calculating the target amount based on the agreed tariff information, which may include the discount amount and refund amount.

[0106] In the specific implementation, based on step 1, the real-time synchronization of source data in the relational database MySQL to the StarRocks analysis library is completed through the Flink tool, including table structure, full data and incremental data; at the same time, the incremental data is synchronized through Kafka to form the finest granularity detailed list, and the data is retained as it is to support long-term massive data management query and processing calculation. When updating data in batches, it is necessary to prevent the avalanche effect from occurring during the synchronization process, which may lead to data congestion. Given that StarRocks is developed based on Doris and is widely praised in the OLAP market for its excellent performance in multi-table join queries, it can effectively solve the avalanche effect problem.

[0107] Step S302: Aggregate the roaming tariff data according to the agreed tariff information to obtain aggregated data of different granularities.

[0108] It is understandable that based on step 4, pre-processing generates aggregated data: since the amount of data within the expiration time of the agreement is very large, it takes a long time to count and calculate the preferential discount price when the agreement expires. This embodiment accumulates the call information to be used in advance by pre-processing the call records. Data processing time can be greatly saved. Specifically, the Apache Flink tool can be used to extract data from various data sources, perform necessary conversions and processing, and then load the data into StarRocks. The processing is mainly based on the protocol tariff information (such as expiration time, minimum billing unit, granularity, etc.) The roaming consumption data is aggregated and stored in four tables with different granularities. Aggregate data.

[0109] Step S303: Calculate a target amount based on the aggregated data, the agreed tariff information, and the discount model, where the target amount includes a discount amount and a refund amount.

[0110] It is understandable that based on step 2, the customer and roaming partner codes are obtained, and the birth and expiration time are used to trigger the statistical calculation task. The minimum billing unit and granular dimension information are obtained for preprocessing the source data, and the available aggregated data is formed according to the processing, including the aggregate table structure, data, etc. The billing model and calculation rules are obtained for calculating the target amount, which may include the discount amount and the refund amount. The above customer and roaming partner codes, birth and expiration time, minimum billing unit and granular dimension information, billing model and calculation rules are all key atomic information.

[0111] It should be understood that based on step 3, the statistical task is triggered according to the customer and roaming partner code and the effective time. The statistical task will obtain the corresponding source data from the effective date, extract the corresponding data and perform pre-processing and pre-pricing. If the tariff information in step 2 changes, it is necessary to trigger the re-statistical calculation task, either the pre-processing stage or the pre-pricing calculation. If the information at the statistical data level such as the customer, roaming partner, and the minimum billing unit changes, it is necessary to re-acquire and extract the corresponding source data and perform aggregation processing; if the information that affects the batch price calculation, such as the billing model and calculation rules, changes, there is no need to re-count the source data, and the data in step 4 can be directly used for batch price calculation. For details, please refer to Figure 8 , Figure 8 This is a schematic diagram of the overall process of calculating the target amount according to an embodiment of the data processing method of the present application.

[0112] In the specific implementation, based on step 5 and the roaming information in step 4, the total business volume is accumulated monthly, and then the corresponding batch price calculation is performed according to the calculation code program of different discount models. It should be noted that if the agreement has a full-year amount commitment, the monthly accumulated batch price amount is not the final discount amount payable. The discount amount payable = Max (discount amount, amount commitment). After calculating the preferential discount amount payable within the agreement period, the difference between the paid TAP amount, that is, the amount that both parties should refund within the agreement period, can be used to calculate the discount amount and refund amount. For details, please refer to Fig. 9 , Fig. 9 This is a detailed flowchart of calculating the target amount according to an embodiment of the data processing method of the present application.

[0113] This embodiment determines the agreed tariff information based on key atomic information, and then aggregates the roaming tariff data based on the agreed tariff information to obtain aggregate data of different granularities, and then calculates the target amount based on the aggregate data, the agreed tariff information, and the discount model, and the target amount includes the discount amount and the refund amount. This embodiment aggregates the roaming tariff data based on the agreed tariff information, and can perform necessary conversion and processing on the roaming tariff data to obtain aggregate data of different granularities, and then calculates the target amount based on the aggregate data, the agreed tariff information, and the discount model, and can automatically and accurately obtain the discount amount and the refund amount.

[0114] Furthermore, in this embodiment, after step S30, it also includes: processing historical business data to obtain processed business data; training an initial prediction model based on the processed business data to obtain a target prediction model; and predicting and analyzing the target amount through the target prediction model to obtain a prediction result.

[0115] It is understandable that this embodiment can also use AI prediction algorithm as the core, integrate business data of different dimensions inside and outside, and be equipped with a single-dimensional and multi-dimensional variable prediction algorithm library and a neural network algorithm library for customized training of the initial prediction model, such as Fig.10 As shown, Fig.10 This is an architecture diagram of the initial prediction model of an embodiment of the data processing method of the present application. The framework diagram of the initial prediction model is mainly divided into a data input layer, a data processing layer, an AI prediction layer, and a functional application layer. The data input layer mainly includes a database and a data warehouse, and the data is stored in the B domain, the O domain, and the M domain according to the location of the source data, and the data warehouse is used to store the data. The data processing layer includes data loading, data extraction, data cleaning, data conversion, data integration, and data standardization to obtain the standardized data required for AI prediction, that is, the processed business data. The AI ​​prediction layer can use algorithms such as time series prediction algorithms (prophet, ARIMA), multivariate time series analysis (Granger causality test), ridge regression (Ridge regression), long short-term memory network (LSTM), etc., to combine multiple algorithms. The functional application layer mainly includes historical monitoring, trend monitoring, abnormal alarm, data analysis, and predicts and analyzes the special tariff business volume and settlement amount of international roaming through the output results of the AI ​​prediction layer. The AI ​​prediction algorithm uses the historical values ​​of business indicators, that is, historical business data, as model input to build a target prediction model, which may include single-variable time series prediction and multi-variable causal prediction models, accurately predict the future values ​​and change trends of business indicators, that is, prediction results, and support fluctuation analysis and causal analysis for indicators. A learning model is established based on historical business data to achieve accurate abnormality detection, explore the periodic laws of special international roaming rates, establish a scientific fluctuation baseline, and achieve business trend prediction. Combined with the Boxplot model, secondary analysis and post-filtering are performed to output abnormal data. This enables rolling prediction of future values ​​of indicators and improves data accuracy.

[0116] For settlement scenarios with more complex tariff models, personalized configuration of tariff verification rules, settlement monitoring and warning thresholds, and business commitment information is used to achieve multi-dimensional settlement management from multiple angles, such as settlement data auditing, fluctuation analysis, and trend forecasting. This is of great significance for ensuring settlement accuracy, resolving international roaming fraud issues, and making roaming optimization decisions.

[0117] In addition, this embodiment can also establish a batch-stream integrated data processing mechanism to charge and settle special international roaming call records. The specific process is as follows: Figure 5As shown. The integrated stream and batch data processing mechanism collects different call bill source data in real time, including incremental call bills and historical call bills, writes the historical call bill source data into the original detailed bill data for storage, and processes the real-time call bill through the transmission queue. The parallel relationship between real-time call bill settlement and historical call bill settlement is adjusted. By first implementing offline SQL, call bill data processing is performed once a day, and then the offline SQL is reused to create the day's call bill data statistics as a View, thereby combining offline and real-time data to achieve stream and batch integration. After the real-time calculation is completed, the settled data is read in batches to write back the results, summarize the results in real time, and finally the result data is batch calculated and written into the batch calculation results. Reasonably select call bill keywords to design summary rules, and perform batch summary calculations on call bills to achieve call bill data condensation. Solve the storage problem, store stream data and batch data in a unified manner, and realize flexible query of call bills. Adopting the integrated stream and batch architecture, a set of SQL can do stream calculations and batch calculations, and process incremental call bills and historical call bill data in real time. Refer to Fig.11 , Fig.11 This is a flow chart of the integrated stream-batch data processing mechanism of an embodiment of the data processing method of the present application.

[0118] Reference Fig.12 , Fig.12 This is a structural block diagram of the first embodiment of the data processing device of the present application.

[0119] like Fig.12 As shown, the data processing device proposed in the embodiment of the present application includes:

[0120] The data acquisition module 10 is used to determine the SIOT protocol file according to the international roaming email, and determine the roaming tariff data according to the SIOT protocol file;

[0121] A model configuration module 20, configured to determine key atomic information according to the roaming tariff data, and configure a discount model according to the key atomic information;

[0122] The amount calculation module 30 is used to calculate the target amount according to the discount model and the roaming fee data.

[0123] This embodiment determines the SIOT protocol file according to the international roaming mail, determines the roaming tariff data according to the SIOT protocol file, then determines the key atomic information according to the roaming tariff data, configures the discount model according to the key atomic information, and then calculates the target amount according to the discount model and the roaming tariff data. This embodiment first automatically identifies and extracts the SIOT protocol file according to the international roaming mail, determines the roaming tariff data according to the SIOT protocol file, further abstracts and defines the roaming tariff data, generates the key atomic information, and then configures the discount model according to the key atomic information, and realizes the rapid settlement of the roaming tariff data through the stop model to obtain the target amount, so that the special international roaming tariff can be automatically and effectively settled.

[0124] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present application. In practical applications, technicians in this field can select part or all of it according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0125] In addition, for technical details not fully described in this embodiment, please refer to the data processing method provided in any embodiment of the present application, and will not be repeated here.

[0126] Based on the above-mentioned first embodiment of the data processing device of the present application, a second embodiment of the data processing device of the present application is proposed.

[0127] In this embodiment, the data acquisition module 10 is also used to determine the SIOT protocol file based on the international roaming mail, and construct an initial execl file based on the SIOT protocol file; correct the misidentified information in the initial execl file to obtain the target execl file; and use the data corresponding to the target execl file as roaming charge data.

[0128] Furthermore, the data acquisition module 10 is also used to determine the text box coordinates, text content, cell coordinates and table structure information according to the SIOT protocol file; determine the cell coordinates according to the text box coordinates and the cell coordinates; determine the cell text according to the text content and the cell coordinates; and construct an initial execl file according to the cell text and the table structure information.

[0129] Furthermore, the data acquisition module 10 is also used to preprocess the SIOT protocol file to obtain a preprocessed protocol file; perform text detection on the preprocessed protocol file to obtain text box coordinates; perform text recognition on the preprocessed protocol file to obtain text content; perform table structure prediction on the SIOT protocol file to obtain cell coordinates and table structure information.

[0130] Furthermore, the amount calculation module 30 is also used to determine the billing model and the agreed tariff information based on the key atomic information; aggregate the roaming tariff data according to the agreed tariff information to obtain aggregate data of different granularities; calculate the target amount based on the aggregate data, the agreed tariff information and the discount model, and the target amount includes a discount amount and a refund amount.

[0131] Furthermore, the amount calculation module 30 is also used to process historical business data to obtain processed business data; train the initial prediction model based on the processed business data to obtain a target prediction model; and perform prediction analysis on the target amount through the target prediction model to obtain a prediction result.

[0132] Other embodiments or specific implementations of the data processing device of the present application can refer to the above-mentioned method embodiments and will not be described in detail here.

[0133] The present application provides a data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method in the above-mentioned embodiment 1.

[0134] Reference below Fig.13 , which shows a schematic diagram of the structure of a data processing device suitable for implementing an embodiment of the present application. The data processing device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.13 The data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0135] like Fig.13As shown, the data processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a data processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0136] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0137] The data processing device provided by the present application, using the data processing method in the above embodiment, can solve the technical problem of how to automatically and effectively settle the special charges for international roaming. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the method of the previous embodiment, which will not be described in detail here.

[0138] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0139] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0140] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data processing method in the above-mentioned embodiment.

[0141] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0142] The computer-readable storage medium may be included in a data processing device, or may exist independently without being incorporated into a data processing device.

[0143] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a data processing device, the data processing device: determines the SIOT protocol file according to the international roaming mail, and determines the roaming tariff data according to the SIOT protocol file; determines the key atomic information according to the roaming tariff data, and configures the discount model according to the key atomic information; calculates the target amount according to the discount model and the roaming tariff data.

[0144] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0147] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data processing method, and can solve the technical problem of how to automatically and effectively settle the special charges for international roaming. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the data processing method provided in the above-mentioned embodiment, and will not be described in detail here.

[0148] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data processing method when executed by a processor.

[0149] The computer program product provided by the present application can solve the technical problem of how to automatically and effectively settle special international roaming charges. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, which will not be repeated here.

[0150] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: The data processing method comprises the following steps: Determine a SIOT protocol document according to the international roaming email, and determine roaming tariff data according to the SIOT protocol document; Determining key atomic information according to the roaming tariff data, and configuring a discount model according to the key atomic information; A target amount is calculated according to the discount model and the roaming tariff data.

2. The data processing method according to claim 1, characterized in that: The step of determining the SIOT protocol file according to the international roaming email, and determining the roaming tariff data according to the SIOT protocol file specifically includes: Determine a SIOT protocol file according to the international roaming mail, and construct an initial execl file according to the SIOT protocol file; Correcting the misidentified information in the initial execl file to obtain a target execl file; The data corresponding to the target execl file is used as roaming charge data.

3. The data processing method according to claim 2, characterized in that: The step of constructing the initial execl file according to the SIOT protocol file specifically includes: Determine text box coordinates, text content, cell coordinates and table structure information according to the SIOT protocol file; Determine the cell coordinates according to the text box coordinates and the cell coordinates; Determine the cell text according to the text content and the cell coordinates; Construct an initial execl file according to the cell text and the table structure information.

4. The data processing method according to claim 3, characterized in that: The step of determining text box coordinates, text content, cell coordinates and table structure information according to the SIOT protocol file specifically includes: Preprocessing the SIOT protocol file to obtain a preprocessed protocol file; Performing text detection on the preprocessed protocol file to obtain text box coordinates; Performing text recognition on the preprocessed protocol file to obtain text content; The table structure of the SIOT protocol file is predicted to obtain cell coordinates and table structure information.

5. The data processing method according to claim 1, characterized in that: The step of calculating the target amount according to the discount model and the roaming tariff data specifically includes: Determine the charging model and the agreed tariff information according to the key atomic information; Aggregating the roaming tariff data according to the agreed tariff information to obtain aggregated data of different granularities; A target amount is calculated according to the aggregated data, the agreed tariff information and the discount model, wherein the target amount includes a discount amount and a refund amount.

6. The data processing method according to any one of claims 1 to 5, characterized in that: After the step of calculating the target amount according to the discount model and the roaming tariff data, the method further includes: Process historical business data to obtain processed business data; Training the initial prediction model according to the processed business data to obtain a target prediction model; The target amount is predicted and analyzed by the target prediction model to obtain a prediction result.

7. A data processing device, characterized in that: The data processing device comprises: A data acquisition module, used to determine a SIOT protocol file according to an international roaming email, and determine roaming tariff data according to the SIOT protocol file; A model configuration module, used to determine key atomic information according to the roaming tariff data, and configure a discount model according to the key atomic information; The amount calculation module is used to calculate the target amount according to the discount model and the roaming fee data.

8. A data processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

Images