Data synchronization method, device and equipment of advertisement putting platform and medium

Abstract

The embodiment of the invention discloses a data synchronization method and device for an advertisement putting platform, equipment and a medium. The method comprises the steps that a big data scheduling component imports multiple pieces of equipment sub-data stored in a Hive data warehouse into an FTP file server, so that the multiple pieces of equipment sub-data are converted into FTP big files; the file cutting component is used for cutting the FTP large file to obtain a plurality of sub-files; the file processing component creates a plurality of analysis threads for the plurality of sub-files, creates a corresponding relationship between each analysis thread and the sub-file, performs downloading analysis processing on the plurality of sub-files through the plurality of analysis threads based on the corresponding relationship, and stores data obtained after downloading analysis into a first message queue; and the data synchronization component synchronizes the data in the first message queue to a memory database in the advertisement putting system according to a preset synchronization data volume. According to the scheme, massive equipment sub-data can be synchronized to the memory database in the advertisement putting platform.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a data synchronization method, device, equipment and medium for an advertising delivery platform. Background Art

[0002] During the advertising process, the media platform screens out suitable traffic for the advertiser based on the target audience requirements provided by the advertiser, and synchronizes the screened traffic information to the advertiser through the Application Programming Interface (API); after receiving these traffic requests, the advertiser will determine the value of the traffic based on its own device classification model and decide whether it is needed; if necessary, it will be returned to the media through the API to participate in bidding exposure; if not, it will return information about not participating in bidding or not placing advertisements.

[0003] In order to improve the decision-making accuracy of advertisers, the device classification model on the advertiser side needs to be updated frequently. To update the device classification model, the newly collected device classification information needs to be synchronized to the memory database corresponding to the device classification model. However, the device classification data that needs to be synchronized into the memory database at a time is as high as hundreds of millions. How to automatically synchronize massive data into the memory database is an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a data synchronization method, apparatus, device and medium for an advertising delivery platform, which can synchronize massive amounts of device data to a memory database in the advertising delivery platform.

[0005] In a first aspect, an embodiment of the present application provides a data synchronization method for an advertising delivery platform, the method being applied to the advertising delivery platform, the advertising delivery platform comprising a data synchronization system and an advertising delivery system, the data synchronization system comprising a big data scheduling component, a file cutting component, a file processing component, a data synchronization component and a message middleware component, the method comprising:

[0006] The big data scheduling component imports the plurality of device score data stored in the Hive data warehouse into a file transfer protocol (FTP) file server to convert the plurality of device scores in the Hive data warehouse into a FTP large file;

[0007] The file cutting component cuts the large FTP file in the FTP file server to obtain a plurality of sub-files;

[0008] The file processing component creates multiple parsing threads for the multiple sub-files and creates a corresponding relationship between each parsing thread and the sub-files, and based on the corresponding relationship, performs download and parsing processing on the multiple sub-files through the multiple parsing threads, and stores the data obtained after downloading and parsing in the first message queue of the message middleware component;

[0009] The data synchronization component synchronizes the data in the first message queue to the memory database in the advertising delivery system according to a preset synchronization data volume, so that the device sub-model in the advertising delivery system is updated through the updated memory database.

[0010] In a second aspect, an embodiment of the present application further provides an advertising delivery platform, the advertising delivery platform comprising a data synchronization system and an advertising delivery system, the data synchronization system comprising a big data scheduling component, a file cutting component, a file processing component, a data synchronization component and a message middleware component, wherein:

[0011] The big data scheduling component is used to import the multiple device score data stored in the Hive data warehouse of the advertising delivery system into the FTP file server to convert the multiple device scores in the Hive data warehouse into FTP large files;

[0012] The file cutting component is used to cut the large FTP file in the FTP file server to obtain multiple sub-files;

[0013] The file processing component is used to create multiple parsing threads for the multiple sub-files and to create a corresponding relationship between each parsing thread and the sub-files, and based on the corresponding relationship, download and parse the multiple sub-files through the multiple parsing threads, and store the data obtained after downloading and parsing in the first message queue of the message middleware component;

[0014] The data synchronization component is used to synchronize the data in the first message queue to the memory database in the advertising delivery system according to a preset synchronization data volume, so that the device sub-model in the advertising delivery system is updated through the updated memory database.

[0015] In a third aspect, an embodiment of the present application further provides a computer device, wherein an advertising delivery platform is deployed in the computer device, wherein the advertising delivery platform includes a data synchronization system and an advertising delivery system, wherein the data synchronization system includes a big data scheduling component, a file cutting component, a file processing component, a data synchronization component and a message middleware component, wherein the computer device includes a memory and a processor, wherein a computer program is stored in the memory, and wherein the above method is implemented when the processor executes the computer program.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and wherein the program instructions can implement the above method when executed by a processor.

[0017] The embodiment of the present application provides a data synchronization method, device, equipment and medium for an advertising delivery platform. On the one hand, when synchronizing massive device sub-data to a memory database, in order to solve the problem that data in a Hive data warehouse cannot be cut and processed, the present application imports massive device sub-data in the Hive data warehouse into an FTP file server, generates a large FTP file corresponding to the massive device sub-data, and then cuts and processes the FTP large file through a preset file cutting component to achieve the cutting of massive device sub-data, and finally synchronizes the multiple sub-files obtained after the segmentation into the memory database through multiple parsing threads, thereby automatically synchronizing the massive device sub-data to the memory database in the advertising delivery platform; on the other hand, after the multiple parsing threads in the present application download and parse the multiple sub-files, the data obtained after the parsing will first be stored in the first message queue of the message middleware component, and the data in the first message queue will be synchronized to the memory database according to the preset synchronization quantity through the data synchronization component. It can be seen that through the embodiment of the present application, the decoupling of sub-file download parsing and data synchronization logic is realized, and the first message queue can achieve the effect of synchronous data peak shaving, avoiding the problem of memory database crash caused by too high synchronization traffic, and improving the stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A schematic diagram of a flow chart of a data synchronization method for an advertisement delivery platform provided in an embodiment of the present application;

[0020] Figure 2 A flowchart of a data synchronization method for an advertisement delivery platform provided in another embodiment of the present application;

[0021] Figure 3 A schematic block diagram of an advertisement delivery platform provided in an embodiment of the present application;

[0022] Figure 4 A schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0025] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0026] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0027] The embodiments of the present application provide a data synchronization method, device, equipment and medium for an advertisement delivery platform.

[0028] The executor of the data synchronization method of the advertising delivery platform can be the advertising delivery platform provided in the embodiment of the present application, or a computer device deployed with the advertising delivery platform, wherein the advertising delivery platform includes a data synchronization system and an advertising delivery system, and the data synchronization system includes a big data scheduling component, a file cutting component, a file processing component, a data synchronization component and a message middleware component.

[0029] Figure 1 1 is a flow chart of a data synchronization method for an advertising delivery platform provided in an embodiment of the present application. Figure 1 As shown, the method includes the following steps S110-S140.

[0030] S110. The big data scheduling component imports multiple device score data stored in the Hive data warehouse of the advertising delivery system into the FTP file server to convert the multiple device scores in the Hive data warehouse into FTP large files.

[0031] In this embodiment, the Hive data warehouse stores a large amount of device segment data to be synchronized into the memory database, and the device segment data collected by the collection system is stored in the Hive data warehouse, so the device segment data in the Hive data warehouse will be frequently updated.

[0032] The decision basis of the device sub-model in the advertisement delivery system of this embodiment depends on the device sub-model in the memory database, so updating the device sub-data in the memory database is to update the device sub-model.

[0033] In order to improve the accuracy of advertisers' decisions, it is necessary to synchronize the device classification data in the Hive data warehouse to the in-memory database according to a pre-set update cycle (for example, update once every day at 0:00 am) to update the device classification model.

[0034] Among them, the big data scheduling component in the present application can be specifically a big data task scheduling tool DolphinScheduler, and the memory database in the embodiment of the present application can be a Redis database.

[0035] In some embodiments, a scheduled task is preset in the big data scheduling component in this embodiment, and the above-mentioned update cycle is executed through the preset scheduled task.

[0036] In this embodiment, since the device segment data in the Hive data warehouse is in Hive format, the data in Hive format cannot be cut, so this application imports the device segment data in the Hive data warehouse into the FTP file server, and converts the device segment data in Hive format in the Hive data warehouse into FTP format to obtain a large FTP file. The FTP large file supports file cutting, thereby solving the problem that the massive device segment data in the Hive data warehouse cannot be cut.

[0037] In some embodiments, the Hive data warehouse and the FTP file server are located outside the advertising delivery platform. The big data scheduling component can import the massive device data in the Hive data warehouse into the FTP file server, the file cutting component can access the FTP file server and cut the FTP large file in the FTP file server, and the file processing component can download and parse the cut sub-files in the FTP file server.

[0038] Among them, the device score data in this embodiment includes a device identifier and at least one user tag corresponding to the device identifier. The device score model can calculate the device score corresponding to the device identifier based on the user tag, and the advertising delivery strategy in the advertising delivery system generates a delivery result (delivery or not) based on the device score.

[0039] S120: The file cutting component cuts the large FTP file in the FTP file server to obtain a plurality of sub-files.

[0040] In this embodiment, after converting the multiple device scores in the Hive data warehouse into a large FTP file, the large FTP file in the FTP file server is cut by a file cutting component to obtain multiple sub-files.

[0041] Furthermore, in order to timely cut the large FTP files in the FTP file server, after the big data scheduling component in this embodiment completely imports the multiple device segment data stored in the Hive data warehouse into the FTP file server, the big data scheduling component will send an import completion message to the third message queue of the message middleware component, so that the file cutting component obtains the import completion message through the third message queue; when the file cutting component obtains the import completion message, the file cutting component queries whether the FTP large file exists in the FTP file server according to a preset time interval within a preset time length after obtaining the import completion message.

[0042] Specifically, the file cutting component listens for DS MQ messages (import completion messages) by subscribing to the third message queue of the message middleware component. When the DS MQ message is listened to, a large file polling task is immediately initiated. The large file polling task indicates that within a preset time length (for example, 5 minutes) of listening to the DS MQ message, the FTP file server is accessed at certain time intervals until a large FTP file is detected to be generated in the FTP file server. At this time, it means that the massive device segmentation data imported into the Hive data warehouse has been successfully converted into a large FTP file. When the FTP large file is detected to exist in the FTP file server, the large FTP file in the FTP file server is immediately cut according to the preset segmentation configuration. The file cutting component can use a shell script to cut the large FTP file.

[0043] The preset segmentation configuration includes sub-file size configuration or sub-file quantity configuration.

[0044] In some embodiments, when the file cutting component fails to find the existence of the FTP large file in the FTP file server within the preset time period, a file conversion failure warning is generated.

[0045] In addition, the data synchronization system in the present application is also provided with a retry mechanism, through which it is possible to periodically determine whether there is an abnormality in the cutting operation in the FTP file server. Specifically, it is possible to determine whether there is an abnormality in the cutting operation by accessing the cutting log of the file cutting component. If an abnormality occurs, the file cutting component is pulled up to re-perform the cutting operation on the large FTP file in the FTP file server.

[0046] S130, the file processing component creates multiple parsing threads for the multiple sub-files and creates a corresponding relationship between each parsing thread and the sub-files, and based on the corresponding relationship, downloads and parses the multiple sub-files through the multiple parsing threads, and stores the data obtained after downloading and parsing in the first message queue of the message middleware component.

[0047] Specifically, the file processing component of this embodiment is preset with a parsing thread creation logic, and the parsing thread creation logic indicates that one parsing thread is created for every n sub-files. For example, there are m sub-files in total. At this time, the number of parsing threads to be created is m / n, where n is an integer greater than 0. The value can be modified according to actual needs. The specific value of n is not limited in this embodiment, and m is the total number of sub-files after cutting.

[0048] After creating a plurality of parsing threads, a corresponding relationship between each of the parsing threads and the sub-files is created through the load balancing rule. Through the corresponding relationship, it can be clearly determined which parsing thread is to download and parse those sub-files.

[0049] In this embodiment, multiple parsing threads download sub-files at the same time, which can improve the file download efficiency and thus improve the data synchronization efficiency. In addition, each parsing thread also parses the downloaded sub-files, such as parsing the sub-file data and assembling it into Redis data. In addition, in order to avoid mutual influence between the parsing threads and the data synchronization components and to reasonably allocate service resources, this embodiment will first store the data obtained after the download and parsing in the first message queue of the message middleware component, rather than performing synchronization processing immediately.

[0050] In some embodiments, in order to improve the stability of the system, the data synchronization system further includes a resource protection component. Before creating the parsing thread, the method further includes:

[0051] The resource protection component determines whether the current remaining service resource amount of the data synchronization system can meet the required service resource amount of the multiple sub-files, and determines whether the remaining memory amount of the memory database in the advertisement delivery system can meet the required memory amount of the multiple sub-files;

[0052] At this time, the file processing component creates multiple parsing threads for the multiple sub-files and creates a corresponding relationship between each parsing thread and the sub-file, including:

[0053] If the current remaining service resource amount meets the required service resource amount, and the remaining memory amount meets the required memory amount, the file processing component creates a plurality of the parsing threads for a plurality of the sub-files and creates the corresponding relationships.

[0054] That is, the creation operation of the parsing thread is performed only when the service resources of the data synchronization system and the storage resources of the memory database in the advertisement delivery system meet the current synchronization requirements.

[0055] Among them, if the current remaining service resource amount does not meet the required service resource amount, or the remaining memory amount does not meet the required memory amount, the resource protection component activates the fuse mechanism and / or generates an insufficient memory warning, thereby avoiding system abnormalities caused by insufficient resources for data synchronization operations.

[0056] In some other embodiments, before the file processing component creates multiple parsing threads for the multiple sub-files and creates a corresponding relationship between each of the parsing threads and the sub-files, the method further includes:

[0057] The resource protection component determines whether the current remaining service resource amount of the data synchronization system can meet the required service resource amount of at least one of the sub-files, and determines whether the remaining memory amount of the memory database in the advertisement delivery system can meet the required memory amount of multiple sub-files;

[0058] The file processing component creates multiple parsing threads for the multiple sub-files and creates a corresponding relationship between each of the parsing threads and the sub-files, and based on the corresponding relationship, performs download and parsing processing on the multiple sub-files through the multiple parsing threads, and stores the data obtained after downloading and parsing in the first message queue of the message middleware component, including:

[0059] If the current remaining service resource amount satisfies the required service resource amount of at least one of the sub-files, and the remaining memory amount satisfies the required memory amount, the file processing component creates at least one parsing thread for at least one of the sub-files based on the current remaining service resource amount and creates the corresponding relationship;

[0060] Based on the corresponding relationship, at least one sub-file is downloaded and parsed by at least one parsing thread currently created, the data obtained after the download and parsing is stored in the first message queue of the message middleware component, and the corresponding service resource amount is released;

[0061] Determine whether the plurality of sub-files have been parsed;

[0062] If the parsing of the plurality of sub-files is not completed, the process returns to the step of creating at least one parsing thread for at least one sub-file based on the current remaining service resource amount and creating the corresponding relationship until the parsing of the plurality of sub-files is completed.

[0063] Compared with the previous embodiment, this embodiment only needs to determine whether the current remaining service resource amount meets the required service resource amount of at least one of the sub-files, and when the remaining memory amount meets the required memory amount, a parsing thread can be created. The number of parsing threads created is the same as the remaining service resource amount to process the corresponding number of sub-files.

[0064] In some embodiments, in order to improve the data synchronization efficiency, after the file cutting component cuts the large FTP file in the FTP file server to obtain a plurality of sub-files, the method further includes:

[0065] The file cutting component sends a cutting completion message to the second message queue of the message middleware component;

[0066] Specifically, after the file cutting component completes the cutting operation, it immediately sends a cutting completion message to the resource protection component through the second message queue. At this time, after the resource protection component obtains the cutting completion message through the second message queue, the resource protection component determines whether the current remaining service resources can meet the required service resources, and determines whether the remaining memory meets the required memory.

[0067] Furthermore, after the downloaded and parsed data is stored in the first message queue of the message middleware component, the system will record the amount of data sent by the first message queue to the memory database, and obtain the number of successful synchronizations on the consumer side (memory database), and determine whether the data is fully synchronized to the memory database based on the number sent and the number of successful synchronizations, and update the corresponding task status.

[0068] If the full synchronization is not successful, the data that was not synchronized successfully needs to be resent in the next data sending cycle to ensure the stability of the system.

[0069] S140. The data synchronization component synchronizes the data in the first message queue to the memory database in the advertising delivery system according to a preset synchronization data volume, so that the device sub-model in the advertising delivery system is updated through the updated memory database.

[0070] In this embodiment, the data synchronization component synchronizes the data in the first message queue to the memory database in batches, which can improve data synchronization efficiency and reduce system throughput resource consumption. In addition, the first message queue can be used to achieve the effect of peak shaving of synchronized data, thereby preventing the memory database from crashing directly due to excessive synchronization traffic, thereby affecting online real-time business.

[0071] Furthermore, the data synchronization component not only synchronizes data according to the preset synchronization data volume, but also sets a specified time in the data synchronization component in consideration of the problem of off-peak synchronization. The data synchronization component can only synchronize data within the specified time.

[0072] To summarize, on the one hand, when synchronizing massive device segment data to the memory database, in order to solve the problem that the data in the Hive data warehouse cannot be cut and processed, the present application imports the massive device segment data in the Hive data warehouse into the FTP file server, generates a large FTP file corresponding to the massive device segment data, and then cuts and processes the FTP large file through a preset file cutting component to achieve the cutting of massive device segment data. Finally, multiple sub-files obtained after segmentation are synchronized into the memory database through multiple parsing threads, thereby automatically synchronizing the massive device segment data to the memory database in the advertising delivery platform; on the other hand, after the multiple parsing threads in the present application download and parse the multiple sub-files, the data obtained after parsing will be first stored in the first message queue of the message middleware component, and the data in the first message queue will be synchronized to the memory database according to the preset synchronization quantity through the data synchronization component. It can be seen that through the embodiment of the present application, the decoupling of sub-file download parsing and data synchronization logic is realized, and the first message queue can achieve the effect of synchronous data peak shaving, avoiding the problem of memory database crash caused by too high synchronization traffic, and improving the stability of the system.

[0073] Figure 2 FIG. 1 is a flow chart of a data synchronization method for an advertising delivery platform provided by another embodiment of the present application. Figure 2 As shown, the data synchronization method of the advertising delivery platform of this embodiment includes steps S210-S260. Steps S210-S240 are similar to steps S110-S140 in the above embodiment, and will not be repeated here. The following is a detailed description of steps S250-S260 added in this embodiment.

[0074] S250: When receiving an advertisement delivery request carrying a target device identifier from a requesting party, the advertisement delivery system determines a target device score corresponding to the target device identifier by using the device score model.

[0075] In this embodiment, the requesting party is a media party that can deliver advertisements, such as Tencent, Baidu, Douyin and other applications. When the requesting party currently has an advertisement slot available for delivery in the device terminal corresponding to the target device identifier, the requesting party will send an advertisement delivery request carrying the target device identifier to the advertisement delivery platform, and the advertisement delivery system in the advertisement delivery platform will determine whether to deliver the advertisement in the device terminal corresponding to the target device identifier.

[0076] Specifically, in order to determine whether it is worthwhile to place an advertisement in the device terminal corresponding to the target device identifier, the advertising delivery system will obtain the target device classification data corresponding to the target device identifier from the memory database according to the device classification model. The target device classification data includes the target device identifier and at least one target user tag corresponding to the target device identifier. For example, the target user tag is: gender - male, age - 24, whether it is a registered user - yes, favorite electronic products - xx mobile phone.

[0077] Then, the device scoring model determines the target device score according to the preset scoring rules and each of the target user tags. Specifically, different scores are set for different types of user tags in the scoring rules. The scores include positive scores or negative scores. The scores corresponding to different user tags are added together to obtain the target device score.

[0078] Since the device score data in the in-memory database that the device score model in this application relies on will be updated by regularly synchronizing the data in the Hive data warehouse, the device score data in the Hive data warehouse will become richer and richer over time, thereby improving the accuracy of the device score calculated by the device score model and further improving the effectiveness of advertising.

[0079] S260: The advertisement delivery system determines a target delivery result according to a preset advertisement delivery strategy and the target device score, and returns the target delivery result to the requesting party.

[0080] Specifically, in some embodiments, the advertisement delivery strategy stipulates that when the target device score is greater than or equal to a preset score threshold, the application for delivery is taken as the target delivery result, otherwise, no delivery is taken as the target delivery result.

[0081] Among them, this embodiment can return the target delivery result to the requester only when applying for delivery. If it is decided not to deliver, the target delivery result may not be returned to the requester.

[0082] Figure 3 is a schematic block diagram of an advertising delivery platform provided in an embodiment of the present application. Figure 3As shown, corresponding to the data synchronization method of the above advertising delivery platform, the present application also provides an advertising delivery platform 30. The advertising delivery platform 30 includes a data synchronization system 31 and an advertising delivery system 32, and the data synchronization system 31 includes a big data scheduling component 311, a file cutting component 312, a file processing component 313, a data synchronization component 314 and a message middleware component 315, wherein:

[0083] The big data scheduling component 311 is used to import the multiple device score data stored in the Hive data warehouse of the advertising delivery system into the FTP file server to convert the multiple device scores in the Hive data warehouse into FTP large files;

[0084] The file cutting component 312 is used to cut the large FTP file in the FTP file server to obtain multiple sub-files;

[0085] The file processing component 313 is used to create multiple parsing threads for the multiple sub-files and to create a corresponding relationship between each parsing thread and the sub-files, and based on the corresponding relationship, download and parse the multiple sub-files through the multiple parsing threads, and store the data obtained after downloading and parsing in the first message queue of the message middleware component 315;

[0086] The data synchronization component 314 is used to synchronize the data in the first message queue to the memory database in the advertising delivery system 32 according to a preset synchronization data volume, so that the device sub-model in the advertising delivery system 32 is updated through the updated memory database.

[0087] In some embodiments, the data synchronization system 31 further includes a resource protection component;

[0088] The resource protection component is used to determine whether the current remaining service resource amount of the data synchronization system 31 can meet the required service resource amount of at least one of the multiple sub-files, and to determine whether the remaining memory amount of the memory database in the advertisement delivery system 32 can meet the required memory amount of multiple sub-files;

[0089] At this time, when the file processing component 313 executes the steps of creating multiple parsing threads for the multiple sub-files and creating a corresponding relationship between each of the parsing threads and the sub-files, and based on the corresponding relationship, downloading and parsing the multiple sub-files through the multiple parsing threads, and storing the downloaded and parsed data in the first message queue of the message middleware component, it is specifically used to:

[0090] If the current remaining service resource amount satisfies the required service resource amount of at least one of the sub-files, and the remaining memory amount satisfies the required memory amount, then creating at least one parsing thread for at least one of the sub-files based on the current remaining service resource amount and creating the corresponding relationship;

[0091] Based on the corresponding relationship, at least one sub-file is downloaded and parsed by at least one parsing thread currently created, the data obtained after the download and parsing is stored in the first message queue of the message middleware component, and the corresponding service resource amount is released;

[0092] Determine whether the plurality of sub-files have been parsed;

[0093] If the parsing of the plurality of sub-files is not completed, the process returns to the step of creating at least one parsing thread for at least one sub-file based on the current remaining service resource amount and creating the corresponding relationship until the parsing of the plurality of sub-files is completed.

[0094] In some embodiments, the file cutting component 312 is further used to:

[0095] Sending a cutting completion message to the second message queue of the message middleware component 315;

[0096] At this time, the resource protection component is specifically used to determine whether the current remaining service resources can meet the required service resources and whether the remaining memory meets the required memory after the resource protection component obtains the cutting completion message through the second message queue.

[0097] In some embodiments, the resource protection component is further configured to:

[0098] If the current remaining service resource amount does not meet the required service resource amount, or the remaining memory amount does not meet the required memory amount, a fuse mechanism is initiated and / or an insufficient memory warning is generated.

[0099] In some embodiments, the big data scheduling component 311 is further used to send an import completion message to the third message queue of the message middleware component 315, so that the file cutting component 312 obtains the import completion message through the third message queue;

[0100] When the file cutting component 312 obtains the import completion message, the file cutting component 312 queries whether the FTP large file exists in the FTP file server according to a preset time interval within a preset time length after obtaining the import completion message;

[0101] At this time, the file cutting component 312 is specifically used for:

[0102] When the file cutting component 312 finds that the FTP large file exists in the FTP file server within the preset time period, the file cutting component 312 cuts the FTP large file in the FTP file server according to the preset segmentation configuration to obtain multiple sub-files.

[0103] In some embodiments, the file cutting component 312 is further used to:

[0104] When the FTP large file is not found in the FTP file server within the preset time period, a file conversion failure warning is generated.

[0105] In some embodiments, the advertising delivery system 32 is used to determine the target device score corresponding to the target device identifier through the device score model when receiving an advertising delivery request carrying a target device identifier sent by a requester; determine the target delivery result according to a preset advertising delivery strategy and the target device score, and return the target delivery result to the requester.

[0106] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned advertising delivery platform and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.

[0107] The above-mentioned advertisement delivery platform can be implemented in the form of a computer program, which can be used in Figure 4 Runs on the computer device shown.

[0108] See also Figure 4 , Figure 4 4 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 400 may be a terminal or a server, wherein an advertisement delivery platform is deployed in the computer device 400, the advertisement delivery platform includes a data synchronization system and an advertisement delivery system, and the data synchronization system includes a big data scheduling component, a file cutting component, a file processing component, a data synchronization component, and a message middleware component.

[0109] See also Figure 4 The computer device 400 includes a processor 402 , a memory and a network interface 405 connected via a system bus 401 , wherein the memory may include a non-volatile storage medium 403 and an internal memory 404 .

[0110] The non-volatile storage medium 403 can store an operating system 4031 and a computer program 4032. The computer program 4032 includes program instructions, and when the program instructions are executed, the processor 402 can execute a data synchronization method for an advertisement delivery platform.

[0111] The processor 402 is used to provide computing and control capabilities to support the operation of the entire computer device 400 .

[0112] The internal memory 404 provides an environment for the operation of the computer program 4032 in the non-volatile storage medium 403. When the computer program 4032 is executed by the processor 402, the processor 402 can execute a data synchronization method for an advertisement delivery platform.

[0113] The network interface 405 is used to communicate with other devices over the network. Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 400 to which the solution of the present application is applied. The specific computer device 400 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0114] The processor 402 is used to run the computer program 4032 stored in the memory to implement the following steps:

[0115] Control the big data scheduling component to import the multiple device score data stored in the Hive data warehouse of the advertising delivery system into the FTP file server, so as to convert the multiple device scores in the Hive data warehouse into FTP large files;

[0116] Controlling the file cutting component to cut the large FTP file in the FTP file server to obtain a plurality of sub-files;

[0117] Control the file processing component to create multiple parsing threads for the multiple sub-files and create a corresponding relationship between each parsing thread and the sub-files, and based on the corresponding relationship, perform download and parsing processing on the multiple sub-files through the multiple parsing threads, and store the data obtained after downloading and parsing in the first message queue of the message middleware component;

[0118] The data synchronization component is controlled to synchronize the data in the first message queue to the memory database in the advertising delivery system according to a preset synchronization data volume, so that the device sub-model in the advertising delivery system is updated through the updated memory database.

[0119] It should be understood that in the embodiment of the present application, the processor 402 may be a central processing unit (CPU), and the processor 402 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0120] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0121] Therefore, the present application also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the following steps:

[0122] Control the big data scheduling component to import the multiple device score data stored in the Hive data warehouse of the advertising delivery system into the FTP file server, so as to convert the multiple device scores in the Hive data warehouse into FTP large files;

[0123] Controlling the file cutting component to cut the large FTP file in the FTP file server to obtain a plurality of sub-files;

[0124] Control the file processing component to create multiple parsing threads for the multiple sub-files and create a corresponding relationship between each parsing thread and the sub-files, and based on the corresponding relationship, perform download and parsing processing on the multiple sub-files through the multiple parsing threads, and store the data obtained after downloading and parsing in the first message queue of the message middleware component;

[0125] The data synchronization component is controlled to synchronize the data in the first message queue to the memory database in the advertising delivery system according to a preset synchronization data volume, so that the device sub-model in the advertising delivery system is updated through the updated memory database.

[0126] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0127] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0128] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0129] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0130] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0131] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data synchronization method for an advertising delivery platform, characterized in that: The method is applied to an advertisement delivery platform, the advertisement delivery platform includes a data synchronization system and an advertisement delivery system, the data synchronization system includes a big data scheduling component, a file cutting component, a file processing component, a data synchronization component and a message middleware component, and the method includes: The big data scheduling component imports the multiple device score data stored in the Hive data warehouse of the advertising delivery system into the FTP file server to convert the multiple device scores in the Hive data warehouse into FTP large files; The file cutting component cuts the large FTP file in the FTP file server to obtain a plurality of sub-files; The file processing component creates multiple parsing threads for the multiple sub-files and creates a corresponding relationship between each parsing thread and the sub-files, and based on the corresponding relationship, performs download and parsing processing on the multiple sub-files through the multiple parsing threads, and stores the data obtained after downloading and parsing in the first message queue of the message middleware component; The data synchronization component synchronizes the data in the first message queue to the memory database in the advertising delivery system according to a preset synchronization data volume, so that the device sub-model in the advertising delivery system is updated through the updated memory database.

2. The method according to claim 1, characterized in that The data synchronization system further includes a resource protection component; before the file processing component creates a plurality of parsing threads for the plurality of sub-files and creates a corresponding relationship between each of the parsing threads and the sub-files, the method further includes: The resource protection component determines whether the current remaining service resource amount of the data synchronization system can meet the required service resource amount of at least one of the sub-files, and determines whether the remaining memory amount of the memory database in the advertisement delivery system can meet the required memory amount of multiple sub-files; The file processing component creates multiple parsing threads for the multiple sub-files and creates a corresponding relationship between each of the parsing threads and the sub-files, and based on the corresponding relationship, performs download and parsing processing on the multiple sub-files through the multiple parsing threads, and stores the data obtained after downloading and parsing in the first message queue of the message middleware component, including: If the current remaining service resource amount satisfies the required service resource amount of at least one of the sub-files, and the remaining memory amount satisfies the required memory amount, the file processing component creates at least one parsing thread for at least one of the sub-files based on the current remaining service resource amount and creates the corresponding relationship; Based on the corresponding relationship, at least one sub-file is downloaded and parsed by at least one parsing thread currently created, the data obtained after the download and parsing is stored in the first message queue of the message middleware component, and the corresponding service resource amount is released; Determine whether the plurality of sub-files have been parsed; If the parsing of the plurality of sub-files is not completed, the process returns to the step of creating at least one parsing thread for at least one sub-file based on the current remaining service resource amount and creating the corresponding relationship until the parsing of the plurality of sub-files is completed.

3. The method according to claim 2, characterized in that After the file cutting component cuts the large FTP file in the FTP file server to obtain a plurality of sub-files, the method further includes: The file cutting component sends a cutting completion message to the second message queue of the message middleware component; The resource protection component determines whether the current remaining service resource amount of the data synchronization system can meet the required service resource amount of the plurality of sub-files, and determines whether the remaining memory amount of the memory database in the advertisement delivery system can meet the required memory amount of the plurality of sub-files, including: After the resource protection component obtains the cutting completion message through the second message queue, the resource protection component determines whether the current remaining service resource amount can meet the required service resource amount, and determines whether the remaining memory amount meets the required memory amount.

4. The method according to claim 2, characterized in that: After the resource protection component determines whether the current remaining service resource amount of the data synchronization system can meet the required service resource amount of the plurality of sub-files, and determines whether the remaining memory amount of the memory database in the advertisement delivery system can meet the required memory amount of the plurality of sub-files, the method further includes: If the current remaining service resource amount does not meet the required service resource amount, or the remaining memory amount does not meet the required memory amount, the resource protection component activates a fuse mechanism and / or generates an insufficient memory warning.

5. The method according to claim 1, characterized in that After the big data scheduling component imports the multiple device sub-data stored in the Hive data warehouse of the advertisement delivery system into the FTP file server, the method further includes: The big data scheduling component sends an import completion message to the third message queue of the message middleware component, so that the file cutting component obtains the import completion message through the third message queue; When the file cutting component obtains the import completion message, the file cutting component queries whether the FTP large file exists in the FTP file server according to a preset time interval within a preset time length after obtaining the import completion message; The file cutting component cuts the large FTP file in the FTP file server to obtain a plurality of sub-files, including: When the file cutting component finds that the FTP large file exists in the FTP file server within the preset time period, the file cutting component cuts the FTP large file in the FTP file server according to the preset segmentation configuration to obtain multiple sub-files.

6. The method according to claim 5, characterized in that After the file cutting component queries the FTP file server whether the FTP large file exists according to a preset time interval within a preset time length after obtaining the import completion message, the method further includes: When the file cutting component fails to find that the FTP large file exists in the FTP file server within the preset time period, a file conversion failure warning is generated.

7. The method according to claim 1, characterized in that The method further comprises: When receiving an advertisement delivery request carrying a target device identifier from a requester, the advertisement delivery system determines a target device score corresponding to the target device identifier through the device score model; The advertisement delivery system determines a target delivery result according to a preset advertisement delivery strategy and the target device score, and returns the target delivery result to the requesting party.

8. An advertising delivery platform, characterized in that: The advertising delivery platform includes a data synchronization system and an advertising delivery system. The data synchronization system includes a big data scheduling component, a file cutting component, a file processing component, a data synchronization component and a message middleware component, wherein: The big data scheduling component is used to import the multiple device score data stored in the Hive data warehouse of the advertising delivery system into the FTP file server to convert the multiple device scores in the Hive data warehouse into FTP large files; The file cutting component is used to cut the large FTP file in the FTP file server to obtain multiple sub-files; The file processing component is used to create multiple parsing threads for the multiple sub-files and to create a corresponding relationship between each parsing thread and the sub-files, and based on the corresponding relationship, download and parse the multiple sub-files through the multiple parsing threads, and store the data obtained after downloading and parsing in the first message queue of the message middleware component; The data synchronization component is used to synchronize the data in the first message queue to the memory database in the advertising delivery system according to a preset synchronization data volume, so that the device sub-model in the advertising delivery system is updated through the updated memory database.

9. A computer device, characterized in that: An advertising delivery platform is deployed in the computer device, and the advertising delivery platform includes a data synchronization system and an advertising delivery system. The data synchronization system includes a big data scheduling component, a file cutting component, a file processing component, a data synchronization component and a message middleware component. The computer device includes a memory, a processor and a computer program stored in the memory and executable on the processor, and is characterized in that when the processor executes the computer program, a data synchronization method for the advertising delivery platform as described in any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor executes the data synchronization method for an advertising delivery platform as described in any one of claims 1 to 7.

Images