Data processing method and apparatus, medium, electronic device, and program product

Abstract

A data processing method, device, medium, electronic equipment and program product, the method comprising: in response to a triggered message consumption request, taking out a storage address from a message queue, the message queue storing a storage address corresponding to an upload task; executing the upload task corresponding to the storage address, the upload task comprising downloading video data corresponding to the storage address from a public database, and uploading the downloaded video data to a private database, the public database being an object-based storage database, and the video data in the private database being used to support a promotion account generating promotion information. Based on queue smoothing public database request traffic, avoid triggering a large number of timeouts due to object storage throttling of the public database, thereby avoiding memory overflow caused by downloading and uploading due to retries, and the resulting unavailable downloading and uploading, thereby improving the stability of data processing for promotion.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of Internet, in particular, to a data processing method and device, medium, electronic equipment and program product. BACKGROUND

[0002] With the rapid development of Internet technology, more and more users obtain information such as news and short videos by accessing the Internet. Short videos have not only become a tool for recording daily life, but also a new online promotion tool.

[0003] The promotion platform can provide video materials to support users to generate short videos for promotion based on the video materials. In uploading video materials to the promotion platform, the promotion platform downloads source video data of the video materials from the object storage based on the storage address and uploads it to the promotion platform. However, due to the influence of the download bandwidth of the object storage, when a large number of requests for downloading and uploading are triggered, a large number of timeouts occur due to the flow limiting of the object storage. If retries are continued at this time, it may cause memory overflow of the service corresponding to the downloading and uploading, and eventually lead to abnormal termination of the service corresponding to the downloading and uploading. SUMMARY

[0004] This summary is provided to introduce a selection of concepts, which will be described with greater detail in the detailed description below. This summary does not intend to identify key or essential features of the claimed technology, nor is it intended for use in determining the scope of the claimed technology.

[0005] In a first aspect, the present disclosure provides a data processing method, comprising:

[0006] In response to a triggered message consumption request, a storage address is taken out from a message queue, and the message queue stores storage addresses corresponding to upload tasks;

[0007] An upload task corresponding to the storage address is executed, the upload task comprising downloading video data corresponding to the storage address from a public database and uploading the downloaded video data to a private database, the public database being a database based on object storage, and the video data in the private database being used to support a promotion account to generate promotion information.

[0008] In a second aspect, the present disclosure provides a data processing device, comprising:

[0009] A consumption module is configured to take out a storage address from a message queue in response to a triggered message consumption request, and the message queue stores storage addresses corresponding to upload tasks;

[0010] The execution module is configured to execute an upload task corresponding to the storage address, the upload task including downloading video data corresponding to the storage address from a public database and uploading the downloaded video data to a private database, the public database being an object-based storage database, and the video data in the private database being used to support a promotion account to generate promotion information.

[0011] In a third aspect, the present disclosure provides a computer readable medium having stored thereon a computer program which, when executed by a processing device, implements the steps of the method of the first aspect.

[0012] In a fourth aspect, the present disclosure provides an electronic device comprising:

[0013] a storage device having stored thereon a computer program;

[0014] a processing device configured to execute the computer program in the storage device to implement the steps of the method of the first aspect.

[0015] In a fifth aspect, the present disclosure provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method of the first aspect.

[0016] Through the above technical solution, the request flow of the public database is smoothed based on the queue, so as to avoid triggering a large number of timeouts due to object storage flow limiting of the public database, thereby avoiding memory overflow caused by downloading and uploading due to retries, and further avoiding the situation of unavailable downloading and uploading, thereby improving the stability of data processing for promotion.

[0017] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which:

[0019] Figure 1 is a flowchart of a data processing method according to an exemplary embodiment of the present disclosure.

[0020] Figure 2 is a process schematic diagram of creating an upload task according to an exemplary embodiment of the present disclosure.

[0021] Figure 3 is a process schematic diagram of a data processing method according to an exemplary embodiment of the present disclosure.

[0022] Figure 4 FIG. 6 is another process schematic diagram of a data processing method according to an example embodiment of the present disclosure.

[0023] Figure 5 FIG. 7 is another process schematic diagram of a data processing method according to an example embodiment of the present disclosure.

[0024] Figure 6 FIG. 8 is a process schematic diagram of querying a task state of a target upload task according to an example embodiment of the present disclosure.

[0025] Figure 7 FIG. 9 is a block diagram of a data processing apparatus according to an example embodiment of the present disclosure.

[0026] Figure 8 FIG. 10 is a structural schematic diagram of an electronic device according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided so as to more completely and thoroughly understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are merely for illustrative purposes and are not intended to limit the scope of protection of the present disclosure.

[0028] It should be understood that each of the steps recited in the method embodiments of the present disclosure can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0029] The term "comprising" and variations thereof as used herein are open-ended, that is, "comprising but not limited to." The term "based on" is "based, at least in part, on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments." Related terms are defined in the description that follows.

[0030] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are merely used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0031] It should be noted that the modification of "one" and "multiple" mentioned in the present disclosure is illustrative rather than limiting, and those skilled in the art should understand that "one or more" should be understood unless the context clearly indicates otherwise.

[0032] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0033] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the type, scope of use, use scenario, etc. of the personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained through appropriate means in accordance with relevant laws and regulations.

[0034] For example, in response to receiving the active request of the user, the user is sent prompt information to explicitly prompt the user that the operation requested to be performed will require obtaining and using the personal information of the user. Thus, the user can voluntarily choose whether to provide personal information to the software or hardware such as electronic device, application program, server or storage medium, etc. that performs the operation of the technical solution of the present disclosure according to the prompt information.

[0035] As an optional but non-limiting implementation, in response to receiving the active request of the user, the prompt information can be sent to the user in the form of a pop-up window, for example, in which the prompt information can be presented in the form of text. In addition, the pop-up window can also carry selection controls for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0036] It can be understood that the above notification and user authorization process is only illustrative and does not limit the implementation of the present disclosure, and other ways that meet the relevant laws and regulations can also be applied to the implementation of the present disclosure.

[0037] At the same time, it can be understood that the data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the relevant laws and regulations and relevant provisions. At present, based on short videos for promotion has become a common way of information promotion. In order to facilitate the promotion of information providers, also known as advertisers, to promote short videos, the promotion platform provides a marketing application program interface (Marketing API, MAPI) for the promotion user to provide account management, promotion information management and data management and other operations. Among them, the promotion user can bind video materials to the promotion account of the promotion user, so that the promotion user can produce promotion information based on video materials through the promotion platform. It can be understood that the promotion information here can be the short video mentioned above.

[0038] In the video material uploading on the promotion platform, the promotion platform calls an interface to download the video material from the object storage based on the storage address based on the data processing service provided by the interface, and upload the video material to the promotion platform. When a large number of calls are made to the interface, due to the limitation of the object storage download bandwidth, the interface may trigger the object storage flow limiting and a large number of timeouts may occur, which eventually leads to the unavailability of the interface, that is, the abnormal termination of the data processing service corresponding to the interface, which causes the binding to fail, and further affects the generation of promotion information.

[0039] Therefore, the embodiments of the present disclosure disclose a data processing method, device, medium, electronic equipment and program product.

[0040] The embodiments of the present disclosure are further explained and described below in combination with the drawings.

[0041] Figure 1 is a flowchart of a data processing method according to an exemplary embodiment of the present disclosure. The data processing method can be applied to an electronic device, and the data processing method can be executed by a data processing device, wherein the data processing device can be implemented by software and / or hardware, and the software and / or hardware can be configured in the electronic device. Referring to Figure 1 The data processing method can include the following steps:

[0042] Step 110, in response to the triggered message consumption request, taking out the storage address from the message queue, and the message queue stores the storage address corresponding to the upload task;

[0043] Step 120, executing the upload task corresponding to the storage address, the upload task including downloading the video data corresponding to the storage address from the public database, and uploading the downloaded video data to the private database, the public database being a database based on object storage, and the video data in the private database being used to support the promotion account to generate promotion information.

[0044] Through the above technical solution, the request flow of the public database is smoothed based on the queue, avoiding the situation that a large number of timeouts occur due to the object storage flow limiting of the public database, thereby avoiding the memory overflow caused by the download and upload due to the retry, and further causing the situation that the download and upload are unavailable, and improving the stability of the data processing for promotion.

[0045] In the present disclosure, the electronic device can implement the above data processing method by calling the MAP I.

[0046] In the present disclosure, the storage address can be a URL (Uniform Resource Locator, Uniform Resource Locator). Through the storage address, the video data corresponding to the storage address can be accessed in the public database.

[0047] In the present disclosure, the triggered message consumption request can be triggered when the resource for consuming the message is idle.

[0048] In the present disclosure, the public database is an object-based storage database, and the pre-uploaded video data can be stored in the public database.

[0049] In the present disclosure, the private database can be a target multimedia platform, and the first user can create and deliver the promotion information on one or more target multimedia platforms through the electronic device by uploading the video data to the target multimedia platform. It should be noted that the video data uploaded to the private database and the video material have the same physical meaning.

[0050] In the present disclosure, the video data in the private database is used to support the generation of promotion information by a promotion account, and the promotion account is an account bound to the video data.

[0051] In a possible manner, the step of executing the uploading task corresponding to the storage address can be implemented in the following manner: executing the uploading task corresponding to the storage address in an asynchronous manner.

[0052] By executing the uploading task asynchronously, the next uploading task can be created without waiting for the uploading result of the previous uploading task, so that the uploading task can be created in a non-blocking manner.

[0053] In a possible manner, the data processing method can further include: obtaining a storage address of the video data to be uploaded; determining whether a preset condition for creating an uploading task corresponding to the storage address is met, the preset condition including that no uploading task corresponding to the storage address has been created; creating the uploading task corresponding to the storage address when the preset condition is met; and adding the storage address corresponding to the uploading task to the message queue.

[0054] The storage address of the video data to be uploaded can be specified by the second user.

[0055] The preset condition includes that no uploading task corresponding to the storage address has been created. Therefore, when no uploading task corresponding to the storage address has been created, the uploading task corresponding to the storage address is created, and it can be understood that when no uploading task corresponding to the storage address has been created is not met, the uploading task corresponding to the storage address is not created, so that repeated uploading tasks can be avoided, i.e., the uploading and downloading of the same video data are repeatedly executed, and the occupation of interface resources and the bandwidth of the public database is avoided.

[0056] In addition, it can be understood that, in the case that the preset condition that no upload task corresponding to the storage address has been created is not met, it indicates that an upload task corresponding to the storage address has been created, that is, the downloading of the video data corresponding to the storage address has been performed.

[0057] In a possible manner, the preset condition includes that no upload task corresponding to the storage address has been created, in which case, the step of determining whether the preset condition of creating the upload task corresponding to the storage address is met can include: obtaining a first video identifier of the video data to be uploaded, the first video identifier including the storage address and a video signature, the video signature being determined according to at least first video information of the video data; and determining whether the preset condition of creating the upload task corresponding to the storage address is met according to whether an identifier identical to the first video identifier exists in a task database, the task database storing first video identifiers of video data corresponding to upload tasks that have been created.

[0058] The video signature is used to uniquely represent the content of the video data, and the first video information can include video content information and video data modification time information. The video content information can be at least one key frame in the video data.

[0059] The first video identifier is obtained by performing information digest algorithm calculation on the storage address and the video signature. As an example, the information digest algorithm can be MD (Message Digest Algorithm) and SHA (Secure Hash Algorithm), and the like.

[0060] In the case that an identifier identical to the first video identifier exists in the task database, it is determined that the preset condition of creating the upload task corresponding to the storage address is not met; and in the case that no identifier identical to the first video identifier exists in the task database, it is determined that the preset condition of creating the upload task corresponding to the storage address is met.

[0061] In the above manner, the video signature is determined according to the first video information of the video data, so that the first video identifier that can uniquely represent the video data is obtained, and the same video data is processed for deduplication, thereby avoiding repeated downloading of the video data from the public database.

[0062] In a possible manner, the video signature is determined according to the first video information of the video data and a preset check code. The preset check code can be used to detect whether an error occurs in the video data during transmission, thereby providing security for data processing.

[0063] In a possible manner, the preset condition can further include that the video data passes a pre-check. As an example, the pre-check can include a check on the length and width of the video and a check on the video code rate, and the like.

[0064] Figure 2 is a process schematic diagram of creating an upload task according to an exemplary embodiment of the present disclosure. Referring to Figure 2 , the electronic device can send an upload request to the MAP I in response to the triggered upload request, the upload request carrying a storage address of the video data to be uploaded; according to the storage address, the MAP I requests video meta information from the public database. The video meta information includes, for example, the video length, the video width, the video code rate, the video signature, and the like.

[0065] The public database queries the video meta information corresponding to the storage address, and returns the video meta information to the MAP I.

[0066] The MAP I performs pre-checking, and after passing the pre-checking, queries according to the URL and the video signature, i.e., determines whether there is an upload task corresponding to the storage address in the task database, and returns the corresponding result. If the result indicates that there is no upload task corresponding to the storage address in the task database, it is determined to create an upload task, and the URL, the corresponding video signature, and the upload task information are saved in the task database.

[0067] The MAP I issues the URL to the message queue, i.e., adds the storage address corresponding to the created upload task to the message queue to wait for being executed.

[0068] The MAP I returns the result of the upload request to the electronic device, where the result of the upload request can be whether the upload task is created successfully and the reason for the creation.

[0069] In a possible manner, the message queue can include multiple message queues, and the step of adding the storage address corresponding to the upload task to the message queue can include: obtaining second video information of the video data corresponding to the upload task; and adding the storage address corresponding to the upload task to the message queue corresponding to the second video information according to the second video information.

[0070] The second video information includes data size information and / or priority information.

[0071] The second video information includes data size information, and correspondingly, the message queue can be divided into a small video queue and a large video queue. As an example, the storage address corresponding to the upload task of the video data with a data size less than or equal to 100 megabytes (M) can be added to the small video queue, and the storage address corresponding to the upload task of the video data with a data size greater than 100 bytes can be added to the large video queue.

[0072] The second video information includes priority information, and the message tasks can be divided into a first priority queue and a second priority queue, the priority of the video data corresponding to the storage address in the first priority queue is higher than the priority of the video data corresponding to the storage address in the second priority queue. According to the priority information of the video data, the URL corresponding to the video data is distributed to the priority queue corresponding to the priority.

[0073] In the above manner, a plurality of message queues can be set, and the URL corresponding to the video data is added to the corresponding message queue according to the second video information of the video data, so that the ability to consume different message queues can be set according to the actual situation.

[0074] In a possible manner, the number of concurrents of the message queue and the number of partitions for processing the upload task can be determined by the download bandwidth of the public database. The number of concurrents is used to represent the maximum number of storage addresses that can be consumed simultaneously by the message queue, and the number of partitions is used to represent the maximum number of resources for consuming the storage addresses. Here, the resources can be threads, processes, or devices.

[0075] For example, taking the case that the message queue can be divided into a small video queue and a large video queue as an example, the number of concurrents of the small video queue can be set to 50, and the number of concurrents of the large video queue can be set to 5. The number of partitions of the small video queue and the large video queue is both 5. Assuming that the time delay is 60 seconds, the maximum bandwidth is (100M*50*3+1000M*5*3) / 60=500 gigabytes / second, wherein 100M and 1000M are the maximum data amounts of the video data corresponding to the storage addresses in the small video queue and the large video queue, respectively, and 500 gigabytes / second can be the download bandwidth of the public database.

[0076] In the above manner, the number of concurrents of the message queue and the number of partitions for processing the upload task are determined according to the download bandwidth of the public database, so as to avoid triggering the bandwidth throttling of the public database.

[0077] In a possible manner, the above data processing method further includes: determining that the storage address corresponding to the downloaded video data is in a first transmission state, the first transmission state being used to represent that the video data is successfully uploaded to the private database; determining whether the video data corresponding to the upload task is valid in the private database; in a case where it is determined that the video data corresponding to the upload task is valid in the private database, generating a binding message and updating the task state of the upload task to a first execution state, the binding message being used to bind the second video identifier of the downloaded video data and the promotion account, the promotion account having the permission to access the video data corresponding to the second video identifier bound with the promotion account from the private database, and the first execution state being used to represent that the video data is in a binding state; processing the binding message.

[0078] It is worth mentioning that the task database stores the transmission state of the storage address corresponding to the downloaded video data. By querying the transmission state of the storage address, it can be determined whether the video data corresponding to the storage address is successfully uploaded to the private database. Among them, the transmission state of the storage address can be automatically queried based on timing.

[0079] Among them, the task database not only records the uploading task download or upload state, but also records the account binding state related to the uploading task.

[0080] Among them, whether the video data is valid in the private database can represent whether the video data is stored in the private database. As an example, the video metadata of the video data in the private database can be queried to determine whether the video data corresponding to the uploading task is valid in the private database. For example, if it is determined that the video metadata of the video data corresponding to the uploading task exists in the private database, it is determined that the video data corresponding to the uploading task is valid in the private database; if it is determined that the video metadata of the video data corresponding to the uploading task does not exist in the private database, it is determined that the video data corresponding to the uploading task is invalid in the private database.

[0081] Among them, after the second video identifier of the downloaded video data is bound to the promotion account, the promotion account has the permission to access the video data corresponding to the second video identifier bound to the promotion account from the private database. As an example, the promotion account can be the account specified in the above uploading request.

[0082] Figure 3 is a process schematic diagram of a data processing method according to an exemplary embodiment of the present disclosure. Figure 3 The schematic diagram shown represents the processing flow in the first transmission state. Referring to Figure 3 , the MAP I can request the task database to query the transmission state of the storage address, and the task database returns the transmission state after querying; in the case that the transmission state is the first transmission state, the MAP I further requests the private database to query the video metadata of the video data corresponding to the uploading task, and in the case that the private database successfully returns the video metadata, it is determined that the video data corresponding to the uploading task is valid in the private database, and then a binding message is generated, and the task state of the uploading task is updated to the first execution state.

[0083] Among them, the private database can provide the second video identifier to support the binding with the promotion account.

[0084] Among them, processing the binding message can be processed by calling resources through the MAP I, and the resources can be thread or process resources, etc.

[0085] By the above manner, the second video identifier of the downloaded video data and the promotion account can enable the promotion account to have the permission to access the video data corresponding to the second video identifier bound with the promotion account from the private database, and further generate promotion information based on the video data. In addition, even in the case of successful downloading and uploading, since there is a case of deleting the uploaded video data from the private database, it is necessary to determine that the storage address corresponding to the downloaded video data is in the first transmission state and that the video data is still valid in the private database to trigger the binding, thereby avoiding resource waste caused by invalid binding. In addition, after generating the binding message, the task state of the uploading task is updated to the first execution state, thereby avoiding repeated generation of the binding message.

[0086] In a possible manner, the binding message can be added to the binding queue, so that a large number of binding messages can be queued for consumption, thereby relieving the pressure of processing the binding message.

[0087] In a possible manner, the above data processing method can further include: determining that the storage address corresponding to the downloaded video data is in a second transmission state, and maintaining the task state of the uploading task in the task database as a second execution state within a first preset time length. The second transmission state is used to represent a failure of uploading to the private database, and the second execution state is used to represent a completion of the video data binding.

[0088] The first preset time length can be set according to actual conditions, for example, the first preset time length can be set according to the time length required by the second user to solve the failure of uploading the downloaded video data to the private database.

[0089] The failure of uploading the downloaded video data to the private database can include a download failure or an upload failure. The download failure is used to represent a failure of successfully downloading the video data from the public database, and the upload failure is used to represent a failure of successfully uploading the video data to the private database.

[0090] Figure 4 FIG. 4 is another process schematic diagram of the data processing method according to an example embodiment of the present disclosure. Figure 4 The schematic diagram shown represents the processing flow of determining the second transmission state. Referring to FIG. 4, Figure 4 When the transmission state of the received storage address is the second transmission state, the MAP I updates the task state of the uploading task to the second execution state, and the maintenance time length of the second execution state is the first preset time length.

[0091] It is worth mentioning that in the case of failure of uploading the downloaded video data to the private database, the binding of the video data and the promotion account cannot be successfully implemented. In the present disclosure, in order to complete the uploading task, the uploading task that has not been completed for a long time can be automatically triggered again in a short time. And it can be understood that before the fault causing the failure of uploading the downloaded video data to the private database is solved, the retriggering still cannot complete the binding. Therefore, in order to avoid the waste of resources caused by retriggering before the above fault is solved, the task state of the uploading task in the task database can be maintained as the second execution state within the first preset time length after the failure of uploading the downloaded video data to the private database, so as to avoid being automatically triggered again.

[0092] In a possible manner, continuing to refer to Figure 4 In the case that the task state of the uploading task is maintained as the second execution state for the first preset time length, the task state of the uploading task is updated to the third execution state, and the third execution state can be a video data unbinding state, which provides a data basis for retriggering binding.

[0093] In a possible manner, when the task state of the uploading task is updated from the first execution state to the second execution state, the reason for the failure of uploading the video data to the private database can be stored in the task database at the same time, so as to facilitate the second user to troubleshoot the fault causing the failure of uploading the downloaded video data to the private database. At the same time, the uploading task whose task state is set to the second execution state due to the normal completion of the binding can also be distinguished, so as to avoid the false triggering of the uploading task that has been normally completed.

[0094] In a possible manner, the above data processing method can further include: determining that the downloaded video data is in a target state, and locking the storage address of the video data to be uploaded; determining the transmission state corresponding to the storage address recorded in the task database; in the case that the transmission state is the second transmission state, re-executing the downloading of the video data corresponding to the storage address from the public database and the uploading of the downloaded video data to the private database, updating the task state of the corresponding uploading task and the corresponding transmission state in the task database according to the execution of the re-execution, and determining whether to release the lock of the storage address.

[0095] It is worth mentioning that the target state can be the case that the video data is successfully downloaded but invalid in the private database; or the case that the video data corresponding to the second execution state reaches a preset time length, i.e. the case that the task state of the uploading task corresponding to the video data is the third execution state, and so on. In these cases, the video data can be re-downloaded and uploaded.

[0096] The transmission state is used to represent whether the video data corresponding to the storage address is successfully processed to the private database or fails

[0097] The locking and releasing of the storage address can be implemented by a distributed lock.

[0098] It can be understood that, in the case that the transmission state is the first transmission state, the binding message can be directly generated.

[0099] The execution case of the re-execution can be a download success, a download failure, an upload success, an upload failure, and the like. The following is exemplarily described by taking the download success and the upload success as examples. Figure 5 is another process schematic diagram of the data processing method according to an example embodiment of the present disclosure. Figure 5 The schematic diagram shown represents the processing flow of determining the target state. Referring to Figure 5 , the MAPI determines to lock the storage address when the video data is in the target state, and confirms the transmission state of the storage address. In the case that the transmission state represents that the video data corresponding to the storage address fails to be processed to the private database, the MAPI re-downloads the video data corresponding to the storage address from the public database, and uploads the downloaded video data to the private database. The private database sends a second video identifier to the MAPI. The MAPI generates a corresponding binding message, updates the transmission state to the first transmission state, the task state to the first execution state, and releases the lock of the storage medium.

[0100] In the above manner, the binding of the account and the video data is implemented.

[0101] In a possible manner, the data processing method can further include: in response to a query operation for the task state of the target upload task, querying the task state of the target upload task based on the task database; and receiving a query result of the query operation, the query result including the task state of the target upload task recorded in the task database.

[0102] The query operation can be triggered by the second user through the electronic device.

[0103] Figure 6 is a process schematic diagram of querying the task state of the target upload task according to an example embodiment of the present disclosure. Referring to Figure 6 , the second user triggers a query operation of the target upload task through an electronic device, and the electronic device implements the query operation by calling the MAPI. The MAPI queries the task state of the target upload task in the task database, the task database returns the task state of the target upload task to the MAPI, and the MAPI returns the task state of the target upload task to the electronic device.

[0104] In the above manner, the query function of the second user querying the task state of the target upload task is provided.

[0105] In a possible implementation, the data processing method can further include: performing a timing task, the timing task being configured to poll a task state of an upload task corresponding to the storage address in the task database based on the task database to determine a timeout task, and re-trigger execution of a corresponding processing flow according to the task state of the timeout task, the timeout task including an upload task whose task state in the task database has not been updated to a video data binding completion within a second preset time length after the task state is added to the task database.

[0106] Figure 6 is a processing process diagram for querying a timeout task according to an example embodiment of the present disclosure. Referring to Figure 6 , the timing script is configured to implement execution of the timing task, if the task state of the timeout task is downloading, the URL is sent to the queue 1, and the download of the URL corresponding video data and the upload of the video data to the private database are instructed; if the task state of the timeout task is binding, the URL is sent to the queue 2, and the re-binding of the URL corresponding video data to the corresponding promotion account is instructed.

[0107] In the above manner, a bottom-up processing flow is set to complete the binding of the video data.

[0108] Figure 7 is a block diagram of a data processing device according to an example embodiment of the present disclosure. Referring to Figure 7 , the data processing device includes:

[0109] a consumption module configured to take out a storage address from a message queue in response to a triggered message consumption request, the message queue storing a storage address corresponding to an upload task;

[0110] an execution module configured to execute the upload task corresponding to the storage address, the upload task including downloading video data corresponding to the storage address from a public database and uploading the downloaded video data to a private database, the public database being an object-based storage database, and the video data in the private database being used to support a promotion account to generate promotion information.

[0111] Optionally, the execution module is further configured to execute the upload task corresponding to the storage address in an asynchronous manner.

[0112] Optionally, the device further includes:

[0113] an acquisition module configured to acquire a storage address of video data to be uploaded;

[0114] a first determination module configured to determine whether a preset condition for creating an upload task corresponding to the storage address is met, the preset condition including that no upload task corresponding to the storage address has been created;

[0115] a creating module, configured to create an upload task corresponding to the storage address when the preset condition is met;

[0116] an adding module, configured to add the storage address corresponding to the upload task to the message queue.

[0117] Optionally, the first determining module is further configured to:

[0118] obtain a first video identifier of the video data to be uploaded, the first video identifier comprising a storage address and a video signature, the video signature being determined according to at least first video information of the video data;

[0119] determine whether a preset condition of creating the upload task corresponding to the storage address is met according to whether an identifier identical to the first video identifier exists in a task database, the task database storing first video identifiers of video data corresponding to created upload tasks.

[0120] Optionally, the message queue comprises a plurality of message queues, and the adding module is further configured to:

[0121] obtain second video information of the video data corresponding to the upload task, the second video information comprising data size information and / or priority information;

[0122] add the storage address corresponding to the upload task to a message queue corresponding to the second video information according to the second video information.

[0123] Optionally, the apparatus further comprises:

[0124] a second determining module, configured to determine that a storage address corresponding to the downloaded video data is in a first transmission state, the first transmission state being used to represent that the video data is successfully uploaded to the private database;

[0125] a third determining module, configured to determine whether the video data corresponding to the upload task is valid in the private database;

[0126] a generating module, configured to generate a binding message and update a task state of the upload task to a first execution state when it is determined that the video data corresponding to the upload task is valid in the private database, the binding message being used to bind a second video identifier of the downloaded video data and the promotion account, the promotion account having a permission of accessing video data corresponding to the second video identifier bound with the promotion account from the private database, the first execution state being used to represent a video data binding state.

[0127] a processing module, configured to process the binding message.

[0128] Optionally, the apparatus further comprises:

[0129] The fourth determining module is configured to determine that the storage address corresponding to the downloaded video data is in a second transmission state, and keep the task state of the uploading task in the task database as a second execution state within a first preset time length, the second transmission state is used to represent that uploading to the private database fails, and the second execution state is used to represent that video data binding is completed.

[0130] Optionally, the apparatus further comprises:

[0131] The fifth determining module is configured to determine that the downloaded video data is in a target state, and lock the storage address of the video data to be uploaded.

[0132] The sixth determining module is configured to determine the transmission state corresponding to the storage address recorded in the task database, the transmission state being used to represent that processing of the video data corresponding to the storage address to the private database is successful or fails.

[0133] The seventh determining module is configured to, in the case that the transmission state represents that processing of the video data corresponding to the storage address to the private database fails, re-execute downloading of the video data corresponding to the storage address from the public database, upload the downloaded video data to the private database, update the task state of the corresponding uploading task and the corresponding transmission state in the task database according to the execution of the re-execution, and determine whether to release the lock of the storage address.

[0134] Optionally, the apparatus further comprises:

[0135] The response module is configured to, in response to a query operation for a task state of a target uploading task, query the task state of the target uploading task based on the task database.

[0136] The receiving module is configured to receive a query result of the query operation, the query result including the task state of the target uploading task recorded in the task database.

[0137] Optionally, the number of concurrences of the message queue and the number of partitions for processing uploading tasks are determined by a download bandwidth of the public database, the number of concurrences being used to represent a maximum number of storage addresses that can be simultaneously consumed by the message queue, and the number of partitions being used to represent a maximum number of resources for consuming storage addresses.

[0138] The embodiments of the modules in the data processing apparatus 700 can refer to the related embodiments described above, and will not be described here.

[0139] Based on the same inventive concept, the embodiments of the present disclosure further provide a computer readable medium having stored thereon a computer program which, when executed by a processing device, implements the steps of the data processing method.

[0140] Based on the same inventive concept, the embodiments of the present disclosure further provide a computer program product comprising a computer program which, when executed by a processor, implements the steps of the data processing method.

[0141] Based on the same inventive concept, the embodiments of the present disclosure further provide an electronic device comprising:

[0142] a storage device having stored thereon a computer program;

[0143] a processing device configured to execute the computer program in the storage device to implement the steps of the data processing method.

[0144] Reference will now be made to the following description Figure 8 , taken in conjunction with the accompanying drawings, which illustrate a structural diagram of an electronic device 800 suitable for use in implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet PCs), PMPs (Portable Multimedia Players), car-mounted terminals (e.g., car-mounted navigation terminals), and the like, as well as fixed terminal devices such as digital TVs, desktop computers, and the like. Figure 8 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0145] As shown in Figure 8 , the electronic device 800 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801 which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 802 or loaded from a storage device 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the electronic device 800 are also stored in the RAM 803. The processing device 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0146] In general, the following devices can be connected to the I / O interface 805: input devices 806, including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 807, including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 808, including, for example, a magnetic tape, a hard disk, and the like; and communication devices 809. The communication devices 809 can allow the electronic device 800 to communicate wirelessly or wired with other devices to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it is understood that all of the shown devices are not required to be implemented or present. More or less devices can alternatively be implemented or present.

[0147] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 809, or installed from the storage devices 808, or installed from the ROM 802. When the computer program is executed by the processing devices 801, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0148] It is noted that the aforementioned computer-readable medium of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example and without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium can include a computer-readable program code transmitted by a computer-readable storage medium or carried by a carrier wave in a baseband or as part of a carrier wave. Such a propagated computer-readable signal medium can take various forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that can be used to carry or store a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to wire, cable, RF (radio frequency), or the like, or any suitable combination of the foregoing.

[0149] In some embodiments, the electronic device can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communications (e.g., a communications network) of any form or medium (e.g., a communications network). Examples of communications networks include local area networks ("LAN"), wide area networks ("WAN"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

[0150] The aforementioned computer-readable medium can be included in the aforementioned electronic device; or can exist separately from the electronic device without being incorporated into the electronic device.

[0151] The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to: in response to a triggered message consumption request, take out a storage address from a message queue, the message queue storing storage addresses corresponding to upload tasks; and execute an upload task corresponding to the storage address, the upload task including downloading video data corresponding to the storage address from a public database and uploading the downloaded video data to a private database, the public database being an object-based storage database, and the video data in the private database being used to support a promotion account to generate promotion information.

[0152] Computer program code for carrying out operations of the present disclosure can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ as well as conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0153] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0154] The modules described in the embodiments of the present disclosure can be implemented by software, or by hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0155] The functionality described above can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, an example type of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0156] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0157] The above description is only preferred embodiments of the present disclosure and the explanation of the technical principles of the application. It should be understood by those skilled in the art that the disclosure range involved in the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosure concept. For example, the technical solutions formed by replacing the above features with the technical features disclosed in the present disclosure (but not limited to) having similar functions.

[0158] Further, although operations are depicted in a particular order, this should not be understood as requiring the operations to be performed in the particular order shown or in sequential order, and that certain operations can be performed in parallel or in any suitably ordered order under certain circumstances. Also, although specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the disclosure. Certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any suitable subcombination.

[0159] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. With respect to the devices in the above-described embodiments, in which various modules perform operations, the specific manner in which the operations are performed by the various modules has been described in detail in the embodiments relating to the method. Here, no detailed explanation will be given.

Claims

1. A data processing method, characterized in that, include: In response to a triggered message consumption request, a storage address is retrieved from the message queue, which stores the storage address corresponding to the upload task. Execute the upload task corresponding to the storage address. The upload task includes downloading video data corresponding to the storage address from a public database and uploading the downloaded video data to a private database. The public database is an object-based database, and the video data in the private database is used to support the generation of promotional information by the promotional account. Upload tasks that have not been completed for a long time will be automatically retried. The method also includes: determining that the storage address corresponding to the downloaded video data is in a second transmission state, and keeping the task status of the upload task in the task database in a second execution state for a first preset time period to avoid being automatically retried. The second transmission state is used to indicate that the upload to the private database has failed, and the second execution state is used to indicate that the second video identifier of the video data has been successfully bound to the promotion account.

2. The method according to claim 1, characterized in that, The execution of the upload task corresponding to the storage address includes: The upload task corresponding to the storage address is executed asynchronously.

3. The method according to claim 1, characterized in that, The method further includes: Get the storage address of the video data to be uploaded; Determine whether the preset conditions for creating an upload task corresponding to the storage address are met, including that no upload task corresponding to the storage address has been created before; Under the condition that the preset conditions are met, an upload task corresponding to the storage address is created; Add the storage address corresponding to the upload task to the message queue.

4. The method according to claim 3, characterized in that, The step of determining whether the preset conditions for creating an upload task corresponding to the storage address are met includes: Obtain a first video identifier of the video data to be uploaded, wherein the first video identifier includes a storage address and a video signature, and the video signature is determined at least based on the first video information of the video data; Based on whether there is an identifier in the task database that is the same as the first video identifier, it is determined whether the preset conditions for creating an upload task corresponding to the storage address are met. The task database stores the first video identifier of the video data corresponding to the already created upload task.

5. The method according to claim 3, characterized in that, The message queue includes multiple queues, and adding the storage address corresponding to the upload task to the message queue includes: Obtain second video information corresponding to the video data of the upload task, wherein the second video information includes data size information and / or priority information; Based on the second video information, the storage address corresponding to the upload task is added to the message queue corresponding to the second video information.

6. The method according to claim 3, characterized in that, The method further includes: The storage address corresponding to the downloaded video data is determined to be in a first transmission state, which indicates that the video data has been successfully uploaded to the private database; Determine whether the video data corresponding to the upload task is valid in the private database; If it is determined that the video data corresponding to the upload task is valid in the private database, a binding message is generated and the task status of the upload task is updated to the first execution status. The binding message is used to bind the second video identifier of the downloaded video data to the promotion account. The promotion account has the permission to access the video data corresponding to the second video identifier bound to the promotion account from the private database. The first execution status is used to indicate that the video data binding is in progress. Process the binding message.

7. The method according to claim 1, characterized in that, The method further includes: Once it is confirmed that the downloaded video data is in the target state, the storage address of the video data to be uploaded is locked; Determine the transmission status corresponding to the storage address recorded in the task database. The transmission status is used to characterize whether the video data corresponding to the storage address was successfully or failed to be processed into the private database. If the transmission status indicates that the video data corresponding to the storage address has failed to be processed to the private database, the process re-executes the download of the video data corresponding to the storage address from the public database, uploads the downloaded video data to the private database, updates the task status and transmission status of the corresponding upload task in the task database according to the execution status of the re-execution, and determines whether to release the lock of the storage address.

8. The method according to claim 6, characterized in that, The method further includes: In response to a query operation on the task status of a target upload task, the task status of the target upload task is queried based on the task database; The query result of the query operation is received, and the query result includes the task status of the target upload task recorded in the task database.

9. The method according to claim 6, characterized in that, The method further includes: A scheduled task is executed. The scheduled task is used to poll the task status of the upload task corresponding to the storage address in the task database to determine the timeout task, and to re-trigger the execution of the corresponding processing flow according to the task status of the timeout task. The timeout task includes the upload task whose task status in the task database has not been updated to video data binding completed within a second preset time period after being added to the task database.

10. The method according to any one of claims 1-9, characterized in that, The concurrency of the message queue and the number of partitions for processing upload tasks are determined by the download bandwidth of the public database. The concurrency is used to represent the maximum number of storage addresses that the message queue can be consumed simultaneously, and the number of partitions is used to represent the maximum number of resources that can be consumed from the storage addresses.

11. A data processing apparatus, characterized in that, include: The consumption module is used to retrieve the storage address from the message queue in response to the triggered message consumption request. The message queue stores the storage address corresponding to the upload task. An execution module is used to execute an upload task corresponding to the storage address. The upload task includes downloading video data corresponding to the storage address from a public database and uploading the downloaded video data to a private database. The public database is an object-based database, and the video data in the private database is used to support the generation of promotional information by promotional accounts. Upload tasks that have not been completed for a long time will be automatically retried. The data processing device also includes a fourth determining module, which is used to determine that the storage address corresponding to the downloaded video data is in a second transmission state. The task status of the upload task in the task database is kept in the second execution state for a first preset time period to avoid being automatically retried. The second transmission state is used to indicate that the upload to the private database has failed. The second execution state is used to indicate that the second video identifier of the video data has been successfully bound to the promotion account.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When executed by a processing device, the computer program performs the steps of the method described in any one of claims 1-10.

13. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1-10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-10.

Images