Label determination method and related device

Abstract

The invention discloses a label determination method and a related device, and the method comprises the steps: obtaining a label production request comprising a target label use scene, obtaining target label metadata suitable for the target label use scene according to the label production request, and obtaining the target label metadata based on the corresponding relation between the label use scene and a label processing level. And determining a target label processing level corresponding to the target label use scene. The higher the real-time performance of the label use scene is, the faster the corresponding label needs to be generated to meet the subsequent use scene, so that in the corresponding relation between the label use scene and the label processing level, the higher the real-time performance of the label use scene is, and the higher the corresponding label processing level is. Therefore, the target label metadata is processed based on the target label processing level to obtain the label of the target label metadata, and the processing speed of the label of the target label metadata is matched with the label processing level, so that the obtained label of the target label metadata can be suitable for a target label use scene, and the application range of the label is expanded.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a label determination method and related devices. Background Art

[0002] Data labels help to classify data and convey information to users. Taking membership tags on video platforms as an example, video platforms add membership tags to users based on member information and behavioral preferences, so as to recommend videos that users may be interested in based on membership tags, thereby providing companies with more comprehensive user insights and personalized and accurate recommendation capabilities.

[0003] In the related art, big data technology is used to directly extract and process metadata to produce labels. For example, offline metadata is processed based on Spark software to obtain labels.

[0004] However, the above-mentioned method of determining labels cannot meet the scenarios that require production performance indicators, and is applicable to fewer scenarios, resulting in a smaller scope of application. Summary of the invention

[0005] In view of the above problems, the present application provides a label determination method and related devices for expanding the application scope of the label.

[0006] Based on this, the embodiments of the present application disclose the following technical solutions:

[0007] In one aspect, an embodiment of the present application provides a method for determining a label, the method comprising:

[0008] Obtain a label production request including a target label usage scenario;

[0009] Acquire target tag metadata corresponding to the target tag usage scenario according to the tag production request;

[0010] Determine the target tag processing level corresponding to the target tag usage scenario according to the correspondence between the tag usage scenario and the tag processing level; wherein, in the correspondence between the tag usage scenario and the tag processing level, the higher the real-time nature of the tag usage scenario, the higher the corresponding tag processing level;

[0011] The target tag metadata is processed according to the target tag processing level to obtain a label of the target tag metadata.

[0012] Optionally, the tag processing level includes a first level and a second level, and the first level is higher than the second level;

[0013] The step of determining the label of the target label metadata according to the target label processing level and the target label metadata includes:

[0014] If the target tag processing level is the first level, the tag of the target tag metadata is determined preferentially;

[0015] If the target tag processing level is the second level, the target tag metadata is added to the message queue so as to be processed in the order of the tag metadata included in the message queue to obtain the tag of the target tag metadata.

[0016] Optionally, the step of preferentially determining the tag of the target tag metadata includes:

[0017] Determine a tag access party, where the tag access party is a producer of the target tag metadata;

[0018] Setting a tag writing permission for the tag access party, wherein the tag writing permission indicates that the tag access party has the permission to directly call an interface to generate a tag;

[0019] The interface is called according to the tag writing permission, and the tag of the target tag metadata is determined through the interface.

[0020] Optionally, the tag processing level includes a first level, a second level and a third level, the first level is higher than the second level, and the second level is higher than the third level;

[0021] The step of determining the label of the target label metadata according to the target label processing level and the target label metadata includes:

[0022] If the target tag processing level is the first level, the tag of the target tag metadata is determined preferentially;

[0023] If the target tag processing level is the second level, determining the tag of the target tag metadata through a stream task;

[0024] If the target tag processing level is the third level, the tag of the target tag metadata is determined by a batch processing task, and the processing speed of the batch processing task is lower than the processing speed of the stream task.

[0025] Optionally, the second level includes a first sub-level and a second sub-level, the first sub-level is higher than the second sub-level, and in the flow task, the time window of the first sub-level is shorter than the time window of the second sub-level.

[0026] Optionally, in the flow task, the time window is determined based on the processing time and tolerance time corresponding to the tag processing level, the tolerance time is the longest delay time allowed by the message queue to receive the target tag metadata, and the tolerance time corresponding to the first sub-level is shorter than the tolerance time corresponding to the second sub-level.

[0027] Optionally, the method further includes:

[0028] Storing the tag of the target tag metadata in a database and determining a storage time;

[0029] Determining a time to acquire the target tag metadata;

[0030] Calculating the time difference between the storage time and the acquisition time;

[0031] If the time difference is greater than a preset time threshold, an alarm message is output.

[0032] On the other hand, the present application provides a label determination device, the device comprising: an acquisition unit, a level determination unit and a label determination unit;

[0033] The acquisition unit is used to acquire a label production request including a target label usage scenario;

[0034] The acquisition unit is further configured to acquire target tag metadata corresponding to the target tag usage scenario according to the tag production request;

[0035] The level determination unit is used to determine the target tag processing level corresponding to the target tag usage scenario according to the correspondence between the tag usage scenario and the tag processing level; wherein, in the correspondence between the tag usage scenario and the tag processing level, the higher the real-time nature of the tag usage scenario, the higher the corresponding tag processing level;

[0036] The label determination unit is used to process the target label metadata according to the target label processing level to obtain a label of the target label metadata.

[0037] Optionally, the label processing level includes a first level and a second level, and the first level is higher than the second level; the label determination unit is specifically configured to:

[0038] If the target tag processing level is the first level, the tag of the target tag metadata is determined preferentially;

[0039] If the target tag processing level is the second level, the target tag metadata is added to the message queue so as to be processed in the order of the tag metadata included in the message queue to obtain the tag of the target tag metadata.

[0040] Optionally, the label determination unit is specifically used to:

[0041] Determine a tag access party, where the tag access party is a producer of the target tag metadata;

[0042] Setting a tag writing permission for the tag access party, wherein the tag writing permission indicates that the tag access party has the permission to directly call an interface to generate a tag;

[0043] The interface is called according to the tag writing permission, and the tag of the target tag metadata is determined through the interface.

[0044] Optionally, the label processing level includes a first level, a second level and a third level, the first level is higher than the second level, and the second level is higher than the third level; the label determination unit is specifically configured to:

[0045] If the target tag processing level is the first level, the tag of the target tag metadata is determined preferentially;

[0046] If the target tag processing level is the second level, determining the tag of the target tag metadata through a stream task;

[0047] If the target tag processing level is the third level, the tag of the target tag metadata is determined by a batch processing task, and the processing speed of the batch processing task is lower than the processing speed of the stream task.

[0048] Optionally, the second level includes a first sub-level and a second sub-level, the first sub-level is higher than the second sub-level, and in the flow task, the time window of the first sub-level is shorter than the time window of the second sub-level.

[0049] Optionally, in the flow task, the time window is determined based on the processing time and tolerance time corresponding to the tag processing level, the tolerance time is the longest delay time allowed by the message queue to receive the target tag metadata, and the tolerance time corresponding to the first sub-level is shorter than the tolerance time corresponding to the second sub-level.

[0050] Optionally, the device further comprises a monitoring unit, configured to:

[0051] Storing the tag of the target tag metadata in a database and determining a storage time;

[0052] Determining a time to acquire the target tag metadata;

[0053] Calculating the time difference between the storage time and the acquisition time;

[0054] If the time difference is greater than a preset time threshold, an alarm message is output.

[0055] In another aspect, the present application provides a computer device, the device comprising a processor and a memory:

[0056] The memory is used to store program code and transmit the program code to the processor;

[0057] The processor is used to execute the method described in the above aspects according to the instructions in the program code.

[0058] On the other hand, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method described in the above aspects.

[0059] On the other hand, an embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method described in the above aspects.

[0060] The advantages of the above technical solution of this application are:

[0061] Obtain a label production request including a target label usage scenario. If target label metadata applicable to the target label usage scenario is obtained, determine the target label processing level corresponding to the target label usage scenario based on the correspondence between the label usage scenario and the label processing level. Among them, the higher the real-time nature of the label usage scenario, the faster the corresponding label needs to be generated to meet the subsequent usage scenario. Thus, in the correspondence between the label usage scenario and the label processing level, the higher the real-time nature of the label usage scenario, the higher the corresponding label processing level. Therefore, the target label metadata is processed based on the target label processing level to obtain the label of the target label metadata, so that the processing speed of the label of the target label metadata matches the label processing level, so that the label of the obtained target label metadata can be applicable to the target label usage scenario.

[0062] Therefore, by pre-establishing the correspondence between label usage scenarios and label processing levels, the higher the real-time nature of the label usage scenarios, the higher the corresponding label processing levels. That is, the labels are divided based on the usage scenarios, so that when processing the label metadata, the usage time limit requirements of the labels in subsequent label usage scenarios can be fully considered, so that they can be processed according to the label processing level corresponding to the label usage scenario, and then the label time limit can be determined to meet the label usage scenario, thereby expanding the application scope of the label. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0064] Figure 1 A flowchart of a method for determining a label provided in an embodiment of the present application;

[0065] Figure 2 A schematic diagram of a label determination system provided in an embodiment of the present application;

[0066] Figure 3 A schematic diagram of a hierarchical configuration provided in an embodiment of the present application;

[0067] Figure 4 A schematic diagram of a label determination device provided in an embodiment of the present application;

[0068] Figure 5 A structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0070] In related technologies, in order to expand the application scope of tags, different software is used to process metadata according to its source. For example, offline metadata is processed based on Spark software to obtain tags; near-real-time metadata is processed based on Flink software to obtain tags, etc. In other words, metadata is divided into two types, offline and near-real-time, according to its source, so that different software is used to process it to ensure the timeliness of tag generation, so that tags can be generated and subsequent processing is performed based on the generated tags.

[0071] However, after analysis, it was found that the method of dividing metadata based on the source and then using the corresponding software to generate labels cannot meet the scenarios that require production performance indicators, such as the generated labels cannot be applied to subsequent usage scenarios. Taking the payment result page recommendation scenario as an example, it is necessary to determine the users who meet the labels based on the relevant data of the payment order that has just occurred and the corresponding labels, that is, after the user completes the payment, jump to the payment success page to display the recommended content. The process takes a short time. Even if the payment-related data is processed based on the Flink software, the recommendation after obtaining the label cannot meet the needs of the usage scenario. Therefore, in order to meet the scenarios that require production performance indicators, the labels can be divided based on the usage scenarios so that the generated labels can meet the subsequent usage scenarios.

[0072] Based on this, the embodiments of the present application provide a label determination method and related devices, which pre-establish a correspondence between label usage scenarios and label processing levels, so that the higher the real-time nature of the label usage scenarios, the higher the corresponding label processing level, that is, the labels are divided based on the usage scenarios, so that when processing the label metadata, the label usage time limit requirements in subsequent label usage scenarios can be fully considered, so that the label can be processed according to the label processing level corresponding to the label usage scenario, and then the label time limit is determined to meet the label usage scenario, that is, to meet the scenario with requirements for production performance indicators, thereby expanding the application scope of the label.

[0073] The tag determination method provided in this application can be applied to computer devices with tag determination capabilities, such as terminal devices and servers. Specifically, the terminal device can be a desktop computer, a laptop computer, a mobile phone, and a tablet computer; the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0074] Combine the following Figure 1 , a method for determining a label provided in an embodiment of the present application is introduced. Figure 1 , this figure is a flowchart of a label determination method provided in an embodiment of the present application, and the method may include S101-S104.

[0075] S101: Obtain a label production request including a target label usage scenario.

[0076] The label generation request is a request for determining the required label. For example, the label generation request can be "the payment type of the last order". The label generation request includes a target label usage scenario, which is the label usage scenario corresponding to the required label. The label usage scenario is the scenario in which the label is applied after the required label is generated. For example, the target label usage scenario included in the label generation request "the payment type of the last order" is the order payment scenario.

[0077] In addition, in the related art, adding new tags requires familiarity with the big data technology stack and going through a complete development process, which results in low label production efficiency. However, in the embodiment of the present application, the user only needs to send a label production request to generate the corresponding label, which reduces the requirements for the user and improves the user experience. Moreover, different levels of label metadata correspond to different processing, and there is no need for label metadata to go through a complete development process, which improves generation efficiency.

[0078] S102: Obtain target tag metadata corresponding to the target tag usage scenario according to the tag production request.

[0079] Tag metadata is metadata for tags. Metadata, also known as intermediary data or relay data, is data about data. It is mainly information describing data properties to support functions such as indicating storage location, historical data, resource search, and file records. Metadata is a kind of electronic catalog. In order to achieve the purpose of cataloging, it is necessary to describe and collect the content or characteristics of the data, thereby achieving the purpose of assisting data retrieval. Taking the member tags of the video platform as an example, tag metadata can include information such as the name of the tag, the classification of the tag, and the field where the tag is located. Tag metadata includes data applicable to a variety of tag usage scenarios. Target tag metadata is tag metadata applicable to target tag usage scenarios.

[0080] After obtaining a label production request, it is necessary to determine the label corresponding to the data based on the label production request, that is, to obtain the target label metadata corresponding to the target label usage scenario included in the label production request, so as to determine the label of the target label metadata. However, in order to reduce the pressure on the system caused by the label determination process, the labels can be divided into multiple categories, and the determination time limits of labels of different categories are different, so that the processing speeds corresponding to labels with different time limits are different. The following is a detailed explanation, see S103.

[0081] S103: Determine a target tag processing level corresponding to the target tag usage scenario according to the correspondence between the tag usage scenario and the tag processing level.

[0082] The correspondence between the tag usage scenario and the tag processing level is pre-built to describe the correspondence between the tag usage scenario and the tag processing level. In the correspondence between the tag usage scenario and the tag processing level, the higher the real-time nature of the tag usage scenario, the faster the tag needs to be determined, and the higher the corresponding tag processing level, so that the tag can be determined faster based on the higher tag processing level.

[0083] The target tag processing level is the tag processing level corresponding to the target tag usage scenario, which is the tag processing level corresponding to the target tag metadata, and is more suitable for the real-time requirements of the target tag usage scenario where the target tag metadata is located.

[0084] S104: Process the target tag metadata according to the target tag processing level to obtain a tag of the target tag metadata.

[0085] The embodiments of the present application do not specifically limit the method of determining the label of the target label metadata. Those skilled in the art can set it according to actual needs, and its processing speed can be consistent with the target label processing level. For example, if the target label processing level is high, it means that the target label metadata needs to be processed more quickly, and it is necessary to use a processing method that can quickly generate labels, such as classifying the target label metadata based on a pre-trained model to obtain the label of the target label metadata.

[0086] The embodiments of the present application do not specifically limit the method of determining the label corresponding to the metadata. For example, after determining the target label processing level corresponding to the target label metadata, the target label metadata is sent to the processor corresponding to the target label processing level, so as to add labels to the target label metadata based on the corresponding processor. For example, a processor with higher processing level requirements will generate production tasks for web services and calculate the labels into the warehouse as quickly as possible. A processor with general processing level requirements can generate corresponding high-speed channel flink task production labels or low-speed channel flink task production labels, etc. A processor with no processing level requirements will generate batch processing tasks and schedule production labels regularly, thereby matching the performance requirements of the usage scenario.

[0087] It can be seen from the above technical solution that a label production request including a target label usage scenario is obtained. If the target label metadata applicable to the target label usage scenario is obtained, then the target label processing level corresponding to the target label usage scenario is determined based on the correspondence between the label usage scenario and the label processing level. Among them, the higher the real-time nature of the label usage scenario, the faster the corresponding label needs to be generated to meet the subsequent usage scenario. Therefore, in the correspondence between the label usage scenario and the label processing level, the higher the real-time nature of the label usage scenario, the higher the corresponding label processing level. Therefore, the target label metadata is processed based on the target label processing level to obtain the label of the target label metadata, and the processing speed of the label of the target label metadata matches the label processing level, so that the label of the obtained target label metadata can be applicable to the target label usage scenario.

[0088] Therefore, by pre-establishing the correspondence between label usage scenarios and label processing levels, the higher the real-time nature of the label usage scenarios, the higher the corresponding label processing levels. That is, the labels are divided based on the usage scenarios, so that when processing the label metadata, the usage time limit requirements of the labels in subsequent label usage scenarios can be fully considered, so that they can be processed according to the label processing level corresponding to the label usage scenario, and then the label time limit can be determined to meet the label usage scenario, thereby expanding the application scope of the label.

[0089] As can be seen from the above, in order to expand the application scope of tags in related technologies, different software is used for processing according to the source of metadata. For example, offline metadata is processed based on Spark software to obtain tags; near real-time metadata is processed based on Flink software to obtain tags, etc. Among them, the processing speed of Spark software is slow, and the processing speed of Flink software is high, so that different processing speeds are used to determine tags for tag metadata from different sources.

[0090] However, both Spark and Flink software process label metadata through message queues, that is, message queues are used to store label metadata, and the label metadata stored first are processed first, and the label metadata stored later are processed later, that is, the "first in, first out" method, so that the label metadata included in the message queue is processed to obtain the label of the target label metadata.

[0091] That is to say, although different software is used to process different types of tag metadata at different processing speeds, both software processes tag metadata based on message queues. If the processing level of the tag metadata is high, it may not be able to wait in a "first in, first out" manner, and thus cannot meet some usage scenarios that require higher processing speeds.

[0092] Based on this, the embodiment of the present application sets a first level and a second level in the tag processing level, and the first level is higher than the second level. The processing methods corresponding to the two levels are described below respectively.

[0093] (I) Regarding the first level.

[0094] If the target tag processing level is the first level, that is, if the tag metadata with a higher tag processing level is obtained, the target tag metadata will no longer be processed in a "first in, first out" manner, but the tag of the target tag metadata will be determined first, so that the target tag metadata does not need to wait in the message queue, shortening the waiting time, satisfying the usage scenarios of the tags, and expanding the scope of application.

[0095] The embodiment of the present application does not specifically limit the method of preferentially determining the tag of the target tag metadata, and those skilled in the art can set it according to actual needs. The following two methods are used as examples for description.

[0096] Method 1: If the message queue is continued to be used for processing, the first-level target tag metadata can be added to the front of the message queue, which is equivalent to inserting the first-level target tag metadata to the front of the message queue so as to prioritize the tag of the target tag metadata.

[0097] Method 2: Although the first-level target tag metadata can be processed first by queue-jumping in method 1, the tag metadata in the message queue still needs to wait for the consumer to process. If the consumer does not have sufficient processing power, the first-level target tag metadata cannot be processed quickly even though it is at the front of the message queue. In other words, the asynchronous processing method cannot meet some high-real-time usage scenarios.

[0098] Based on this, the embodiment of the present application no longer uses asynchronous processing, but implements synchronous processing by setting interface permissions. Specifically, the tag access party is determined, and the tag access party is the producer of the target tag metadata, that is, the party that generates the target tag metadata, so as to set the tag write permission for the tag access party, wherein the tag write permission indicates that the tag access party has the permission to directly call the interface to generate the tag, and the interface can be an application programming interface (API), thereby realizing the interaction between the two software functions.

[0099] That is to say, if the tag access party has the tag write permission, then according to the tag write permission, the interface can be directly called to realize the function of generating tags, that is, the tag access party and the tag production function (such as a processor, etc.) are connected through the interface, so as to help the tag access party no longer need to wait for the consumer party (such as the corresponding tag generation function) to process and realize tag production through a queue, but can directly instruct the consumer party (that is, the processor with the tag production function) to process, that is, call the interface according to the tag write permission, and enable the consumer party to determine the tag of the target tag metadata through the interface. For example, based on the JAVA Web service implementation interface form, the tag access party directly calls the interface to synchronously return the response result. Thereby, the passive process of generating tags (that is, waiting for the consumer party to process) is converted into an active process (that is, directly instructing the consumer party to process), thereby improving the processing speed.

[0100] Therefore, when the tag access party generates the target tag metadata, since the target tag processing level corresponding to the target tag metadata is the first level, that is, the processing level is relatively high, the tag access party is set with tag write permissions, so that the tag access party can directly call the interface according to the tag write permissions, and directly connect to the consumer party through the interface, so that the consumer party can determine the label of the target tag metadata, that is, there is no need to store it through the message queue, but to process the tag metadata by directly instructing the consumer party, which is equivalent to synchronous processing, thereby avoiding the delay caused by asynchronous processing, and is more in line with the real-time nature of tag metadata with a higher processing level.

[0101] (ii) Regarding the second level.

[0102] If the target tag processing level is the second level, the target tag metadata can continue to be processed in a "first in, first out" manner, that is, the target tag metadata is added to the message queue so that the consumer can process it in the order of the tag metadata included in the message queue to obtain the label of the target tag metadata.

[0103] As a possible implementation, the embodiment of the present application sets the first level, the second level and the third level in the tag processing level, the first level is higher than the second level, and the second level is higher than the third level. The processing methods corresponding to the three levels are described below.

[0104] (1) If the target tag processing level is the first level, it means that the target tag processing level is relatively high, and the tag of the target tag metadata is determined first. Specifically, the above-mentioned method 1 or method 2 can be used to implement it, which will not be described in detail here.

[0105] (2) If the target tag processing level is the second level, the metadata belonging to the second level is equivalent to the near real-time metadata mentioned above, that is, there are some requirements for the processing speed, but the requirements are not high, and the label of the target tag metadata can be determined by the stream task. For example, the second-level target tag metadata can be streaming data with second-level performance requirements (such as Kafka, RocketMQ message data, etc.). Among them, the stream task can be a task implemented based on Flink software. Flink is a framework focused on real-time stream processing. It is often used with message queues to process real-time data streams. Flink is designed and optimized to achieve low-latency and high-throughput data processing. The processing method of the stream task is to use real-time stream data processing technology, which can calculate and process data immediately when it arrives. This method reduces the time for data to wait for processing, thereby improving the processing speed. By determining the label of the target tag metadata through the stream task, if there is no data accumulation in the queue, the data can be processed and responded to almost all the time, and the label of the target tag metadata can be quickly determined.

[0106] (3) If the target tag processing level is the third level, the metadata belonging to the third level is equivalent to the offline metadata mentioned above, that is, there is no requirement for the processing speed, and the label of the target tag metadata can be determined by the batch processing task. For example, the target tag metadata belonging to the third level can be metadata that needs to produce labels on a daily basis. Among them, the batch processing task can be a task implemented based on the Spark software. Although Spark supports reading and processing data from the message queue, its data processing method does not completely rely on the message queue. Spark provides a variety of computing modes, including batch processing, interactive query and real-time stream processing, among which real-time stream processing (such as structured streaming) can process real-time data streams from message queues. Batch processing tasks use offline processing technology and need to wait for the amount of data within a certain time range to accumulate to a certain extent before computing and processing. This processing method may cause a long delay when processing a large amount of data. Compared with the processing speed of stream tasks, the processing speed of batch processing tasks is relatively slow, because batch processing tasks need to wait until the data reaches a certain threshold or when the system pressure is low before processing. Although the processing speed is slow, it will not increase the processing pressure of the system.

[0107] It can be understood that in this embodiment, the combination of (2) the second level and (3) the third level can be equivalent to the (ii) second level in the aforementioned embodiment, because whether the label of the target label metadata is determined based on a stream processing task or a batch processing task, it can be regarded as a solution implemented through a message queue, that is, adding the target label metadata to the message queue so that it can be processed in the order of the label metadata included in the message queue to obtain the label of the target label metadata.

[0108] As a possible implementation, for (2), the second level can be further subdivided into a first sub-level and a second sub-level, where the first sub-level is higher than the second sub-level. Thus, in a stream task, the time window of the first sub-level is shorter than the time window of the second sub-level, such as the time window of the first sub-level is 4 seconds, and the time window of the second sub-level is 4 minutes.

[0109] Among them, the time window defines the time range in which the data stream is divided and processed. It is used to limit the triggering of certain actions or the execution of certain data processing tasks within a specific time period. For example, the processing speed of the first sub-level and the second sub-level is adjusted by a rolling time window, wherein the rolling time window refers to a time window of fixed size, and the starting position and the end position of the rolling time window are continuously rolled over as time goes by. For example, every five minutes is a rolling time window, the first window is 00:00-00:05, the second window is 00:01-00:06, and so on. Therefore, by controlling the time window, the tag processing level of the tag metadata can be further refined, thereby further reducing the pressure on the system.

[0110] As a possible implementation method, in a stream task, the sum of the time window and the tolerance time is the processing time corresponding to the label processing level, so that the time window can be determined based on the tolerance time and the processing time corresponding to the label processing level. The tolerance time is the longest delay time allowed for the message queue to receive target label metadata. For example, the processing time corresponding to the first sub-level is 5 seconds, and its corresponding tolerance time is 1 second, then the time window of the first sub-level is 4 seconds. In addition, since the first sub-level is higher than the second sub-level, the tolerance time of the first sub-level is shorter than the tolerance time of the second sub-level. For example, the processing time corresponding to the second sub-level is 5 minutes, and its corresponding tolerance time is 1 minute, then the time window of the second sub-level is 4 minutes.

[0111] Therefore, by setting the tolerance time, the processing capacity of tag metadata can be increased while minimizing the increase in the queue waiting time of messages, thereby improving the stability of the system.

[0112] As a possible implementation method, a detection alarm mechanism may be added to detect production performance problems, which is described in detail below.

[0113] In the related art, the execution time of Flink software or Spark software is generally detected. However, this execution time can only indicate the time from when Flink software or Spark software obtains the tag metadata to when it processes the tag metadata. There is a gap between this duration and the time from when the tag metadata is generated by the tag access party to when the corresponding tag is determined. The start times of the two are different, so the accuracy of the execution time determined based on the software is low.

[0114] Based on this, the embodiment of the present application stores the label of the target tag metadata to the database, determines the storage time, and then determines the acquisition time of the target tag metadata, and calculates the time difference between the storage time and the acquisition time. If the time difference is greater than the preset time threshold, it means that the corresponding processing process may have problems such as lag, resulting in a large time difference, and an alarm message is output.

[0115] That is to say, after the tag access party generates the tag metadata, it sends the tag metadata to the computer device (i.e., the execution entity) of the embodiment of the present application. The time when the computer device obtains the tag metadata is the actual start time of the tag metadata, and the time when the software in the related technology obtains the tag metadata is equivalent to the time of adding the target tag metadata to the message queue step in the embodiment of the present application. There is a difference between the two times, and the time difference determined by the embodiment of the present application is equivalent to end-to-end timing, which is more accurate, and thus detection based on a more accurate time difference is also more accurate.

[0116] In order to make the technical solution provided by the embodiments of the present application clearer, Figure 2 and Figure 3 The method for determining a label provided in an embodiment of the present application is described with an example.

[0117] In the embodiment of the present application, the tags are divided into four levels, namely, S, A, B and C. Among them, S is equivalent to the first level (1), A is equivalent to the first sub-level in the second level (2), B is equivalent to the second sub-level in the second level (2), and C is equivalent to the third level (3). Each level is described below.

[0118] S level: real-time synchronization level, such as recommendation on identity authentication page, payment result page and other pages, which requires synchronous production within the transaction to ensure completion before use.

[0119] Grade A: Nearly real-time within seconds, such as automatic renewal page, order recommendation, etc., production is completed within seconds (within 5S) after the business occurs and is ready for use.

[0120] Class B: Nearly real-time and minute-level, such as making recommendations after user behaviors such as watching movies and downloading, and completing production within minutes (within 5 minutes) after the business occurs, and meeting the conditions for use.

[0121] Level C: Offline level, such as member conversion probability recommendation and other predictions of user future behavior, etc., production is completed n days after the business occurs (n>=1 according to the label definition) and is ready for use.

[0122] See also Figure 2, which is a schematic diagram of a label determination system provided in an embodiment of the present application. The label determination system includes a grading device 210, a processing device 220, a label library 230 and a detection device 240, which are described below respectively.

[0123] The grading device 210 is used to obtain a label production request including a target label usage scenario; if the target label metadata corresponding to the target label usage scenario is obtained, the target label processing level corresponding to the target label usage scenario is determined according to the correspondence between the label usage scenario and the label processing level, and the target label metadata is sent to the module corresponding to the target label processing level in the processing device 220. For example, the front end of the grading device 210 is developed using Vue software, and the back end is developed using JAVA software. The grading device 210 includes a metadata management module 211, an access module 212, and a grading module 213.

[0124] The metadata management module 211 is used to obtain the tag production request and the target tag metadata, and can also implement the definition, classification, and authority management of tags.

[0125] The access module 212 is used to determine the tag access party corresponding to the target tag metadata, and can also be responsible for managing data source, message source information, etc.

[0126] See also Figure 3 , which is a schematic diagram of a hierarchical configuration provided in an embodiment of the present application. The system will display the configuration of processing tasks corresponding to different levels according to the linkage of the tag processing level. This configuration is implemented based on the access module 21. The S level needs to configure the relevant information and tag set of the tag access party; the A, B and C levels need to configure the message source, tag set, extraction configuration and task configuration. Among them, the message source is used to indicate the source of the message, such as the message sources of A and B levels are message queues and consumer groups, value acquisition methods, etc., and the message sources of C level are databases, data warehouses, table fields, etc. The tag set represents all tags corresponding to the target tag usage scenario. The extraction configuration is used to indicate how to extract fields from tag metadata, etc. The task configuration is used to indicate how to process tag metadata.

[0127] The grading module 213 is used to determine the target tag processing level corresponding to the target tag usage scenario according to the correspondence between the tag usage scenario and the tag processing level, and send the target tag metadata to the module corresponding to the target tag processing level in the processing device 220.

[0128] Specifically, the target tag metadata determined as level S is sent to the real-time tag processing module 221, the target tag metadata determined as level A is sent to the near real-time second-level tag processing module 222, the target tag metadata determined as level B is sent to the near real-time minute-level tag processing module 223, and the target tag metadata determined as level C is sent to the offline tag processing module 224.

[0129] The processing device 220 determines the label of the target label metadata according to the target label processing level and the target label metadata. That is, different processing modules in the processing device 220 are implemented in different technical ways to meet the scene performance index requirements, and finally the processed labels are stored in the label library 230. The processing device 220 includes a real-time label processing module 221, a near real-time second-level label processing module 222, a near real-time minute-level label processing module 223 and an offline label processing module 224.

[0130] Among them, after obtaining the target tag metadata of level S, the real-time tag processing module 221 determines the tag access party, sets the tag write permission for the tag access party, calls the interface according to the tag write permission, and determines the tag of the target tag metadata through the interface. After obtaining the target tag metadata of level A, the near real-time second-level tag processing module 222 determines the tag of the target tag metadata through the stream task, and the time window is set to 4 seconds by default. After obtaining the target tag metadata of level B, the near real-time minute-level tag processing module 223 determines the tag of the target tag metadata through the stream task, and the time window is 4 minutes. After obtaining the target tag metadata of level C, the offline tag processing module 224 determines the tag of the target tag metadata through the batch processing task. For example, according to the configured data source and value logic, the scheduled production label is completed, based on Hadoop (an open source framework) workflow and crontab (a command for setting periodically executed instructions) scheduled tasks, and Spark software implementation.

[0131] The tag library 230 is used to store tags of tag metadata. For example, for S-level target tag metadata, the processor Web storage service is directly associated.

[0132] The detection device 240 is used to collect the throughput, time and other indicators of the grading device 210 and the processing device 220, so as to calculate the relevant performance indicators such as the time consumption of label production, so as to intuitively display the production time consumption of each label, etc., to optimize the processing device 220 and configure alarms, perceive production performance problems, find bottlenecks and configure monitoring alarms to perceive problems, adjust and optimize, optimize resources, and ensure the quality of label production. Ensure that the quality of label production is guaranteed with reasonable resource costs. The detection device 240 includes a storage module 241, an alarm module 242, a display module 243 and a monitoring and collection module 244.

[0133] For example, the detection device 240 can be implemented based on Prometheus (an open source system monitoring and alarm toolkit) and Grafana (a monitoring instrument system). The monitoring collection module 244 integrates native Flink software and Spark software, uses the indicators and reporting methods provided by the metrics-core library (a class library that provides service performance detection tools), and provides Web services to facilitate the integration of software development kits (Software development kits, SDKs), such as based on micrometer (a performance data collection that provides a common API) and spring-boot-starter-actuator (a module for exposing its own information, its main function is to monitor and manage) encapsulation, the monitoring collection module 244 is integrated in the grading device 210 and the processing device 220, generates relevant indicators at key links such as metadata access and processing completion, and reports to the storage module 241 through Metric Reporter (for monitoring and reporting performance indicators of flow tasks). The storage module 241 implements Prometheus Pushgateway (a data collection method that uses passive push to obtain monitoring data), which is used to receive indicator data and provide it to Prometheus for pulling data for use. The display module 243 is implemented based on Grafana, and configures corresponding indicators to monitor the dashboard and charts. The alarm module 242 relies on the Grafana alarm mechanism to manage alarm rules.

[0134] It can be seen from the above technical solution that the embodiment of the present application has the following advantages:

[0135] (1) A universal platform solution for label production was designed to produce labels in a configurable manner, reduce learning costs, and significantly shorten the label production cycle.

[0136] (2) The design classifies tags according to usage scenarios and adapts them to different processing modules, which not only supports all business scenarios but also meets the requirements of business performance indicators.

[0137] (3) A monitoring and alarm mechanism was designed to intuitively display label production performance, identify bottlenecks, issue problem alarms, optimize production processes and resources, improve label quality, and reduce resource costs.

[0138] The above solution can be applied to the production of various labels, realizing label production within a relatively short period of time with simple configuration of labor costs and reasonable resource costs, and ensuring production quality. The grading based on business indicators can support full-scenario business and can be horizontally expanded with the development of business, helping to make accurate recommendations, efficiently support business, and improve member conversion and revenue.

[0139] In addition to the label determination method provided in the embodiment of the present application, a label determination device is also provided, such as Figure 4 As shown, it includes: an acquisition unit 401, a level determination unit 402 and a label determination unit 403;

[0140] The acquisition unit 401 is used to acquire a label production request including a target label usage scenario;

[0141] The acquisition unit 401 is further configured to acquire target tag metadata corresponding to the target tag usage scenario according to the tag production request;

[0142] The level determination unit 402 is used to determine the target tag processing level corresponding to the target tag usage scenario according to the correspondence between the tag usage scenario and the tag processing level; wherein, in the correspondence between the tag usage scenario and the tag processing level, the higher the real-time nature of the tag usage scenario, the higher the corresponding tag processing level;

[0143] The label determination unit 403 is used to process the target label metadata according to the target label processing level to obtain a label of the target label metadata.

[0144] It can be seen from the above technical solution that the label determination device provided in the embodiment of the present application includes an acquisition unit, a level determination unit and a label determination unit. The acquisition unit acquires a label production request including a target label usage scenario. If the target label metadata applicable to the target label usage scenario is acquired, the level determination unit determines the target label processing level corresponding to the target label usage scenario based on the correspondence between the label usage scenario and the label processing level. Among them, the higher the real-time performance of the label usage scenario, the faster the corresponding label needs to be generated to meet the subsequent usage scenario. Therefore, in the correspondence between the label usage scenario and the label processing level, the higher the real-time performance of the label usage scenario, the higher the corresponding label processing level. Therefore, the label determination unit processes the target label metadata based on the target label processing level to obtain a label of the target label metadata, and the label of the target label metadata can be applicable to the target label usage scenario.

[0145] Therefore, by pre-establishing the correspondence between label usage scenarios and label processing levels, the higher the real-time nature of the label usage scenarios, the higher the corresponding label processing levels. That is, the labels are divided based on the usage scenarios, so that when processing the label metadata, the usage time limit requirements of the labels in subsequent label usage scenarios can be fully considered, so that they can be processed according to the label processing level corresponding to the label usage scenario, and then the label time limit can be determined to meet the label usage scenario, thereby expanding the application scope of the label.

[0146] As a possible implementation manner, the label processing level includes a first level and a second level, and the first level is higher than the second level; the label determination unit 403 is specifically configured to:

[0147] If the target tag processing level is the first level, the tag of the target tag metadata is determined preferentially;

[0148] If the target tag processing level is the second level, the target tag metadata is added to the message queue so as to be processed in the order of the tag metadata included in the message queue to obtain the tag of the target tag metadata.

[0149] As a possible implementation manner, the tag determining unit 403 is specifically configured to:

[0150] Determine a tag access party, where the tag access party is a producer of the target tag metadata;

[0151] Setting a tag writing permission for the tag access party, wherein the tag writing permission indicates that the tag access party has the permission to directly call an interface to generate a tag;

[0152] The interface is called according to the tag writing permission, and the tag of the target tag metadata is determined through the interface.

[0153] As a possible implementation manner, the label processing level includes a first level, a second level and a third level, the first level is higher than the second level, and the second level is higher than the third level; the label determination unit 403 is specifically configured to:

[0154] If the target tag processing level is the first level, the tag of the target tag metadata is determined preferentially;

[0155] If the target tag processing level is the second level, determining the tag of the target tag metadata through a stream task;

[0156] If the target tag processing level is the third level, the tag of the target tag metadata is determined by a batch processing task, and the processing speed of the batch processing task is lower than the processing speed of the stream task.

[0157] As a possible implementation, the second level includes a first sublevel and a second sublevel, the first sublevel is higher than the second sublevel, and in the stream task, the time window of the first sublevel is shorter than the time window of the second sublevel.

[0158] As a possible implementation method, in the flow task, the time window is determined based on the processing time and tolerance time corresponding to the label processing level, the tolerance time is the longest delay time allowed by the message queue to receive the target label metadata, and the tolerance time corresponding to the first sub-level is shorter than the tolerance time corresponding to the second sub-level.

[0159] As a possible implementation, the device further includes a monitoring unit, which is used to:

[0160] Storing the tag of the target tag metadata in a database and determining a storage time;

[0161] Determining a time to acquire the target tag metadata;

[0162] Calculating the time difference between the storage time and the acquisition time;

[0163] If the time difference is greater than a preset time threshold, an alarm message is output.

[0164] The present application also provides a computer device, see Figure 5 , which shows a structural diagram of a computer device provided in an embodiment of the present application, such as Figure 5 As shown, the device includes a memory 510 and a processor 520:

[0165] The memory 510 is used to store program codes and transmit the program codes to the processor;

[0166] The processor 520 is used to execute any label determination method provided in the above embodiments according to the instructions in the program code.

[0167] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program is used to execute any one of the label determination methods provided in the above embodiments.

[0168] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the label determination method provided in various optional implementations of the above aspects.

[0169] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0170] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0171] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0172] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0173] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for determining a label, characterized in that: The method comprises: Obtain a label production request including a target label usage scenario; Acquire target tag metadata corresponding to the target tag usage scenario according to the tag production request; Determine the target tag processing level corresponding to the target tag usage scenario according to the correspondence between the tag usage scenario and the tag processing level; wherein, in the correspondence between the tag usage scenario and the tag processing level, the higher the real-time nature of the tag usage scenario, the higher the corresponding tag processing level; The target tag metadata is processed according to the target tag processing level to obtain a label of the target tag metadata.

2. The method according to claim 1, characterized in that The tag processing level includes a first level and a second level, the first level being higher than the second level; The step of determining the label of the target label metadata according to the target label processing level and the target label metadata includes: If the target tag processing level is the first level, the tag of the target tag metadata is determined preferentially; If the target tag processing level is the second level, the target tag metadata is added to the message queue so as to be processed in the order of the tag metadata included in the message queue to obtain the tag of the target tag metadata.

3. The method according to claim 2, characterized in that The step of preferentially determining the tag of the target tag metadata includes: Determine a tag access party, where the tag access party is a producer of the target tag metadata; Setting a tag writing permission for the tag access party, wherein the tag writing permission indicates that the tag access party has the permission to directly call an interface to generate a tag; The interface is called according to the tag writing permission, and the tag of the target tag metadata is determined through the interface.

4. The method according to claim 1, characterized in that: The label processing levels include a first level, a second level and a third level, the first level is higher than the second level, and the second level is higher than the third level; The step of determining the label of the target label metadata according to the target label processing level and the target label metadata includes: If the target tag processing level is the first level, the tag of the target tag metadata is determined preferentially; If the target tag processing level is the second level, determining the tag of the target tag metadata through a stream task; If the target tag processing level is the third level, the tag of the target tag metadata is determined by a batch processing task, and the processing speed of the batch processing task is lower than the processing speed of the stream task.

5. The method according to claim 4, characterized in that The second level includes a first sub-level and a second sub-level, the first sub-level is higher than the second sub-level, and in the flow task, the time window of the first sub-level is shorter than the time window of the second sub-level.

6. The method according to claim 5, characterized in that In the flow task, the time window is determined based on the processing time and tolerance time corresponding to the tag processing level. The tolerance time is the longest delay time allowed by the message queue to receive the target tag metadata. The tolerance time corresponding to the first sub-level is shorter than the tolerance time corresponding to the second sub-level.

7. The method according to claim 1, characterized in that The method further comprises: Storing the tag of the target tag metadata in a database and determining a storage time; Determining a time to acquire the target tag metadata; Calculating the time difference between the storage time and the acquisition time; If the time difference is greater than a preset time threshold, an alarm message is output.

8. A device for determining a label, characterized in that: The device comprises: an acquisition unit, a level determination unit and a label determination unit; The acquisition unit is used to acquire a label production request including a target label usage scenario; The acquisition unit is further used to acquire the target tag metadata corresponding to the target tag usage scenario according to the tag production request; The level determination unit is used to determine the target tag processing level corresponding to the target tag usage scenario according to the correspondence between the tag usage scenario and the tag processing level; wherein, in the correspondence between the tag usage scenario and the tag processing level, the higher the real-time nature of the tag usage scenario, the higher the corresponding tag processing level; The label determination unit is used to process the target label metadata according to the target label processing level to obtain a label of the target label metadata.

9. A computer device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the method according to any one of claims 1 to 7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method according to any one of claims 1 to 7.

Images