Data sampling method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a data sampling method and device, electronic equipment and storage medium. The method comprises: obtaining a target sampling task from a first message queue, wherein the first message queue is used to manage all to-be-processed sampling tasks, and each sampling task comprises a filtering condition for filtering a corresponding to-be-inspected object based on a time dimension; obtaining a target to-be-inspected object from a target cache queue according to a target filtering condition in the target sampling task; wherein the target cache queue is used to cache to-be-inspected objects obtained from a second message queue according to a preset mechanism, and the second message queue is used to manage all to-be-processed to-be-inspected objects. The method can timely perform sampling processing on to-be-inspected objects without affecting the stability of the system to obtain a target to-be-inspected object.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of Internet, and more particularly, to a data sampling method and device, an electronic device and a computer readable storage medium. The present disclosure also relates to a content label review method. BACKGROUND

[0002] At present, for the objects to be reviewed, the data review system generally needs to extract the objects to be reviewed that meet the requirements of the sampling inspectors from the original objects to be reviewed as the objects to be reviewed, and then the sampling inspectors review the objects to be reviewed.

[0003] For example, in a content generation and recommendation platform, for the content data generated by the user, in order to intelligently recommend the content data to different users, the reviewer generally assigns a label to the content data. In order to detect and ensure the accuracy of the label assigned to the content data by the reviewer, the sampling inspector generally needs to sample the label result of the content data assigned by the reviewer. The current data sampling method generally selects the objects to be reviewed that meet the sampling requirements of the sampling inspector from the batch of objects to be reviewed as the objects to be reviewed to be displayed to the sampling inspector for review after all or phased objects to be reviewed are warehoused.

[0004] Therefore, the existing data sampling method based on all or phased objects to be reviewed warehoused before sampling at least has the problems of large delay and not real-time. At the same time, this kind of method often needs to process a large amount of data in batches, and due to the limitation of hardware resources, it may also have the problem of long time consumption, which leads to slow system running and affects the stability of the system. SUMMARY

[0005] An object of the present disclosure is to provide a new technical solution for sampling data to solve the problems of large delay and possible impact on system stability that may exist in the existing data sampling method.

[0006] In a first aspect, the present disclosure provides a data sampling method, comprising:

[0007] obtaining a target sampling task from a first message queue, wherein the first message queue is used to manage all to-be-processed sampling tasks, and each sampling task includes a filtering condition for filtering a corresponding object to be reviewed based on a time dimension;

[0008] obtaining a target object to be reviewed from a target cache queue according to a target filtering condition in the target sampling task;

[0009] wherein the target cache queue is used to cache the objects to be reviewed obtained from a second message queue according to a preset mechanism, and the second message queue is used to manage all to-be-processed objects to be reviewed.

[0010] The second aspect of the present disclosure further provides a content label review method, which comprises:

[0011] obtaining and displaying a target content work order, wherein the target content work order comprises a first label set by a first user, and the target content work order is obtained according to the data sampling method of any one of claims 1-8;

[0012] receiving a second label set by a second user for the target content work order;

[0013] obtaining a target sampling result by judging whether the first label and the second label are consistent.

[0014] The third aspect of the present disclosure further provides a data sampling device, which comprises:

[0015] a sampling task obtaining module, configured to obtain a target sampling task from a first message queue, wherein the first message queue is used to manage all to-be-processed sampling tasks, and each sampling task comprises a filtering condition for filtering a corresponding to-be-inspected object based on a time dimension;

[0016] a target to-be-inspected object obtaining module, configured to obtain a target to-be-inspected object from a target cache queue according to a target filtering condition in the target sampling task;

[0017] wherein the target cache queue is used to obtain to-be-inspected objects from a second message queue according to a preset mechanism, and the second message queue is used to manage all to-be-processed to-be-inspected objects.

[0018] The fourth aspect of the present disclosure further provides an electronic device, which comprises:

[0019] a memory, configured to store executable instructions;

[0020] a processor, configured to control the electronic device to execute the method according to the first aspect or the second aspect of the present disclosure according to the executable instructions.

[0021] The fifth aspect of the present disclosure further provides a computer-readable storage medium, which stores a computer program readable and executable by a computer, and the computer program is used to execute the method according to the first aspect or the second aspect of the present disclosure when being read and executed by the computer.

[0022] One beneficial effect of the embodiments of the present disclosure is that, according to the embodiments of the present disclosure, the target sampling task is obtained from the first message queue for managing all to-be-processed sampling tasks, and then each to-be-inspected object in the target cache queue is filtered based on the target screening condition in the target sampling task, so that the target inspected object satisfying the target screening condition is obtained. In this method, since the target cache queue caches the to-be-inspected object obtained from the second message queue for managing all to-be-inspected objects according to a preset mechanism, when data sampling is performed, the method can not have to wait for all or phased to-be-inspected objects to be stored before sampling, but can timely sample the target inspected object satisfying the screening condition of the target sampling task in the first message queue by taking the second message queue as a first-level cache and the target cache queue as a second-level cache. Since the calculation pressure is dispersed in the whole data generation process, the method can enable the system to perform the sampling processing based on smaller calculation amount and fewer hardware resources, so as to avoid the problem that batch centralized processing of a large amount of data may cause the system stability.

[0023] Other features of the present specification and its advantages will become apparent from the following detailed description of exemplary embodiments of the present specification with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present specification and, together with the description, serve to explain the principles of the present specification.

[0025] Figure 1 is a schematic diagram of a framework of an existing data sampling system.

[0026] Figure 2 is a schematic diagram of a data sampling method provided by an embodiment of the present disclosure.

[0027] Figure 3 is a schematic diagram of a framework of a data sampling processing provided by an embodiment of the present disclosure.

[0028] Figure 4 is a schematic diagram of a process of screening an inspected object provided by an embodiment of the present disclosure.

[0029] Figure 5 is a schematic diagram of a content label review method provided by an embodiment of the present disclosure.

[0030] Figure 6 is a schematic diagram of a data sampling device provided by an embodiment of the present disclosure.

[0031] Figure 7 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the drawings. Note that the relative arrangement, numerical expressions, and numerical values of components and steps set forth in these embodiments are not limiting to the scope of the present disclosure unless otherwise specifically stated.

[0033] The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended to limit the scope of the application, its application, or uses.

[0034] Techniques, methods, and apparatus known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered part of the specification.

[0035] In all of the examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Thus, other examples of the exemplary embodiments can have different values.

[0036] Note that like numbers and letters refer to like elements throughout the several views of the drawings and, as such, no further discussions on them will be presented.

[0037] <Method Embodiment One>

[0038] Please refer to Figure 1 , which is a schematic diagram of the framework of the existing data sampling system. As Figure 1As shown, in the existing data sampling system, when implementing data sampling processing, the to-be-inspected object is generally written into the database, and the sampling task set by the sampler in the current system is written into the database; at the same time, a sampling resident process is set in the system, which usually detects whether there is a sampling task in the sampling task table of the database in the form of scanning the table; in addition, for the scanned sampling task, the method usually waits for all or part of the to-be-inspected objects to be warehoused, and then acquires the to-be-inspected objects that meet the sampling conditions in a one-time acquisition or phased acquisition manner through execution of a query statement to acquire the inspected objects. In this process, for the one-time acquisition of all to-be-inspected objects to construct the inspected objects, it can be marked as the sampling task being completed to avoid repeated processing in the next time; and for the phased acquisition of part of the to-be-inspected objects to construct the inspected objects, it often needs to maintain related flags to determine which to-be-inspected objects have been processed and which to-be-inspected objects have not been processed. It can be known that this kind of way has problems of large delay and not real-time on the one hand; on the other hand, when the sampling task level is large, it usually needs to perform the sampling task in parallel and increase the distributed lock when performing data sampling, which will cause the system complexity to greatly increase, not only needing to consume a long time, but also possibly causing the system to run slowly and affecting the stability of the system.

[0039] To solve the above problems, an embodiment of the present disclosure provides a data sampling method, please refer to Figure 2 which is a flowchart of the data sampling method provided by the embodiment of the present disclosure. The method can be implemented by an electronic device, which can be a server, for example, can be a blade server, a rack server, etc.; or, according to the needs, the electronic device can also be a terminal device, for example, can be a mobile phone, a tablet computer, etc., which is not specially limited here; it should be noted that in the embodiment of the present disclosure, the electronic device is exemplified as a server without special instructions.

[0040] As Figure 2 described, the method of the embodiment can include the following steps S2100-S2200, which are described in detail below.

[0041] Step S2100, acquiring a target sampling task from a first message queue, wherein the first message queue is used to manage all to-be-processed sampling tasks, and each sampling task includes a screening condition for screening a corresponding to-be-inspected object based on a time dimension; and step S2200, obtaining a target inspected object from a target cache queue according to a target screening condition in the target sampling task; the target cache queue is used to cache to-be-inspected objects acquired from a second message queue according to a preset mechanism, and the second message queue is used to manage all to-be-processed to-be-inspected objects.

[0042] For ease of understanding, please refer to Figure 3 which is a framework schematic diagram of the data sampling processing provided by the embodiments of the present disclosure, and the following will be combined with Figure 3 The method will be described in detail.

[0043] The first message queue can be a message queue built based on a message middleware and used for managing all sampling tasks, wherein the message middleware can be Kafka, RocketMQ, RabbitMQ, etc.

[0044] The second message queue can be a message queue built based on a message middleware and used for managing all to-be-inspected objects in the system.

[0045] It should be noted that in specific implementation, the message middleware used for building the first message queue and the second message queue can be the same middleware or different middleware, which is not specially limited here.

[0046] The target sampling task can include a filtering condition for filtering the corresponding to-be-inspected object based on a time dimension. In a preferred embodiment of the present disclosure, the creation time of the target sampling task needs to be earlier than the start time of the filtering condition for filtering the to-be-inspected object.

[0047] For example, if the sampling task is used to filter the to-be-inspected object after the time point of “xxxx-12-01 10:00:00”, the sampling task needs to be created before the time point, so that the corresponding to-be-inspected object generated in real time can be filtered based on the sampling task and output as an inspected object.

[0048] In specific implementation, the method can be applied to a content label auditing scene, in which the to-be-inspected object can be a to-be-inspected content work order including a first label set by a first user, for example, an auditor. Based on the method, a content work order meeting the target filtering condition in the target sampling task can be extracted from the to-be-inspected content work order as an inspected content work order and displayed to a second user, for example, a sampler for viewing. The sampler sets a second label for the inspected content work order and compares the first label and the second label to obtain a target auditing result indicating whether the first label set by the auditor is correct. Of course, the method can also be applied to other scenes as needed, for example, when the method is applied to a log monitoring and log matching scene, the to-be-inspected object can be business log data generated by a business system.

[0049] Please continue to refer to Figure 3In the embodiments of the present disclosure, in order to solve the problems of large delay and possible impact on system stability caused by centralized sampling processing in the prior art, when performing data sampling, the electronic device can manage the sampling tasks created in the system in real time based on the first message queue, and manage all to-be-inspected objects generated in the system in real time based on the second message queue. At the same time, in order to enable the system to flexibly and controllably extract to-be-inspected objects, the electronic device further sets a target cache queue as shown in Figure 3 The target cache queue is used to cache the to-be-inspected objects obtained by the message consumption process from the second message queue according to a preset mechanism, so that one or more, for example, n sampling processes in the system can perform filtering processing on each to-be-inspected object in the target cache queue based on the target sampling task allocated to each to-be-inspected object, to determine whether it is the to-be-inspected object of the corresponding sampling task. At the same time, based on the target cache queue, in the case of abnormal restart of the sampling process, the target cache queue can be reset based on the state information corresponding to each sampling task in the sampling process persistently stored in the state module, so that the system is flexible and controllable. The method is described in detail below.

[0050] In the following description, for the convenience of understanding, only one sampling process is described as shown in Figure 3 Of course, in specific implementation, the method does not limit the number of sampling processes used to perform sampling processing, that is, according to the order of magnitude of the to-be-inspected objects, the electronic device can set multiple sampling processes, and each sampling process can receive all to-be-processed sampling tasks broadcast by the first message queue and process all to-be-processed sampling tasks in a parallel processing manner. In addition, in the case where the electronic device includes multiple sampling processes, each to-be-inspected object in the target cache queue can be distributed to each sampling process for processing by a preset hash algorithm, for example, a hash algorithm, to avoid the problem of possible system performance degradation caused by uneven message distribution of the sampling process.

[0051] In one embodiment, the target to-be-inspected object is obtained from the target cache queue according to the target screening condition in the target sampling task, including: obtaining a first screening condition and a second screening condition from the target screening condition, wherein the first screening condition includes a target start time and a target end time, and the second screening condition is a screening condition based on a dimension other than time; obtaining the to-be-inspected object corresponding to the generation time within the target start time and the target end time range and satisfying the second screening condition from the target cache queue as the target to-be-inspected object.

[0052] Specifically, the target sampling task can include a first screening condition of a time dimension, and can also include a second screening condition based on other dimensions. For example, the first screening condition can be: the start time is "xxxx-12-01 10:00:00", and the end time is "xxxx-12-01 18:59:59", and the second screening condition can be: the auditor is "user01", and the object classification is "science and technology".

[0053] As shown in Figure 3 , in a specific implementation, each sampling process can process multiple sampling tasks at the same time, for example, 1-p sampling tasks. The sampling process can specifically process each sampling task by creating a coroutine for each sampling task.

[0054] Please see Figure 4 , which is a flowchart of obtaining a screened object provided by an embodiment of the present disclosure. As shown in Figure 4 , in one embodiment, the step of obtaining, from the target cache queue, a to-be-inspected object corresponding to a generation time within a target start time and a target end time range and satisfying a second screening condition as a target to-be-inspected object includes the following steps S2210-S2250.

[0055] Step S2210: obtaining any to-be-inspected object from the target cache queue as a to-be-determined to-be-inspected object.

[0056] Specifically, in the case where only one sampling process as shown in Figure 3 is provided in the electronic device, the sampling process can directly obtain a to-be-inspected object from the target cache queue as a to-be-determined to-be-inspected object, and give the target screening conditions of the target sampling task contained in the to-be-determined to-be-inspected object to screen the to-be-determined to-be-inspected object. In the case where multiple sampling processes are provided in the electronic device, the to-be-inspected objects in the target cache queue can be distributed to the target sampling processes in the multiple sampling processes through a preset hash algorithm, for example, through a hash algorithm. The target sampling process obtains the to-be-inspected object as a to-be-determined to-be-inspected object, and determines whether the to-be-determined to-be-inspected object can be a to-be-inspected object of one or more sampling tasks according to the corresponding screening conditions in the one or more sampling tasks processed by the target sampling process.

[0057] Step S2220: determining whether the generation time of the to-be-determined to-be-inspected object satisfies the first screening condition; and performing step S2230, if the generation time satisfies the first screening condition and the to-be-determined to-be-inspected object satisfies the second screening condition, the to-be-determined to-be-inspected object is determined as a first to-be-inspected object.

[0058] In the sampling process, for example, the target sampling process for processing the target sampling task obtains the to-be-determined to-be-sampled object, and can first determine whether the generation time, i.e., the creation time, of the to-be-determined to-be-sampled object meets the first screening condition, i.e., whether it is within the time range limited by the target start time and the target end time defined in the first screening condition. If it is, then it is determined according to the second screening condition whether it can be used as a matching record, i.e., a record that meets all the target screening conditions of the target sampling task. If the to-be-determined to-be-sampled object meets all the screening conditions of the target sampling task, it can be regarded as a matched record, i.e., a first to-be-sampled object, and the first to-be-sampled object is output as a target to-be-sampled object.

[0059] However, the screening conditions of the sampling task are often not set for a single to-be-sampled object. For example, for business log data, a log data usually only contains data of a business operation, but it is often necessary to sample whether multiple consecutive operations conform to the convention. Therefore, in the embodiments of the present disclosure, in order to enable the electronic device to process a sampling task that requires consecutive matching of multiple to-be-sampled objects, as shown in Figure 3 The state module is used to store state information corresponding to each sampling task in each sampling process. The state information can include historical matching records of the corresponding sampling task and the message offset of the first to-be-sampled object that is the currently matched record.

[0060] Specifically, after step S2230, step S2240 is performed, in which the target state information corresponding to the target sampling task is updated according to the first to-be-sampled object and the target task identifier of the target sampling task. The target state information includes historical matching records and a current message offset. The historical matching records are record information of the second to-be-sampled object that is filtered before the current time and meets the target screening condition. The current message offset is the message identifier corresponding to the first to-be-sampled object in the second message queue.

[0061] In this embodiment, the message offset of each matching record, i.e., each to-be-sampled object, is used to identify the position of the to-be-sampled object in the second message queue. In the embodiments of the present disclosure, the message offset can be the message identifier corresponding to the to-be-sampled object. Of course, in specific implementation, the message offset can also be in other forms, which are not specially limited here.

[0062] Specifically, in order to enable the electronic device to process a sampling task that requires consecutive matching of multiple to-be-sampled objects, as shown in Figure 3 The method further includes a state module for storing state information of each sampling task. Of course, as shown in Figure 3As shown, in the case where multiple sampling processes are provided in the electronic device, the state information of each sampling task corresponds to the sampling task under the corresponding sampling process, and for example, the state information of the corresponding sampling task under each sampling process can be managed in the form of (sampling process identifier, sampling task identifier, sampling task state information).

[0063] In step S2250, the target object is obtained according to the first to-be-inspected object and the target state information.

[0064] In one embodiment, after performing the above step S2220, i.e., determining whether the generation time of the to-be-determined to-be-inspected object satisfies the first screening condition, if the generation time of the to-be-determined to-be-inspected object does not satisfy the first screening condition and is later than the target end time, the target sampling task is ended, and the target object is obtained according to the historical matching record in the target state information.

[0065] Specifically, if the creation time of the to-be-determined to-be-inspected object currently obtained from the target cache queue is greater than the target end time in the screening condition of the target sampling task, it can be considered that the target sampling task has been completed, and at this time, the target sampling task can be ended, for example, the target sampling task is deleted from the first message queue or the processing state thereof is set to information indicating that the processing is completed.

[0066] In addition, in one embodiment, after performing the above step S2220, i.e., determining whether the generation time of the to-be-determined to-be-inspected object satisfies the first screening condition, if the generation time of the to-be-determined to-be-inspected object does not satisfy the first screening condition and is earlier than the target end time, the time interval between the generation time of the last matching record in the historical matching record in the target state information and the current time is obtained, and if the time interval is greater than a preset interval threshold, the target object is obtained according to the historical matching record in the target state information.

[0067] Specifically, for the target sampling task, in the process of processing each to-be-determined to-be-inspected object in a polling manner, the generation time of the last matching record matched in the target sampling task can also be obtained, and if the time interval between the generation time and the current time is greater than a preset interval threshold, it can be considered that the continuous operation specified in the target sampling task has not occurred within a period of time, and at this time, the historical matching record that has been matched can be output as the target object in advance to avoid the problem of large time delay.

[0068] In the above description, for each sampling task in each sampling process, for example, the target state information of the target sampling task, is stored in the form of caching in the memory. In an embodiment, to improve the stability of the system, the method further comprises: receiving an indication message indicating to persist the state information; in response to the indication message, suspending the step of obtaining any to-be-inspected object from the target cache queue as a to-be-determined to-be-inspected object; and in the case that the to-be-determined to-be-inspected object obtained at the current time is processed, persisting the target state information corresponding to the target sampling task to the target storage device; and after successfully persisting the target state information to the target storage device, continuing to execute the step of obtaining any to-be-inspected object from the target cache queue as a to-be-determined to-be-inspected object.

[0069] Specifically, as shown in Figure 3 To improve the stability of the system, the state module can send an indication message to each sampling process at a preset time interval to instruct each sampling process to perform state information persistence processing. After the sampling process receives the indication message, it can suspend the step of obtaining or receiving a to-be-determined to-be-inspected object from the target cache queue. After the to-be-determined to-be-inspected object at the current time flows through all the sampling tasks in the process, the sampling process sends the state information corresponding to all the sampling tasks it processes to the state module to trigger the state module to persist the state information in the storage device, for example, the disk.

[0070] In this embodiment, after successfully persisting the target state information of the target sampling task to the target storage device, the method further comprises: according to the historical matching record of the target state information, taking the second to-be-inspected object that has been screened at the current time as an intermediate result of the target to-be-inspected object, and outputting the intermediate result; and after successfully outputting the intermediate result, emptying the historical matching record to update the target state information.

[0071] Specifically, in this embodiment, after the sampling process completes the persistence, the to-be-inspected object that has been matched can be outputted as an intermediate result of the target to-be-inspected object according to the historical matching record in the state information corresponding to each sampling task in the sampling process, so as to avoid the problem of resource consumption caused by storing too many records in the memory for a long time.

[0072] That is, in order to reduce memory consumption, after the sampling process persists the state information corresponding to all sampling tasks in the sampling process to the storage device according to the received indication information indicating the persistent state information, the current filtered record can be output as an intermediate result of the target object for each sampling task according to the historical matching record in the state information corresponding to each sampling task. After outputting the intermediate result, the historical matching record in the state information corresponding to each sampling task in the sampling process can be emptied to avoid repeated output of the matching record after the persistent processing.

[0073] In one embodiment, the method further comprises: in the case of receiving an abnormal restart instruction, obtaining the historical state information corresponding to the target sampling task stored in the target storage device as the target state information at the current time; emptying the target cache queue, and reacquiring the target object from the target cache queue from the position identified by the current message offset in the target state information; and obtaining the target object to be detected according to the target filtering condition, the target state information, and the target cache queue performing the reacquisition processing.

[0074] Specifically, if the sampling process for processing the sampling task has an abnormality, an abnormal restart instruction can be sent to the sampling process at this time. After receiving the abnormal restart instruction, the sampling process first acquires the latest historical state information corresponding to the target sampling task from the target storage device, i.e., the state module for performing the persistent processing, and performs the abnormal pre-site recovery processing according to the historical state information. At this time, the state module can reset the starting message offset of the message consumption process to the value in the current message offset in the target state information, so as to re-consume the target object from the position represented by the message offset to the target cache queue.

[0075] It should be noted that in this embodiment, if multiple sampling processes are provided in the electronic device, the method further comprises: in the case of the abnormal restart instruction indicating that the target sampling process in the multiple sampling processes has an abnormal restart, after reacquiring the target object to the target cache queue, the other sampling processes perform filtering processing on the target object according to the message offset of the target object or the generation time of the target object after acquiring the target object from the target cache queue performing the reacquisition processing, wherein the other sampling processes are sampling processes other than the target sampling process.

[0076] Specifically, after the target sampling process is abnormally restarted, the target cache queue is rolled back, i.e., the target cache queue consumes the to-be-inspected objects from the second message queue again from the position indicated by the message offset indicated in the state information of the target sampling process after the abnormal recovery.

[0077] In a specific implementation, to save storage and network bandwidth resources, the to-be-inspected objects in the second message queue and the target cache queue can be to-be-inspected objects compressed or encrypted by using a preset compression algorithm, and data restoration processing is performed again in the sampling process. The detailed processing process is not described here. In addition, in a specific implementation, if the target cache queue is full, i.e., reaches the upper limit of the queue cache objects, the method can also slow down the speed of obtaining the to-be-inspected objects from the second consumption queue to the target cache queue, so as to reduce the system resource pressure and ensure the system stability.

[0078] In summary, the method provided by the embodiments of the present disclosure obtains the target sampling task from the first message queue for managing all to-be-processed sampling tasks, and then filters each to-be-inspected object in the target cache queue based on the target screening condition in the target sampling task, to obtain the target to-be-inspected object satisfying the target screening condition. In the method, since the target cache queue caches the to-be-inspected objects obtained from the second message queue for managing all to-be-inspected objects according to a preset mechanism, the method can not have to wait for all or phased to-be-inspected objects to be stored before sampling, but can timely sample the target to-be-inspected object satisfying the screening condition of the target sampling task in the first message queue by taking the second message queue as a first-level cache and the target cache queue as a second-level cache. Since the calculation pressure is dispersed in the whole data generation process, the method can enable the system to perform the sampling processing based on smaller calculation amount and fewer hardware resources, to avoid the problem that batch centralized processing of a large amount of data may cause the system stability.

[0079] <Method Embodiment Two>

[0080] Corresponding to the above-mentioned method embodiment one, in the embodiments of the present disclosure, a content label review method is also provided. Please refer to Figure 5 which is a flowchart of the content label review method provided by the embodiments of the present disclosure. The method can be implemented by an electronic device.

[0081] As Figure 5As shown, the method can include the following steps S5100-S5300, which are described in detail below.

[0082] Step S5100, obtaining and displaying a target content work order, wherein the target content work order includes a first label set by a first user, and the target content work order is obtained according to the data sampling method of method embodiment one.

[0083] Step S5200, receiving a second label set by a second user for the target content work order.

[0084] Step S5300, obtaining a target sampling result by judging whether the first label and the second label are consistent.

[0085] <Device Embodiment>

[0086] Corresponding to the above method embodiment one, in this embodiment, a data sampling device is also provided, such as Figure 6 The device 600 can include a sampling task acquisition module 610 and a target inspected object acquisition module 620.

[0087] The sampling task acquisition module 610 is configured to acquire a target sampling task from a first message queue, wherein the first message queue is configured to manage all to-be-processed sampling tasks, and each sampling task includes a filtering condition for filtering a corresponding to-be-inspected object based on a time dimension; and the target inspected object acquisition module 620 is configured to acquire a target to-be-inspected object from a target cache queue according to a target filtering condition in the target sampling task, wherein the target cache queue is configured to acquire to-be-inspected objects from a second message queue according to a preset mechanism, and the second message queue is configured to manage all to-be-processed to-be-inspected objects.

[0088] <Device Embodiment>

[0089] In this embodiment, an electronic device is also provided, such as Figure 7 The electronic device 700 can further include a processor 720 and a memory 710, the memory 700 is configured to store executable instructions; and the processor 720 is configured to control the electronic device to execute the method according to any embodiment of the present disclosure according to the instructions.

[0090] The electronic device 700 can be a server, or can also be a terminal device, which is not limited here.

[0091] <Computer-readable storage medium embodiment>

[0092] The embodiment provides a computer readable storage medium, and the storage medium stores executable commands. When the executable commands are executed by a processor, the method described in any method embodiment of the specification is executed.

[0093] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present specification. In this regard, each block in the flow diagrams and the block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logic function. In some alternative implementations, the functions noted in the boxes can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, can be implemented by dedicated hardware-based systems that carry out the specified functions or acts, or combinations of hardware and software.

[0094] The above has described the embodiments of the specification, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, practical application, or technical improvement in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein. The scope of the present application is defined by the appended claims.

Claims

1. A data sampling method, characterized by, The method comprises the following steps: obtaining a target sampling task from a first message queue, wherein the first message queue is used for managing all to-be-processed sampling tasks, and each sampling task comprises a filtering condition for filtering a corresponding to-be-inspected object based on a time dimension; obtaining a target to-be-inspected object from a target cache queue according to a target filtering condition in the target sampling task; wherein the target cache queue is used for caching to-be-inspected objects obtained from a second message queue according to a preset mechanism, and the second message queue is used for managing all to-be-processed to-be-inspected objects; the step of obtaining the target to-be-inspected object from the target cache queue according to the target filtering condition in the target sampling task comprises: obtaining any to-be-inspected object from the target cache queue as a to-be-determined to-be-inspected object; judging whether a generation time of the to-be-determined to-be-inspected object satisfies a first filtering condition, wherein the first filtering condition comprises a target start time and a target end time; if the generation time satisfies the first filtering condition and the to-be-determined to-be-inspected object satisfies a second filtering condition at the same time, taking the to-be-determined to-be-inspected object as a first to-be-inspected object, wherein the second filtering condition is a filtering condition based on a dimension other than the time dimension; updating target state information corresponding to the target sampling task according to the first to-be-inspected object and a target task identifier of the target sampling task, wherein the target state information comprises a historical matching record and a current message offset, the historical matching record is record information of a second to-be-inspected object that is filtered before a current time and satisfies the target filtering condition, and the current message offset is a message identifier corresponding to the first to-be-inspected object in the second message queue; obtaining the target to-be-inspected object according to the first to-be-inspected object and the target state information.

2. The method of claim 1, wherein, the step of obtaining the target to-be-inspected object from the target cache queue according to the target filtering condition in the target sampling task comprises: obtaining the first filtering condition and the second filtering condition from the target filtering condition; obtaining a to-be-inspected object corresponding to a generation time within a range of the target start time and the target end time and satisfying the second filtering condition from the target cache queue as the target to-be-inspected object.

3. The method of claim 2, wherein, the step of obtaining the to-be-inspected object corresponding to the generation time within the range of the target start time and the target end time and satisfying the second filtering condition from the target cache queue as the target to-be-inspected object further comprises: if the generation time does not satisfy the first filtering condition and the generation time is later than the target end time, ending the target sampling task, and obtaining the target to-be-inspected object according to the historical matching record in the target state information.

4. The method of claim 2, wherein, the step of obtaining the to-be-inspected object corresponding to the generation time within the range of the target start time and the target end time and satisfying the second filtering condition from the target cache queue as the target to-be-inspected object further comprises: In a case where the generation time does not satisfy the first screening condition and the generation time is earlier than the target end time, a time interval between a generation time of a last matching record in the historical matching records in the target state information and a current time is obtained; In a case where the time interval is greater than a preset interval threshold, the target inspected object is obtained according to the historical matching records in the target state information.

5. The method of claim 2, wherein, The method further comprises: receiving an indication message for indicating the persistent state information; in response to the indication message, suspending the step of obtaining any to-be-inspected object in the target cache queue as a to-be-determined to-be-inspected object; in a case where processing of the to-be-determined to-be-inspected object obtained at a current time is completed, persistently storing the target state information corresponding to the target sampling task into a target storage device; after successfully persistently storing the target state information into the target storage device, continuing to execute the step of obtaining any to-be-inspected object in the target cache queue as a to-be-determined to-be-inspected object.

6. The method of claim 5, wherein, after successfully persistently storing the target state information into the target storage device, the method further comprises: according to a historical matching record of the target state information, taking the second to-be-inspected object screened at a current time as an intermediate result of the target inspected object, and outputting the intermediate result; after successfully outputting the intermediate result, emptying the historical matching record to update the target state information.

7. The method of claim 5, wherein, The method further comprises: in a case where an abnormal restart instruction is received, obtaining historical state information corresponding to the target sampling task and stored in the target storage device as target state information at a current time; emptying the target cache queue and re-obtaining to-be-inspected objects from the target state information stored in the second message queue from a position identified by a current message offset in the target state information to the target cache queue; obtaining the target inspected object according to the target screening condition, the target state information, and the target cache queue on which the re-obtaining processing is performed.

8. The method of claim 7, wherein, The method processes sampling tasks through multiple sampling processes, and to-be-inspected objects in the target cache queue are allocated to the multiple sampling processes for processing through a preset hash algorithm; the method further comprises: in a case where the abnormal restart instruction indicates that a target sampling process in the multiple sampling processes abnormally restarts, after re-obtaining to-be-inspected objects to the target cache queue, other sampling processes perform filtering processing on the to-be-inspected objects according to message offsets of the to-be-inspected objects or generation times of the to-be-inspected objects after obtaining the to-be-inspected objects from the target cache queue on which the re-obtaining processing is performed, wherein the other sampling processes are sampling processes other than the target sampling process.

9. A content label review method characterized by, comprises: obtaining and displaying a target inspected content work order, wherein the target inspected content work order includes a first label set by a first user, and the target inspected content work order is obtained according to the data sampling method of any one of claims 1-8; receiving a second label set by a second user for the target inspected content work order; By judging whether the first label and the second label are consistent, a target sampling result is obtained.

10. A data sampling device, characterized by comprising: The method comprises the steps of: a sampling task acquisition module is configured to acquire a target sampling task from a first message queue, wherein the first message queue is configured to manage all sampling tasks to be processed, and each sampling task comprises a filtering condition for filtering a corresponding object to be inspected based on a time dimension; a target object acquisition module is configured to acquire a target object to be inspected from a target cache queue according to a target filtering condition in the target sampling task; wherein the target cache queue is configured to acquire an object to be inspected from a second message queue according to a preset mechanism, and the second message queue is configured to manage all objects to be inspected to be processed; the sampling task acquisition module is further configured to acquire any object to be inspected from the target cache queue as a to-be-determined object to be inspected; determine whether a generation time of the to-be-determined object to be inspected meets a first filtering condition, wherein the first filtering condition comprises a target start time and a target end time; if the generation time meets the first filtering condition and the to-be-determined object to be inspected meets a second filtering condition, the to-be-determined object to be inspected is taken as a first object to be inspected, and the second filtering condition is a filtering condition based on a dimension other than the time dimension; update target state information corresponding to the target sampling task according to the first object to be inspected and a target task identifier of the target sampling task, wherein the target state information comprises a historical matching record and a current message offset, the historical matching record is record information of a second object to be inspected that is filtered before the current time and meets the target filtering condition, and the current message offset is a message identifier corresponding to the first object to be inspected in the second message queue; and acquire the target object to be inspected according to the first object to be inspected and the target state information.

11. An electronic device, comprising: The method comprises the steps of: a memory is configured to store executable instructions; a processor is configured to control the electronic device to execute the method according to any one of claims 1-9 according to the instructions.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program that can be read and executed by a computer, and the computer program is configured to execute the method according to any one of claims 1-9 when being read and executed by the computer. The computer readable storage medium stores a computer program that can be read and executed by a computer, and the computer program is configured to execute the method according to any one of claims 1-9 when being read and executed by the computer.

Images