Question and answer collection method and device, electronic equipment and storage medium

Abstract

The application discloses a question and answer collection method and device, electronic equipment and a storage medium, relates to the technical field of computers, and in particular to the field of artificial intelligence such as deep learning and large models. The specific implementation scheme is as follows: an entry request of a first collection task of a business collection target is acquired; wherein the business collection target refers to a single collection intention of a question and answer result on a target question and answer service platform for the same question; an active task index corresponding to the business collection target is acquired according to a business key of the business collection target; wherein the active task index is used for recording a task identifier of a collection task currently allocated for the business collection target; whether the first collection task is written into a task queue corresponding to the target question and answer service platform is controlled based on the active task index; wherein the task queue is used for scheduling a collection task of a question and answer result of the target question and answer service platform.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to the field of artificial intelligence such as deep learning and large models, specifically to a question-and-answer acquisition method, device, electronic device and storage medium. Background Technology

[0002] With the rapid development of AI (Artificial Intelligence) and information retrieval technologies, question-answering systems have become an important way for users to obtain knowledge and instant answers. Various question-answering service platforms, including AI applications based on generative large language models, traditional search engines, and vertical domain question-answering engines, respond to user questions through natural language understanding and content generation capabilities. As question-answering services are widely used in search, customer service, and intelligent assistant scenarios, their output quality, stability, and consistency are receiving increasing attention. Summary of the Invention

[0003] This application provides a question-and-answer data collection method, apparatus, electronic device, and storage medium. The specific solution is as follows: According to one aspect of this application, a question-and-answer collection method is provided, comprising: The first collection task of the business collection target is enqueued; wherein, the business collection target refers to a single collection intention for the question and answer results of the same question on the target question and answer service platform; Based on the service key of the service collection target, obtain the active task index corresponding to the service collection target; wherein, the active task index is used to record the task identifier of the collection task currently assigned to the service collection target; Based on the active task index, control whether the first collection task is written into the task queue corresponding to the target question-and-answer service platform; wherein, the task queue is used to schedule the collection task of question-and-answer results of the target question-and-answer service platform.

[0004] According to another aspect of this application, a question-and-answer collection device is provided, comprising: The first acquisition module is used to acquire the queuing request of the first acquisition task of the business acquisition target; wherein, the business acquisition target refers to a single acquisition intention for the question and answer results of the same question on the target question and answer service platform; The second acquisition module is used to acquire the active task index corresponding to the service acquisition target based on the service key of the service acquisition target; wherein, the active task index is used to record the task identifier of the acquisition task currently assigned to the service acquisition target; The control module is used to control whether the first collection task is written into the task queue corresponding to the target question-and-answer service platform based on the active task index; wherein, the task queue is used to schedule the collection task of question-and-answer results of the target question-and-answer service platform.

[0005] According to another aspect of this application, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described in the above embodiments.

[0006] According to another aspect of this application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform the method described in the above embodiments.

[0007] According to another aspect of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in the above embodiments.

[0008] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0009] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this application. Wherein: Figure 1 A flowchart illustrating a question-and-answer data collection method provided in an embodiment of this application; Figure 2 A flowchart illustrating a question-and-answer data collection method provided in another embodiment of this application; Figure 3 A flowchart illustrating a question-and-answer data collection method provided in another embodiment of this application; Figure 4 A schematic diagram illustrating the correspondence between a business key and a task identifier provided in this application; Figure 5 A schematic diagram of an active task index-driven deduplication process provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a question-and-answer data collection device provided in one embodiment of this application; Figure 7 This is a block diagram of an electronic device used to implement the question-and-answer data collection method of the embodiments of this application. Detailed Implementation

[0010] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0011] It should be noted that the acquisition, storage, use, and processing of data in this application comply with the relevant provisions of national laws and regulations and do not violate public order and good morals.

[0012] The question-and-answer collection method, apparatus, electronic device, and storage medium of this application are described below with reference to the accompanying drawings.

[0013] Figure 1 This is a flowchart illustrating a question-and-answer data collection method provided in one embodiment of this application.

[0014] The question-and-answer collection method of this application embodiment can be executed by the question-and-answer collection device of this application embodiment, which can be configured in an electronic device.

[0015] Among them, electronic devices can be any device with computing capabilities, such as personal computers, mobile terminals, servers, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0016] like Figure 1 As shown, the question-and-answer data collection method includes: Step 101: Obtain the enqueue request for the first collection task of the business collection target.

[0017] In the optimization of generative question answering systems, it is usually necessary to collect questions in batches or collect questions and answers from multiple platforms, and then monitor, train or optimize the generative question answering system based on the obtained answers.

[0018] In this application, answers to the same question can be collected on one or more question-and-answer service platforms. The business collection target can refer to the single collection intent of the question-and-answer results of the same question on the target question-and-answer service platform.

[0019] The target question-answering service platform can be any question-answering service platform. Question-answering service platforms can include AI applications (such as generative question-answering systems based on large models), search engines, etc.

[0020] For example, a specific data collection target could be the Mth question in a batch, and the Nth round of data collection of the answer to that question on a target question-and-answer service platform. Here, N and M are positive integers.

[0021] In this application, an enqueue request may refer to a request used to request entry into the task queue corresponding to the target question-and-answer service platform.

[0022] For example, the task queue can be divided by platform dimension, with each Q&A service platform maintaining its own task queue. Business collection targets from different batches, questions, or rounds within the same platform share the task queue of that Q&A service platform, while the task queues of different Q&A service platforms are independent of each other. Task processing units (such as Workers) can retrieve and execute collection tasks from the task queue of that Q&A service platform. The retrieval speed of task processing units can be controlled through platform capacity snapshots and retrieval quotas. These task processing units can be implemented as background worker processes or service instances, etc.

[0023] In this application, one or more collection tasks can be initiated for the same business collection target, and the first collection task can be any collection task.

[0024] For example, the first acquisition task can be a normal acquisition task or a retry acquisition task, etc., without limitation.

[0025] Among them, normal collection tasks can be collection tasks generated for the first time for business collection targets, while retry collection tasks (or supplementary collection tasks) can refer to collection tasks that are regenerated when it is necessary to supplement or reacquire the original collection results.

[0026] The retry acquisition task can include automatic retry acquisition tasks and manual retry acquisition tasks. Automatic retry acquisition tasks are automatically generated by the system according to rules, while manual retry acquisition tasks are triggered by manual operation.

[0027] Step 102: Obtain the active task index corresponding to the business collection target based on the business key of the business collection target.

[0028] In this application, the business key (or business identifier) ​​can be used to uniquely identify the business collection target.

[0029] For example, each business acquisition target maintains its own active task index, which can be used to record the task identifier of the acquisition task currently assigned to the business acquisition target. Here, "record" can also be understood as "storage".

[0030] For example, the value of the active task index can be the task identifier of the collection task currently assigned to the collection target of this service.

[0031] Among them, "the collection task assigned to the business collection target" can refer to the collection task that has been accepted by the task scheduling unit and associated with the business collection target. The task identifier of the assigned collection task has been recorded in the active task index, and the collection task that is refused to be enqueued is not considered as an assigned task.

[0032] For example, the active task index can be derived from the business key of the business collection target, so that the active task index corresponding to the business collection target can be queried according to the business key.

[0033] As an example, an active task index can be generated based on the business key of the business acquisition target, according to the target rules. For instance, the active task index can be obtained by concatenating a fixed prefix or namespace with the business key of the business acquisition target.

[0034] Assuming the business key is business_key, the active task index could be "active:{business_key}".

[0035] In this application, the existence of an active task index for a service acquisition target is determined by whether an acquisition task has been assigned to it. If an acquisition task has been assigned to the service acquisition target, the active task index exists and is used to record the task identifier of the acquisition task; if no acquisition task has been assigned to the service acquisition target, or its historical acquisition tasks have been explicitly cleared, the active task index does not exist.

[0036] It is understandable that if the active task index exists, the collection task currently assigned to the business collection target is the collection task indicated by the active task index.

[0037] Step 103: Based on the active task index, control whether the first collection task is written into the task queue corresponding to the target question and answer service platform.

[0038] Since the active task index can record the task identifier of the collection task currently assigned to the business collection target, in this application, the existence of the active task index and / or the task status of the collection task recorded in the active task index can be used to control whether the first collection task is written into the task queue corresponding to the target question and answer service platform.

[0039] The task queue can be used to schedule tasks for collecting question and answer results from the target question and answer service platform.

[0040] For example, answers to questions can be collected on the target question-and-answer service platform by executing a collection task in the task queue corresponding to that platform.

[0041] The question-and-answer collection method of this application can be applied to scenarios such as batch question collection, multi-platform mobile content collection, automated question-and-answer collection, and supplementary collection and retry system.

[0042] For example, for the same question, data collection tasks can be performed in parallel on multiple question-and-answer service platforms to obtain the answers returned by each platform. By comparing the answer results of different question-and-answer service platforms horizontally and analyzing the time sequence, data monitoring and diagnosis of the question-and-answer service platforms can be achieved.

[0043] As an example, specific applications may include, but are not limited to: Anomaly detection: Identifies whether the response content is missing, truncated, empty, or has an abnormal format; Accuracy assessment: Determine whether the answer contains factual errors, logical contradictions, or inconsistencies with authoritative sources of knowledge; Policy change awareness: Monitor whether the Q&A service platform returns templated rejections, restrictive prompts, or error codes due to risk control policies, content security filtering, or service degradation; Model behavior analysis: By collecting data over a long period of time, we analyze whether generative AI models exhibit performance degradation, output drift, or changes in sensitivity to specific problems; Cross-platform consistency verification: Evaluate the differences in answers to the same question across different question-and-answer service platforms to help determine whether the discrepancies are caused by algorithm updates, data source switching, or deployment configuration changes.

[0044] The collected results can be used as a diagnostic basis, enabling operations, product, or algorithm teams to quickly locate the root cause of problems (such as model version rollback, interface compatibility failure, third-party service anomalies, etc.), and further used for service quality assessment, A / B testing effect verification, and generative engine optimization strategy iteration, thereby improving the stability, reliability, and user experience of the question-answering system.

[0045] In this embodiment, upon receiving a queuing request for the first data collection task, the active task index of the business data collection target is obtained based on the business key of the business data collection target. Based on this active task index, the system controls whether the first data collection task is written into the task queue of the target question-and-answer service platform. Therefore, by introducing an active task index for the business data collection target and using it as a unified basis for controlling the queuing of data collection tasks under that business data collection target, centralized management of the queuing behavior of data collection tasks for the same business data collection target can be achieved. This ensures the orderly scheduling of tasks and significantly improves the accuracy of data collection task scheduling and resource utilization efficiency.

[0046] Figure 2 This is a flowchart illustrating a question-and-answer data collection method provided in another embodiment of this application.

[0047] like Figure 2As shown, the question-and-answer data collection method includes: Step 201: Obtain the enqueue request for the first collection task of the business collection target.

[0048] Step 202: Obtain the active task index corresponding to the business collection target based on the business key of the business collection target.

[0049] In this application, steps 201-202 can be implemented in any of the embodiments of this application, so they will not be described again.

[0050] In some embodiments, the business key may be generated by the following method: obtaining the collection request of the business collection target, and obtaining the batch identifier to which the question belongs, the question identifier, the target Q&A service platform identifier, the collection round of the question, etc. according to the collection request, and then generating the business key according to the batch identifier, the question identifier, the target Q&A service platform identifier, and the collection round of the question.

[0051] The collection request may include information such as the batch identifier of the question to which the answer to be collected belongs, the question identifier, the target question-and-answer service platform, and the planned collection round.

[0052] For example, the batch identifier, the question identifier, the target question-and-answer service platform identifier, and the question collection round can be concatenated in a preset order to obtain the business key.

[0053] As an example, the business key `business_key` = `batch_id` + `question_id` + `platform_id` + `round_index`. Here, `batch_id` represents the batch identifier, `question_id` represents the question identifier, `platform_id` represents the identifier of the question-and-answer service platform, and `round_index` represents the round of data collection.

[0054] For example, batch_id=B20260601001, where B2026060100 represents a batch; question_id=Q001, where Q001 represents the first question in that batch; platform_id=aa, where aa represents the identifier of the question-and-answer service platform (e.g., the platform name); and round_index=1, indicating the first round of data collection. This business key uniquely corresponds to a single data collection target: "Perform the first round of data collection on platform aa for the first question in batch B2026060100." If the same question needs to be collected in multiple rounds on the same question-and-answer service platform, different business keys are generated by incrementing round_index (e.g., 2, 3, 4, etc.), with each round of collection corresponding to an independent business collection target.

[0055] In this embodiment, by generating business keys based on key dimensions such as batch identifier, question identifier, target question-and-answer service platform identifier, and collection round, a precise, unambiguous, and reproducible unique identifier for the business collection target can be achieved. Since each field characterizes the collection task in terms of time, content, platform, and round, the generated business key can distinguish collection intentions in different contexts. Even if the questions are the same, as long as the batch, target platform, or collection round is different, they are considered independent business collection targets. This method of determining business keys not only provides a reliable basis for task deduplication, status tracking, and concurrency control, but also supports cross-dimensional data aggregation and analysis, significantly improving the manageability, scalability, and multi-scenario adaptability of the collection system.

[0056] Optionally, in addition to generating business keys based on batch identifier, question identifier, target question and answer service platform identifier, and collection round, one or more of the following information can also be combined to generate business keys: the identifier of the project to which the batch belongs, the identifier of the tenant, the language used in the answer, the identifier of the geographical region, etc.

[0057] The tenant's identifier can be used to identify different customers or business entities using the target question-and-answer service platform.

[0058] For example, the batch identifier, question identifier, Q&A service platform identifier, collection round, batch to which the project belongs, tenant identifier, language used in the answer, and geographical region identifier can be concatenated into strings according to a preset concatenation order to generate a business key.

[0059] Step 203: Determine whether there are currently any active collection tasks for the business collection target based on the active task index.

[0060] In this application, an active acquisition task can refer to an acquisition task that is in an incomplete state as pointed to by the active task index.

[0061] For example, the incomplete state may include the following states: pending, delayed pending, running, etc.

[0062] The pending execution status indicates that the data acquisition task has entered the task queue and is waiting to be scheduled for execution; the delayed execution status indicates that the data acquisition task has entered the task queue, but the data acquisition task has triggered a delay policy and will not participate in scheduling for a specified waiting period; the execution status indicates that the data acquisition task has been picked up by the task processing unit and is currently performing the data acquisition operation.

[0063] In this application, it can be determined whether there is an active collection task for the business collection target based on whether the active task index exists and the task status of the second collection task indicated by the active task index.

[0064] In some embodiments, if the active task index does not exist, it can be determined that there is currently no active collection task for the service collection target; if the active task index exists, it can be determined whether the second collection task indicated by the active task index exists.

[0065] Due to factors such as data loss or restarts, automatic cleanup of expired tasks, or task deletion due to anomalies without synchronized cleanup of the active task index, the collection task indicated by the active task index may not exist. Therefore, if the second collection task indicated by the active task index does not exist, it can be determined that there are currently no active collection tasks for the business collection target. If the second collection task indicated by the active task index exists, the task status of the second collection task is obtained, and based on the task status, it is determined whether there are currently active collection tasks for the business collection target.

[0066] The second acquisition task indicated by the active task index can be the acquisition task identified by the value of the active task index.

[0067] In practical applications, there may be situations where the active task index indicates that the collection task is in a completed or failed state, but the active task index has not yet been cleared. Based on this, for example, if the task status indicates that the second collection task is in an incomplete state, it is determined that there are currently active collection tasks for the business collection target; if the task status indicates that the second collection task is in a completed or failed state, it is determined that there are currently no active collection tasks for the business collection target.

[0068] In other words, if the second collection task indicated by the active task index exists, and the task status of the second collection task is incomplete, it is determined that there is currently an active collection task for the business collection target; if the task status of the second collection task is completed or failed, it is determined that there is currently no active collection task for the business collection target.

[0069] Therefore, when the second collection task pointed to by the active task index exists and is in an incomplete state, it is determined that the business collection target has an active collection task. If the collection task is in a completed or failed state, it is considered that the business collection target currently has no active collection task, regardless of whether its record is retained. This allows for accurate determination of whether the business collection target has an active collection task even in abnormal or delayed cleanup scenarios such as active task index residues, avoiding misjudgments caused by outdated active task indexes. It provides a basis for subsequent judgment on whether to allow the first collection task to be enqueued, and can improve the system's fault tolerance and robustness of state judgment while ensuring the correctness of concurrency control.

[0070] In this embodiment, a preliminary screening is first performed based on whether an active task index exists; if an active task index exists, it is further determined whether the second collection task it points to exists; when the second collection task exists, a final determination is made based on its specific task status, thereby achieving accurate identification of the active collection task of the business collection target through a hierarchical and progressive status judgment mechanism.

[0071] Step 204: In response to determining that there are currently active collection tasks for the business collection target, refuse to write the first collection task into the task queue.

[0072] In actual question-and-answer data collection, for the same business data collection target, there may be situations where normal data collection tasks are submitted repeatedly, or supplementary data collection tasks are submitted concurrently. Therefore, in this application, when it is determined that there are currently active data collection tasks for the business data collection target, the first data collection task can be refused to be written into the task queue, thereby avoiding parallel and repeated data collection for the same business data collection target.

[0073] Step 205: In response to determining that there are currently no active collection tasks for the business collection target, the first collection task is written into the task queue, and the active task index is set as the task identifier of the first collection task.

[0074] In this application, setting the active task index as the task identifier of the first acquisition task can mean updating the value of the active task index to the task identifier of the first acquisition task.

[0075] In this application, when it is determined that there is no active collection task for the business collection target, the first collection task can be written into the task queue and the active task index can be set as the task identifier of the first collection task. This ensures that there is at most one active collection task for the same business collection target at any time, avoids multiple collection tasks from collecting data repeatedly, and improves resource utilization.

[0076] For example, when the first data collection task request enters the task queue, the active task index corresponding to the business data collection target is queried. If the second data collection task pointed to by the active task index exists and the task status of the second data collection task is incomplete, it means that there is currently an active data collection task for the business data collection target, and the first data collection task is refused to be written into the task queue. If the second data collection task pointed to by the active task index does not exist, or the task status of the second data collection task pointed to is complete or failed, the first data collection task can be written into the task queue, and the active task index is set as the task identifier of the first data collection task.

[0077] To improve the accuracy of active data collection task identification, optionally, the active task index and the task status of the second data collection task indicated by the active task index can be queried in real time. If the second data collection task does not exist, or the task status of the second data collection task indicated by the active task index is completed or failed, the active task index can be cleared. Therefore, when a new data collection task is enqueued, since the active task index does not exist, it indicates that there are currently no active data collection tasks for the business data collection target. Thus, the new data collection task can be written to the task queue, and the active task index can be set as the task identifier of the new data collection task, ensuring that there is at most one active data collection task for the same business data collection target.

[0078] For example, a real-time query can be a periodic query or a query triggered in other ways, without limitation.

[0079] Taking a normal data acquisition task as an example, when a normal data acquisition task enters the task queue, if the active task index does not exist, the active task index is set to the task identifier of the normal data acquisition task. After the task execution unit completes the normal data acquisition task, it marks the task status of the normal data acquisition task as either completed or failed, and releases the active task index.

[0080] In related technologies, a business key is typically used directly as the deduplication key: as long as the collection task record corresponding to the business key exists (regardless of the current status of the collection task), the business collection target is considered to have been processed, thus preventing new collection tasks for that business key from entering the task queue again. While this prevents the same business collection target from being executed concurrently and repeatedly, when a historical collection task has ended (regardless of whether the collection task was successful or failed) and the business side needs to re-initiate supplementary collection for that business collection target, the new collection task will be mistakenly judged as a duplicate and blocked because the historical record of the business key still exists, preventing legitimate supplementary collection tasks from entering the task queue.

[0081] In this embodiment, when the task status of the acquisition task indicated by the active task index is in a completed or failed state, the active task index can be cleared, allowing the generation of supplementary acquisition tasks (i.e., retry acquisition tasks) and allowing supplementary acquisition tasks to be written into the task queue. This supports legitimate supplementary acquisition after the historical acquisition tasks have ended, solving the problem of legitimate supplementary acquisition being blocked due to historical acquisition tasks in a completed or failed state. This can significantly improve the reliability, fault tolerance, and scheduling flexibility of acquisition.

[0082] In this embodiment, a new collection task is written into the task queue and its active task index is updated to its task identifier only when it is confirmed that there are no active collection tasks for the service collection target; if there are active collection tasks, the new collection task is refused to be written into the queue. Therefore, by dynamically determining whether there are active collection tasks for the service collection target based on the active task index, and deciding whether to allow the first collection task to be queued accordingly, it can be ensured that at most one active collection task exists for the same service collection target at any given time. This effectively avoids concurrent and repeated collection for the same service collection target, ensuring the uniqueness of task execution and the rational utilization of system resources.

[0083] In some embodiments of this application, the first acquisition task described above can be an automatic retry acquisition task, as described below. Figure 3 This section explains the method for generating the first data acquisition task when the first acquisition task is an automatically retried acquisition task. Figure 3 This is a flowchart illustrating a question-and-answer data collection method provided in another embodiment of this application.

[0084] like Figure 3 As shown, this question-and-answer data collection method also includes: Step 301: Query the business status of the business data collection target in real time.

[0085] In this application, the business status may include, but is not limited to, completed, failed, and pending try status.

[0086] Among them, the business status is "Completed" to indicate that complete and valid Q&A content has been collected; the business status is "Failed" to indicate that the business side has determined it to be a final failure after retrying the collection (for example, the maximum number of retry collection attempts has been reached, or the platform returns an unrecoverable error, such as content violation, account ban, or other permanent issues); the business status is "Pending Retry" to indicate that the current collection results do not meet the business requirements (such as missing content, or temporary platform traffic restrictions, etc.), and it is necessary to supplement or re-acquire the data by initiating a retry collection task.

[0087] For example, a real-time query can refer to a periodic query or a check triggered by other means, without limitation.

[0088] Step 302: In response to the business status being pending retry and the active task index not existing, generate the first collection task.

[0089] In this application, if the business status is pending retry and the active task index does not exist, a first collection task can be generated to re-collect data through the first collection task.

[0090] If the business status is pending retry, but the active task index exists and the task status of the second collection task indicated by the active task index is completed or failed, then the active task index can be cleared first, and then the first collection task can be generated.

[0091] For example, for any data collection task, a task identifier can be generated based on the business key of the business data collection target, the execution type of the data collection task, the number of retry data collections to which the data collection task belongs, etc.

[0092] The execution type can include, but is not limited to, normal data collection, automatic retry data collection, and manual retry data collection.

[0093] As an example, the business key of the business data collection target, the execution type of the data collection task, and the number of retries to which the data collection task belongs can be concatenated according to a preset concatenation order to generate the task identifier of the data collection task.

[0094] For example, if attempt_type=normal and retry_seq=0, it indicates that this is the first (and currently the only) normal data collection task for this business objective. If subsequent data collection is needed, a new data collection task identifier can be generated with attempt_type=retry (or manual_retry) and retry_seq incrementing from 1 to distinguish it from the normal data collection task.

[0095] The following is combined with Figure 4 Explain the mapping relationship between business keys and task identifiers. For example... Figure 4 As shown, one business key can correspond to multiple task identifiers, which can include task identifiers for normal data collection tasks, task identifiers for automatically retried data collection tasks, and task identifiers for manually retried data collection tasks. Furthermore, the active task index only constrains currently incomplete data collection tasks.

[0096] In some embodiments, when generating the first acquisition task, a task identifier for the first acquisition task may also be generated.

[0097] For example, the task identifier of the first data acquisition task can be generated by the following method: the execution type of the first data acquisition task and the number of retry acquisitions to which the first data acquisition task belongs can be determined, and the task identifier of the first data acquisition task can be generated according to the business key, execution type and number of retry acquisitions.

[0098] As an example, the task identifier of the first data collection task can be obtained by concatenating strings such as business key, execution type and number of retry collections according to the preset concatenation order.

[0099] In this embodiment, by determining the execution type of the first acquisition task and its retry acquisition count, and generating a task identifier in conjunction with the business key, since the task identifier not only inherits the business context represented by the business key, but also integrates the execution semantics and the retry acquisition count, even if multiple acquisition tasks correspond to the same business objective, they can be accurately identified and independently managed according to their triggering reasons and retry order. This achieves refined differentiation and unique identification of different acquisition intentions under the same business acquisition objective, providing a reliable basis for task tracking, state isolation, result attribution, and anomaly analysis, and can effectively support the traceability and controllability of tasks in complex acquisition scenarios.

[0100] Optionally, if the business status of the business acquisition target is completed or failed, and there is already a clear business conclusion, no retry acquisition task will be generated.

[0101] "Clear business conclusions" can refer to the final business results that have been determined by the business data collection target and do not require re-collection.

[0102] Optionally, after generating the first data acquisition task, the metadata of the first data acquisition task can be stored. For example, the metadata of the first data acquisition task may include, but is not limited to, the task identifier of the first data acquisition task, the business key of the business acquisition target, the parameters of the first data acquisition task, and the task status of the first data acquisition task.

[0103] For example, the metadata of each collection task, including the first collection task, can be stored in a storage system (such as Redis).

[0104] Because there is a time window between determining that the active task index does not exist and writing the first acquisition task into the task queue, other acquisition tasks (such as another concurrent normal acquisition submission, or another concurrently triggered supplementary acquisition judgment) may preemptively occupy the active task index within this time window.

[0105] Based on this, when the first data acquisition task request enters the task queue, the active task index can be queried. If the data acquisition task indicated by the active task index exists and the status of the data acquisition task is incomplete, then the first data acquisition task will not be written into the task queue. If the active task index does not exist, or the data acquisition task indicated by the active task index does not exist, or the indicated data acquisition task is in a completed or failed state, then the first data acquisition task will be written into the task queue, and the active task index will be set as the task identifier of the first data acquisition task.

[0106] In some embodiments, if the collection results need to be supplemented, the business status of the business collection target can be updated to a pending retry status first, then the current active collection task can be ended and the active task index released. Subsequently, an automatic retry collection task can be generated and written to the task queue, and the active task index can be set as the task identifier of the automatic retry collection task. If the pending retry status of the business collection target has been written, but the metadata storage of the automatic retry collection task fails, the business status can be obtained as a pending retry status, and the automatic retry collection task can be regenerated and written to the task queue.

[0107] In this embodiment of the application, if the first acquisition task is an automatic acquisition task, the business status of the business acquisition target can be queried in real time, and when it is detected that it is in a state of waiting for retry and there is no active task index, the first acquisition task is automatically generated. This triggers automatic supplementary acquisition for recoverable failure scenarios under the premise of ensuring no concurrent tasks. This avoids the waste of resources caused by repeated retries and ensures that the retry requirements to be processed can be converted into executable acquisition tasks in a timely and reliable manner, effectively improving the system's self-healing ability and acquisition integrity when facing temporary anomalies.

[0108] To facilitate understanding, the following will be combined with... Figure 5 To explain, Figure 5 This is a schematic diagram of an active task index-driven deduplication process provided in an embodiment of this application.

[0109] like Figure 5 As shown, when a new data collection task is enqueued, it is checked whether the active task index exists. If the active task index exists, it is checked whether the active task index points to a data collection task in an incomplete state. If so, the new data collection task is rejected from being written to the task queue to avoid concurrent duplicate data collection. If not, that is, the data collection task pointed to by the active task index does not exist or points to a data collection task in a completed or failed state, the new data collection task is written to the task queue, and the active task index is set as the task identifier of the new data collection task. If the active task index does not exist, the new data collection task is written to the task queue, and the active task index is set as the task identifier of the new data collection task.

[0110] The beneficial effects of the question-and-answer data collection method in this application embodiment may include: (1) Historical data collection tasks in the completed or failed state will no longer block legitimate data collection in the waiting-for-retry state.

[0111] (2) Active task indexing can prevent the same business objective from being collected repeatedly and concurrently.

[0112] (3) Normal data collection tasks, automatic data collection tasks and manual data collection tasks can be uniformly entered into the task queue.

[0113] (4) Even if the retry task fails to be enqueued after the pending retry status is written, the retry acquisition task can be regenerated and written to the task queue by restoring the pending retry status.

[0114] (5) Business keys facilitate business tracking, and task identifiers facilitate task tracking.

[0115] In related technologies, a single business key is typically used for deduplication to prevent duplicate execution, but this can block legitimate data acquisition. This application decouples the business acquisition target from the acquisition task. By using an active task index, it intercepts acquisition tasks that are about to be enqueued when it determines that there are still acquisition tasks in the execution chain, and allows new acquisition tasks to be regenerated based on the same business key in a retry state. This simultaneously solves the problems of duplicate acquisition and the blocking of legitimate data acquisition.

[0116] To implement the above embodiments, this application also proposes a question-and-answer collection device. Figure 6 This is a schematic diagram of the question-and-answer collection device provided in one embodiment of this application.

[0117] like Figure 6 As shown, the question-and-answer data collection device 600 includes: The first acquisition module 610 is used to acquire the queuing request of the first acquisition task of the business acquisition target; wherein, the business acquisition target refers to a single acquisition intention for the question and answer results of the same question on the target question and answer service platform; The second acquisition module 620 is used to acquire the active task index corresponding to the service acquisition target based on the service key of the service acquisition target; wherein, the active task index is used to record the task identifier of the acquisition task currently assigned to the service acquisition target; The control module 630 is used to control whether the first collection task is written into the task queue corresponding to the target question and answer service platform based on the active task index; wherein, the task queue is used to schedule the collection task of question and answer results of the target question and answer service platform.

[0118] Optionally, the control module 630 is used for: Based on the active task index, determine whether there are currently any active collection tasks for the service collection target; In response to determining that there is currently an active collection task for the service collection target, the first collection task is refused to be written into the task queue; In response to determining that there are currently no active collection tasks for the service collection target, the first collection task is written into the task queue, and the active task index is set as the task identifier of the first collection task.

[0119] Optionally, the control module 630 is used for: In response to the absence of the active task index, it is determined that there are currently no active collection tasks for the service collection target; In response to the existence of the active task index, determine whether the second acquisition task indicated by the active task index exists; In response to the absence of the second collection task indicated by the active task index, it is determined that there is currently no active collection task for the service collection target; In response to the existence of a second collection task indicated by the active task index, the system determines whether there is currently an active collection task for the service collection target based on the task status of the second collection task.

[0120] Optionally, the control module 630 is used for: In response to the task status indication that the second collection task is in an incomplete state, it is determined that there is currently an active collection task for the service collection target; In response to the task status indication that the second acquisition task is in a completed or failed state, it is determined that there is currently no active acquisition task for the service acquisition target.

[0121] Optionally, the device may further include: The third acquisition module is used to acquire the acquisition request of the business acquisition target; The fourth acquisition module is used to acquire, according to the acquisition request, the identifier of the batch to which the question belongs, the identifier of the question, the identifier of the target question-and-answer service platform, and the acquisition round of the question; The first generation module is used to generate the business key based on the batch identifier, the question identifier, the target question-and-answer service platform identifier, and the question collection round.

[0122] Optionally, wherein, in response to the first acquisition task being an automatic retry acquisition task, the device may further include: The first query module is used to query the business status of the business collection target in real time. The second generation module is used to generate the first collection task in response to the business status being in a retry state and the active task index not existing.

[0123] Optionally, the device may further include: The determination module is used to determine the execution type of the first acquisition task and the number of retry acquisitions to which the first acquisition task belongs; The third generation module is used to generate a task identifier for the first acquisition task based on the business key, the execution type, and the number of retry acquisitions.

[0124] Optionally, the device may further include: The second query module is used to query the active task index and the task status of the second acquisition task indicated by the active task index in real time. The cleanup module is used to clean up the active task index in response to the second acquisition task not existing, or the task status being completed or failed.

[0125] It should be noted that the explanation of the above-mentioned question-and-answer collection method embodiment also applies to the question-and-answer collection device of this embodiment, so it will not be repeated here.

[0126] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.

[0127] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0128] like Figure 7As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 702 or a computer program loaded from storage unit 708 into RAM (Random Access Memory) 703. RAM 703 can also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. I / O (Input / Output) interface 705 is also connected to bus 704.

[0129] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0130] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the question-and-answer acquisition method. For example, in some embodiments, the question-and-answer acquisition method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the question-and-answer acquisition method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform a question-and-answer collection method by any other suitable means (e.g., by means of firmware).

[0131] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0133] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0134] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0135] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0136] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0137] According to an embodiment of this application, this application also provides a computer program product that, when the instruction processor in the computer program product is executed, performs the question-and-answer collection method proposed in the above embodiments of this application.

[0138] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0139] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A question-and-answer data collection method, comprising: The first collection task of the business collection target is enqueued; wherein, the business collection target refers to a single collection intention for the question and answer results of the same question on the target question and answer service platform; Based on the service key of the service collection target, obtain the active task index corresponding to the service collection target; wherein, the active task index is used to record the task identifier of the collection task currently assigned to the service collection target; Based on the active task index, control whether the first collection task is written into the task queue corresponding to the target question-and-answer service platform; wherein, the task queue is used to schedule the collection task of question-and-answer results of the target question-and-answer service platform.

2. The method as described in claim 1, wherein, The step of controlling whether the first collection task is written into the task queue corresponding to the target question-and-answer service platform based on the active task index includes: Based on the active task index, determine whether there are currently any active collection tasks for the service collection target; In response to determining that there is currently an active collection task for the service collection target, the first collection task is refused to be written into the task queue; In response to determining that there are currently no active collection tasks for the service collection target, the first collection task is written into the task queue, and the active task index is set as the task identifier of the first collection task.

3. The method as described in claim 2, wherein, Determining whether there are currently active collection tasks for the service collection target based on the active task index includes: In response to the absence of the active task index, it is determined that there are currently no active collection tasks for the service collection target; In response to the existence of the active task index, determine whether the second acquisition task indicated by the active task index exists; In response to the absence of the second collection task indicated by the active task index, it is determined that there is currently no active collection task for the service collection target; In response to the existence of a second collection task indicated by the active task index, the system determines whether there is currently an active collection task for the service collection target based on the task status of the second collection task.

4. The method of claim 3, wherein, Determining whether there is an active collection task for the service collection target based on the task status of the second collection task includes: In response to the task status indication that the second collection task is in an incomplete state, it is determined that there is currently an active collection task for the service collection target; In response to the task status indication that the second acquisition task is in a completed or failed state, it is determined that there is currently no active acquisition task for the service acquisition target.

5. The method of claim 1, wherein, The business key is generated using the following method: Obtain the collection request from the target service collection point; Based on the collection request, obtain the batch identifier to which the question belongs, the question identifier, the identifier of the target question-and-answer service platform, and the collection round of the question; The business key is generated based on the batch identifier, the question identifier, the target question-and-answer service platform identifier, and the question collection round.

6. The method of claim 1, wherein, In response to the first acquisition task being an automatic retry acquisition task, the first acquisition task is generated using the following method: Real-time query of the service status of the service collection target; In response to the business status being in a pending retry state and the active task index not existing, the first collection task is generated.

7. The method of claim 6, further comprising: Determine the execution type of the first data acquisition task and the number of retry acquisition attempts for the first data acquisition task; The task identifier of the first acquisition task is generated based on the business key, the execution type, and the number of retry acquisition attempts.

8. The method of any one of claims 1-7, further comprising: Real-time query of the active task index and the task status of the second acquisition task indicated by the active task index; In response to the absence of the second acquisition task, or the task status being either completed or failed, the active task index is cleared.

9. A question-and-answer data collection device, comprising: The first acquisition module is used to acquire the queuing request of the first acquisition task of the business acquisition target; wherein, the business acquisition target refers to a single acquisition intention for the question and answer results of the same question on the target question and answer service platform; The second acquisition module is used to acquire the active task index corresponding to the service acquisition target based on the service key of the service acquisition target; wherein, the active task index is used to record the task identifier of the acquisition task currently assigned to the service acquisition target; The control module is used to control whether the first collection task is written into the task queue corresponding to the target question-and-answer service platform based on the active task index; wherein, the task queue is used to schedule the collection task of question-and-answer results of the target question-and-answer service platform.

10. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-8.

Images