Fault detection and repair method, apparatus, and electronic device

Abstract

The present disclosure provides a fault detection and repair method, device and electronic equipment, when an application process starts, each task record in a preset database is scanned, a first task record meeting a first fault condition is obtained, the first task record is repaired based on an intermediate product field corresponding to the first task record, and the repaired first task record is written into the preset database; when the application program runs, each task record in the preset database is scanned at a regular time, a second task record meeting a second fault condition is obtained, and the second task record is repaired based on an intermediate product field corresponding to the second task record, and the repaired second task record is written into the preset database. In this way, fault detection of the task record is performed in the application process startup and running stages, and the fault record is automatically recovered based on the intermediate product field already written in the preset database, thereby avoiding the phenomenon that the successfully completed stage product is discarded and the process is run again.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a fault detection and repair method, apparatus and electronic device. Background Technology

[0002] In multi-stage content generation scenarios, a single complete generation task is broken down into multiple sequentially executed stages. During task execution, dozens of external AI service calls are triggered, making it prone to various processing failures. Therefore, it is necessary to detect and repair these failures during task execution. Related technologies use periodic scanning or fixed-threshold stuck-inspection mechanisms to trigger resets and retries for nodes that have not been completed for an extended period. However, this fault detection relies solely on a single timed inspection mechanism, which cannot promptly detect faults within the inspection cycle. Furthermore, when the instance carrying the inspection task crashes, the inspection also stops, creating a blind spot where the failure of the inspector itself can lead to overall system failure. Summary of the Invention

[0003] In view of this, the purpose of this disclosure is to provide a fault detection and repair method, apparatus and electronic device, so as to detect faults in a timely manner and automatically repair them.

[0004] In a first aspect, embodiments of this disclosure provide a fault detection and repair method, the method comprising: when an application process starts, scanning each task record in a preset database to obtain a first task record that meets a first fault condition, repairing the first task record based on the intermediate product field corresponding to the first task record, and writing the repaired first task record into the preset database; wherein, each task record in the preset database corresponds to an independent scheduling unit in a preset pipeline; the intermediate product field is a stage output of the preset pipeline and a field written into the preset database; during the operation of the application, periodically scanning each task record in the preset database to obtain a second task record that meets a second fault condition, repairing the second task record based on the intermediate product field corresponding to the second task record, and writing the repaired second task record into the preset database.

[0005] Secondly, this disclosure also provides a fault detection and repair device, which includes: a startup recovery module, used to scan each task record in a preset database when the application process starts, obtain a first task record that meets a first fault condition, repair the first task record based on the intermediate product field corresponding to the first task record, and write the repaired first task record into the preset database; wherein, each task record in the preset database corresponds to an independent scheduling unit in a preset pipeline; the intermediate product field is a stage output of the preset pipeline and a field written into the preset database; and a timed inspection module, used to periodically scan each task record in the preset database during the application's operation, obtain a second task record that meets a second fault condition, repair the second task record based on the intermediate product field corresponding to the second task record, and write the repaired second task record into the preset database.

[0006] Thirdly, this disclosure provides an electronic device including a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the above-described fault detection and repair method.

[0007] Fourthly, this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when invoked and executed by a processor, cause the processor to implement the aforementioned fault detection and repair method.

[0008] The embodiments disclosed herein bring the following beneficial effects: This disclosure provides a fault detection and repair method, apparatus, and electronic device. When an application process starts, it scans each task record in a preset database to obtain a first task record that meets a first fault condition. Based on the intermediate product field corresponding to the first task record, the first task record is repaired, and the repaired first task record is written into the preset database. Each task record in the preset database corresponds to an independent scheduling unit in a preset pipeline. The intermediate product field is the stage output of the preset pipeline and is written into the preset database. During application execution, it periodically scans each task record in the preset database to obtain a second task record that meets a second fault condition. Based on the intermediate product field corresponding to the second task record, the second task record is repaired, and the repaired second task record is written into the preset database. This method performs task record fault detection during both application process startup and execution, and automatically recovers fault records based on the intermediate product fields already written into the preset database. This avoids discarding successfully completed stage products and rerunning the process, and also improves task record repair efficiency.

[0009] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objects and other advantages of this disclosure are realized and obtained through the structures particularly pointed out in the description, claims and drawings.

[0010] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0012] Figure 1 A flowchart of a fault detection and repair method provided in this embodiment of the present disclosure; Figure 2 This is an overall architecture diagram of fault detection and repair provided in an embodiment of the present disclosure; Figure 3 This is a schematic diagram of the structure of a fault detection and repair device provided in an embodiment of the present disclosure; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0014] Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely to illustrate selected embodiments of the disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0015] AI-driven automatic content generation (AIGC) technology, based on Large Language Model (LLM), Visual Language Model (VLM), and Diffusion Model, has been widely applied in multi-stage content generation scenarios such as "song to music video," "long text to storyboard," and "script to video." Typical characteristics of this type of system are: a single complete generation task is broken down into multiple sequentially executed stages (e.g., 4-8 stages such as timeline extraction, narrative planning, visual description, image generation, video generation, and video compositing); a single generation takes 5-30 minutes; dozens of external AI service calls are triggered during the process; and there are strict dependencies between the outputs of each stage. For example, a subsequent stage can only continue based on intermediate outputs (such as timestamps, prompts, and images) written to the database by the previous stage.

[0016] Due to their long pipelines, long processing times, intensive AI calls, and typical multi-instance horizontal scaling deployments, such systems frequently experience various anomalies during runtime, including forced process termination (e.g., memory overflow leading to system kill, rolling deployment causing restarts), external AI services timeouts, event loop blocking, multi-instance lock leaks, and interrupted asset file downloads. This places high demands on fault self-healing mechanisms. Related technologies provide two fault detection methods: Option 1: Process-level failure restart solution based on a container orchestration platform (such as Kubernetes). This solution relies on the container runtime to configure a liveness probe and restart strategy. When the probe detects that a process is unavailable, it forcibly kills and restarts the container. After restarting, the application itself is responsible for handling the residual state. However, this solution only performs coarse-grained recovery at the "process level." The probe is completely unaware of scenarios where "the process is still alive but internal tasks are stuck" (e.g., partial blocking of the event loop, or upstream AI services not returning for a long time), causing the task to hang indefinitely. After restarting, the "generating in progress" status field in the database (e.g., "video generating in progress") still has a residual value. The platform itself is unaware of the business state, requiring the application to implement state repair logic separately. After restarting, the task starts running again from the initial stage, and all successfully completed early stage AI calls (timeline, visual description, etc.) are discarded, wasting the AI ​​call cost of a single restart.

[0017] Option 2: A node-level retry scheme based on a general workflow orchestration engine (such as Airflow / Dagster / asynchronous task queue). This technical solution models the pipeline as a directed acyclic graph, configuring a maximum number of retries and a retry interval for each node (task); through periodic scanning or a fixed threshold-based deadlock inspection mechanism, nodes that have not completed for a long time are triggered to reset and retry. However, this fault detection method relies solely on a "single scheduled inspection" mechanism, which cannot detect faults within the inspection cycle in a timely manner. When the instance carrying the inspection task crashes, the inspection also stops, creating a blind spot where "the inspector's own failure leads to overall failure." Furthermore, this method typically uses a single time threshold (e.g., a fixed 60 minutes) to filter recoverable projects, failing to address both early failure scenarios like "failure on the first AI call after user submission" and regular failure scenarios like "unfinished concurrent tasks after a pipeline failure." The former results in excessively long waiting times for users, while lowering the threshold for the latter leads to data contention with ongoing concurrent tasks. Additionally, workflow engines typically only maintain a state machine view within the database and are unaware of the actual asset files produced in the local file system. When a download stream is forcibly interrupted, resulting in "the database recording completion but the local file not existing or having zero bytes," this state cannot be recognized, causing downstream consumers to directly report errors when reading empty files.

[0018] In view of the above problems, this disclosure provides a fault detection and repair method, apparatus and electronic device, which can be applied to fault detection and automatic fault repair scenarios.

[0019] To facilitate understanding of this disclosure, a fault detection and repair method provided by an embodiment of this disclosure will first be described in detail, such as... Figure 1 As shown, the method includes the following specific steps: Step S102: When the application process starts, scan each task record in the preset database, obtain the first task record that meets the first fault condition, repair the first task record based on the intermediate product field corresponding to the first task record, and write the repaired first task record into the preset database; wherein, each task record in the preset database corresponds to an independent scheduling unit in the preset pipeline; the intermediate product field is the stage output of the preset pipeline and the field written into the preset database.

[0020] In practical implementation, the aforementioned preset database includes multiple task projects, each task project includes at least one task record, and each task record corresponds to an independently callable unit in the preset pipeline. This independently callable unit is the smallest unit that can be independently invoked. A task project is a business entity containing at least one task record, and one task project corresponds to one complete end-to-end generation request. Task records carry multiple resource dimensions (including prompts, images, videos, etc.), and each resource dimension maintains its own independent state and lock state.

[0021] The aforementioned pre-defined pipeline includes a timeline stage, a director stage, a visual description stage, an image generation stage, and a video generation stage, executed sequentially. The intermediate output fields are fields produced at a specific stage of the pre-defined pipeline and written to a pre-defined database; the output of this stage can be consumed by the next stage. These stage outputs include, but are not limited to, timestamps, prompts, images, and videos.

[0022] When the application process starts, it scans each task record in the preset database, retrieves the first task record that meets the first fault condition, and uses the intermediate product fields corresponding to the first task record already written to the preset database to infer which stage the preset pipeline has completed. This allows it to precisely locate the recovery entry point to the next stage that is not yet completed, avoiding discarding and rerunning the results of successfully completed earlier stages. Then, it starts execution from the recovery entry point of the preset pipeline to repair the first task record and writes the repaired first task record to the preset database.

[0023] In an optional embodiment, the task record maintains at least three independent resource dimension status fields. Specifically, the task record includes a prompt word status field, an image status field, and a video status field. The prompt word status field indicates the generation and completion status of the prompt word, the image status field indicates the generation and completion status of the image, and the video status field indicates the generation and completion status of the video. Based on this, the aforementioned first fault condition includes: the prompt word status field is in the "generating" state but the prompt word generation time exceeds a first preset time; the image status field is in the "generating" state and the image generation time exceeds a second preset time; the video status field is in the "generating" state and the video generation time exceeds a third preset time; or the lock expiration time of the task record is less than the current time.

[0024] The specific durations corresponding to the first, second, and third preset durations can be determined according to the research and development needs. For example, the first, second, and third preset durations can all be 600 seconds; or the first preset duration can be 500 seconds, the second preset duration can be 600 seconds, and the third preset duration can be 650 seconds, etc.

[0025] Step S104: During the operation of the application, scan each task record in the preset database at regular intervals, obtain the second task record that meets the second fault condition, repair the second task record based on the intermediate product field corresponding to the second task record, and write the repaired second task record into the preset database.

[0026] During application execution, a periodic inspection task scans each task record in the preset database to obtain the second task record that meets the second fault condition. Based on the intermediate product fields corresponding to the second task record already written to the preset database, it infers which stage the preset pipeline has completed, thus accurately locating the recovery entry point to the next stage that is not yet completed, avoiding discarding and rerunning the results of successfully completed earlier stages. Then, execution begins from the recovery entry point of the preset pipeline to repair the second task record, and the repaired second task record is written to the preset database.

[0027] In an optional embodiment, the second fault condition includes: the prompt word status field is in a generating state but the prompt word generation time exceeds a first preset time; the image status field is in a generating state and the image generation time exceeds a second preset time; the video status field is in a generating state and the video generation time exceeds a third preset time; or the lock expiration time of the task record is less than the current time. The second fault condition also includes: the task record in the task project meets the first fault condition and the task project meets the preset recovery mechanism; wherein, the task project includes at least one task record. The preset recovery mechanism here refers to the differentiated processing mechanism between the early failure window and the regular recovery window, which will be described in detail in subsequent embodiments.

[0028] The fault detection and repair method provided in this embodiment performs task record fault detection during the application process startup and running phases, and automatically recovers fault records based on intermediate product fields already written in a preset database. This avoids the phenomenon of discarding successfully completed stage products and rerunning the process, and also improves the efficiency of task record repair.

[0029] The following examples are used to describe a preset rescue mechanism.

[0030] Specifically, the following method is used to determine whether a task item meets the preset recovery mechanism: obtain the target task item that meets the first fault condition and determine the survival time of the target task item; if the survival time of the target task item is greater than or equal to the first time threshold, determine that the target task item meets the preset recovery mechanism; if the target task item is less than the first time threshold but greater than or equal to the second time threshold, determine the number of task records contained in the target task item, and determine that the target task item meets the preset recovery mechanism when the number of task records meets the preset number requirement.

[0031] In practice, the second time threshold is less than the first time threshold. For example, the first time threshold can be set to 60 minutes or 50 minutes, and the second time threshold can be set to 5 minutes or 10 minutes, etc.

[0032] The target task items that meet the first failure condition are the task items corresponding to the task records that meet the first failure condition. After obtaining the target task items that meet the first failure condition, it is necessary to calculate the survival time of the target task items. For example, survival time = (current time - last update time of the project) ÷ 60 seconds.

[0033] If the survival time of the target task item is greater than or equal to the first time threshold, it is unconditionally included in the rescue process, meaning the target task item is determined to meet the preset rescue mechanism and needs to be automatically repaired. If the survival time of the target task item is less than the first time threshold but greater than or equal to the second time threshold, the number of task records contained in the target task item is determined. If the number of task records meets a preset requirement, the target task item is determined to meet the preset rescue mechanism. This preset requirement can be that the number of task records is greater than zero, or that the number of task records is a preset value, etc.

[0034] In practical implementation, if the number of task records is zero, it means that the preset pipeline has not yet produced any intermediate products, and there is no risk of data contention. In this case, the target task can be included in the recovery process. If the number of task records is not zero, skip this round and wait for the survival time of the target task to increase to the first time threshold. It should be noted that the preset quantity requirement is set to "zero number of task records" because this range belongs to the early failure window. Its typical scenario is that the first external AI call after the user submits the data fails, and the pipeline has not yet produced any intermediate data. At this time, there is no risk of competing for resources with concurrent tasks that are being executed, and aggressive recovery can be carried out to quickly respond to the user. However, when the number of task records is greater than zero, it means that the pipeline has started to produce intermediate data. If the recovery is carried out in advance, it may compete for data with concurrent tasks that are being retried. Therefore, it is necessary to wait until the survival time of the target task increases to the first time threshold before recovering the target task.

[0035] In one optional embodiment, if the survival time of the target task item is less than the second time threshold, the current round is skipped directly (to avoid accidentally damaging a normal task that has just been submitted).

[0036] This disclosure employs at least two time thresholds for recovering failed projects: a shorter second time threshold and a longer first time threshold. This method triggers a preset recovery mechanism through these two time thresholds, and this mechanism balances "rapid recovery of early failures (reducing the perceived waiting time for users)" and "safe recovery of regular failures (avoiding collisions with concurrent tasks)".

[0037] The following embodiments describe a method for obtaining and repairing the second task record.

[0038] Specifically, during application operation, the process of periodically scanning each task record in the preset database to obtain the second task record that meets the second fault condition can include: During application operation, executing a first inspection task once every first time period; wherein the first inspection task is used to: scan each task record in the preset database and obtain the second task record whose image status field is in the generating state and whose image generation time exceeds the second preset time, and the second task record whose video status field is in the generating state and whose video generation time exceeds the third preset time; executing a second inspection task once every second time period; wherein the second inspection task is used to: scan each task record in the preset database and obtain the second task record whose prompt word status field is in the generating state but whose prompt word generation time exceeds the first preset time; executing a third inspection task once every third time period; wherein the third inspection task is used to: obtain the target task record that meets the first fault condition and satisfies the preset recovery mechanism; and determine the task record contained in the target task record as the second task record; executing a lock expiration inspection task once every fourth time period; wherein the lock expiration inspection task is used to scan the second task record whose lock expiration time is less than the current time.

[0039] In practical implementation, during the application process's runtime, a preset database is periodically scanned according to multiple independent scheduled inspection tasks to obtain second task records that meet the second fault condition. These scheduled inspection tasks include a first inspection task, a second inspection task, a third inspection task, and a lock expiration inspection task. The inspection period for each scheduled inspection task can be set according to requirements; for example, the first inspection task might have a first time period of 5 minutes, the second inspection task a second time period of 5 minutes, the third inspection task a third time period of 30 minutes, and the lock expiration inspection task a fourth time period of 2 minutes, etc.

[0040] In practical applications, a first inspection task is executed every first time period to obtain a second task record where the image / video status field is "generating" and the generation time exceeds a preset time. This preset time can also be understood as the dead threshold corresponding to the image / video generation time. At this time, the status of the second task record is reset to the corresponding "image failed" or "video failed", and the lock field is cleared. A second inspection task is executed every second time period to obtain a second task record where the prompt word status field is "generating" and the prompt word generation time exceeds a first preset time. This first preset time is the dead threshold corresponding to the prompt word generation time. At this time, the prompt word status field of the second task record is reset to "prompt word failed", and the prompt word lock field is cleared.

[0041] Every third time period, a third inspection task is triggered to obtain target task records whose project status is "failed" (equivalent to meeting the first fault condition) and which meet the preset recovery mechanism. Every fourth time period, a lock expiration inspection task is triggered to obtain all task records whose "lock expiration time is less than the current time" and whose status is still "generating". For the matched records, the status is reset to the corresponding failure status and the lock field is cleared.

[0042] In practical applications, the prompt word dimension is inspected by a separate second inspection task. Upon a hit, only the prompt word status field and the prompt word lock field are reset, without affecting the image / video status field. The image / video dimension is inspected by a separate first inspection task. Upon a hit, only the top-level status field and the general lock field are reset, without affecting the prompt word status field. The state degradation of each dimension is decoupled, and the front end can refresh different UI areas of the prompt word button, image thumbnail, and video player separately by subscribing to state change events.

[0043] It's important to note that the state / lock / inspection of the three dimensions—prompt words, images, and videos—are completely separated because prompt word calls are typically based on lightweight text models (VLM / LLM), with a single call taking sub-seconds and should normally complete within a few seconds. Image and video generation, on the other hand, are based on diffusion or video generation models, with single calls taking minutes. If all three share the same set of state fields and the same threshold, the prompt word dimension is forced to use an excessively long dead threshold (causing dead prompt words to wait too long to recover); the image / video dimension is forced to use an excessively short dead threshold (causing normally running long tasks to be mistakenly judged as dead). Independent dimensions allow for individual tuning for each resource. The dead threshold here indicates the preset duration that the generation time of each dimension exceeds when the first failure condition is met.

[0044] In practical applications, each task record maintains at least three independent resource dimension status fields (typically prompt word status field, image status field, and video status field). Each dimension is independently configured with its own lock field (including lock holder identifier and lock expiration time), dead threshold, inspection trigger cycle (equivalent to the above time cycle), and degradation path. The status degradation of different dimensions is not coupled to each other, and the front end can refresh different UI areas based on the resource dimension identifier in the event payload.

[0045] In an optional embodiment, before the task corresponding to the task record is executed, a lock is requested from a preset database; after the task successfully acquires the lock, a heartbeat timer is started, and the lock expiration time is updated to the preset database once every heartbeat cycle; wherein, the lock expiration time update includes: extending the lock expiration time of the task record to the current lock expiration time.

[0046] In practical implementation, the tasks corresponding to the above task records can be prompt word generation tasks, image generation tasks, and video generation tasks, etc. Before execution, each task requests a lock from a preset database, and the holder periodically renews the lock: the lock field is only allowed to be updated to "current instance identifier + current time and lock expiration time (e.g., 5 minutes)" when the lock field is empty, the lock field is the identifier of the current instance, or the lock expiration time is less than the current time. This operation ensures that in a multi-instance environment, no two instances will simultaneously acquire the lock for the same task.

[0047] After successfully acquiring the lock, a heartbeat timer is started. Every heartbeat cycle (e.g., 60 seconds), an update is performed on the preset database: the lock expiration time field is extended to the current time lock expiration duration; the update is only allowed if the lock field is still the current instance identifier, ensuring that locks that have been preempted by other instances are not renewed incorrectly.

[0048] Furthermore, the specific process of repairing the second task record based on the intermediate product field corresponding to the second task record may include: obtaining the second task item corresponding to the second task record and determining the total number of task records contained in the second task item; if the total number of records is greater than a preset number threshold, counting the number of task records in the second task item that contain the intermediate product field; determining the repair stage in the preset pipeline based on the number of records and the total number of records, and starting the repair of the second task record from the repair stage.

[0049] In practice, the system retrieves the second task project containing the second task record, then counts all task records under the second task project to obtain the total number of task records contained in the second task project. If the total number of records is zero, it directly returns to the draft state and ends. If the total number of records is greater than a preset threshold (which can be zero or 1), the system counts the number of task records in the second task project that contain intermediate product fields. Based on the record count and the total number of records, it determines the repair stage in the preset pipeline, and begins repairing the second task record from this repair stage. Different record counts correspond to different repair stages.

[0050] Furthermore, the aforementioned intermediate product fields include a timestamp field, an image description field, and a visual description field. Based on this, the specific process for counting the number of task records containing intermediate product fields in the second task project can include: counting the first number of task records containing an image description field in the second task project; counting the second number of task records containing a visual description field in the second task project; and counting the third number of task records containing a timestamp field in the second task project. Wherein, a task record containing an image description field indicates that the task record has completed image generation; a task record containing a visual description field indicates that the task record has completed visual description generation; and a task record containing a timestamp field indicates that the task record has completed time extraction.

[0051] The specific process of determining the stage to be repaired in the preset pipeline based on the number of records and the total number of records may include: determining the stage to be repaired in the preset pipeline based on the preset judgment priority, the first number of records, the second number of records, the third number of records, and the total number of records.

[0052] The aforementioned preset decision priority can be: the decision priority of the image description field is higher than that of the visual description field, and the decision priority of the visual description field is higher than that of the timestamp field; the preset pipeline includes the timeline stage, director stage, visual description stage, image generation stage, and video generation stage executed in sequence.

[0053] Based on the above description, the specific process of determining the stages to be repaired in the preset pipeline based on preset judgment priorities, the number of first records, the number of second records, the number of third records, and the total number of records may include: determining whether the ratio of the number of first records to the total number of records is greater than a preset ratio threshold; if so, determining the video generation stage as the stage to be repaired; if not, determining whether the ratio of the number of second records to the total number of records is greater than the preset ratio threshold; if so, determining the visual description stage as the stage to be repaired; if not, determining whether the number of third records is equal to the total number of records; if so, determining the directing stage as the stage to be repaired; if not, determining the draft stage as the stage to be repaired. The preset ratio threshold here can be 0.5 or 0.6, etc.

[0054] In practical applications, the process first determines whether the ratio of the first record count to the total number of records is greater than a preset ratio threshold. If it is, the image generation stage is complete, and the video generation stage is designated as the stage to be repaired. If it is not, the process then determines whether the ratio of the second record count to the total number of records is greater than a preset ratio threshold. If it is, the visual description stage is complete, and the visual description stage is designated as the stage to be repaired. If it is not, the process then determines whether the third record count is equal to the total number of records. If the third record count is less than the total number of records but greater than zero, the timeline stage is partially complete, and the draft state is designated as the stage to be repaired. If the third record count is zero, all stages are incomplete, and the draft state is designated as the stage to be repaired.

[0055] In the specific implementation, the image / visual description stage uses a preset ratio threshold for judgment, while the timestamp stage uses the total number of records for judgment because: image generation and visual description are independent calls at the task level, and the failure of a small number of tasks does not affect the overall recognition of the stage as completed (the small number of failed tasks can be partially retried when resuming the run); while timestamp extraction is the overall output at the project level, and inconsistent timestamp fields mean that the pipeline cannot proceed normally based on inconsistent timelines, so all timestamps must be completed before the stage is considered complete.

[0056] Furthermore, some timelines are downgraded to the draft state instead of the director stage because: if forced into the director stage, the director stage will read some task records with missing timestamps, causing a second failure and falling into a failure restart loop; although downgrading to the draft stage sacrifices some completed timestamps, it ensures that the subsequent pipeline runs on the premise of data integrity.

[0057] Furthermore, this disclosure reverse-engineers the "completeness of intermediate product fields" rather than explicitly recording the "completed stage" field because: intermediate products of the pipeline are to be written to the database for consumption in the next stage. The reverse-engineering scheme does not require adding fields or writing explicit markers at each stage, and is naturally robust to boundary cases such as partial write failures or rollbacks that do not touch the marked fields.

[0058] For example, for a failed project P, assuming it has 30 task records (N = 30), where 28 task records have non-empty image description fields, all 30 task records have non-empty visual description fields, and all 30 task records have non-empty timestamp fields. According to this algorithm, 28 / 30 ≈ 0.933 > 0.5, hitting priority 1, returning "Video Generation Stage" as the stage to be repaired, and the algorithm no longer evaluates subsequent priorities. Subsequent pipeline restarts will directly start from the video generation stage, with the AI ​​call results from the first four stages (timeline, director, visual description, image generation) fully reused. Assuming these four stages take an average of 6 minutes and cost approximately $4.20, this recovery is equivalent to saving $4.20 in AI call costs and 6 minutes of user waiting time.

[0059] In the recovery process of failed projects in a multi-stage AI content generation pipeline, it does not rely on any explicit "completed stage" marker field. Instead, it statistically analyzes the presence ratio of at least three different intermediate product fields in all task records under the project and matches the judgment conditions in order from high priority to low priority (high priority uses "majority presence > preset proportion threshold" for judgment, low priority uses "all presence = 100%" for judgment, and defensive downgrade is adopted for partial presence cases). Once a certain priority is hit, the corresponding recovery target state is returned, thereby accurately locating the pipeline's resume entry point to the earliest uncompleted stage.

[0060] The following embodiments describe a method for obtaining and repairing the first task record.

[0061] Specifically, the process of scanning each task record in the preset database to obtain the first task record that meets the first fault condition may include: scanning the task records in the preset database that are in a target state; wherein, the target state includes the downloaded completed state and the generated completed state; determining whether the local file corresponding to the task record in the target state exists, and if it exists, determining whether the file size of the local file is greater than zero; if the local file does not exist or the file size is not greater than zero, downgrading the task record in the target state to the download failed state; determining the task record in the download failed state as the first task record that meets the first fault condition; and scanning each task record in the preset database to obtain the first task record that meets the first fault condition.

[0062] After the application process starts, a preset time delay (e.g., 5 seconds, to wait for dependent modules to fully initialize) is applied. The system then scans the preset database for task records in the target state. For each record, the "local asset path" field is read, and the existence of the local file pointed to by that path is verified, along with its file size (in bytes). If the file does not exist or its size is zero, the system further checks whether the "original URL" field returned by the upstream AI service is saved. If the original URL is saved, the status is downgraded to download failure (the download can be re-initiated based on this URL). If the original URL is not saved, the status is downgraded to complete failure (the upstream AI service needs to be called again to generate it). This method prevents subsequent steps from misjudging "completed" records (marked as complete in the database but with missing files) as truly completed, causing downstream consumption links to fail to read empty files.

[0063] Next, scan all first task records where the prompt word status field is in the "generating" state and the prompt word generation time exceeds the first preset time. Downgrade the prompt word status field of this first task record to "prompt word failed" and clear the prompt word lock field. Scan all first task records where the image status field is in the "generating" state and the image generation time exceeds the second preset time. Downgrade the image status field of this first task record to the corresponding "image failed" and clear the lock field. Scan all first task records where the video status field is in the "generating" state and the video generation time exceeds the third preset time. Downgrade the video status field of this first task record to the corresponding "video failed".

[0064] Furthermore, scan all project records for tasks with a "failed" status. For the matched tasks, calculate the target state to restore, update the project state, and then call the pipeline continuation entry point. Scan all project records for tasks with a "partially successful" status (i.e., the pipeline has reached the final stage but a small number of tasks have failed). For the matched tasks, only trigger partial retries for the failed tasks, without reverting the entire project state.

[0065] Furthermore, after repairing the first task record based on the intermediate product field corresponding to the first task record and writing the repaired first task record into the preset database, it is determined whether the task record in the download failure state has been successfully re-downloaded. If so, the task record in the download failure state is upgraded to the download completed state.

[0066] In practice, after all failed states are repaired, the task records in the "download failed" state are scanned again to verify whether the download has been successfully restarted. If so, the status is upgraded to "download completed". This step forms a closed loop with the steps for downgrading the status, ensuring that the status field is completely consistent with the file system after the recovery is completed.

[0067] Furthermore, the specific process of repairing the first task record based on the intermediate product field corresponding to the first task record may include: obtaining the first task item corresponding to the first task record and determining the total number of task records contained in the first task item; if the total number of records is greater than a preset number threshold, counting the number of task records in the first task item that contain the intermediate product field; determining the repair stage in the preset pipeline based on the number of records and the total number of records, and starting the repair of the first task record from the repair stage.

[0068] In a specific implementation, the intermediate product fields include a timestamp field, an image description field, and a visual description field. Based on this, the specific process of counting the number of task records containing intermediate product fields in the first task project can include: counting the first number of task records containing an image description field in the first task project; counting the second number of task records containing a visual description field in the first task project; and counting the third number of task records containing a timestamp field in the first task project. Wherein, a task record containing an image description field indicates that the task record has completed image generation; a task record containing a visual description field indicates that the task record has completed visual description generation; and a task record containing a timestamp field indicates that the task record has completed time extraction.

[0069] Furthermore, the specific process of determining the stage to be repaired in the preset pipeline based on the number of records and the total number of records may include: determining the stage to be repaired in the preset pipeline based on the preset judgment priority, the first number of records, the second number of records, the third number of records, and the total number of records.

[0070] In specific implementation, the aforementioned preset judgment priorities include: the judgment priority of the image description field is higher than that of the visual description field, and the judgment priority of the visual description field is higher than that of the timestamp field; the preset pipeline includes the timeline stage, director stage, visual description stage, image generation stage, and video generation stage executed in sequence; based on this, the specific process of determining the stage to be repaired in the preset pipeline may include: judging whether the ratio of the first number of records to the total number of records is greater than a preset ratio threshold; if so, the video generation stage is determined as the stage to be repaired; if not, judging whether the ratio of the second number of records to the total number of records is greater than a preset ratio threshold; if so, the visual description stage is determined as the stage to be repaired; if not, judging whether the third number of records is equal to the total number of records; if so, the director stage is determined as the stage to be repaired; if not, the draft stage is determined as the stage to be repaired.

[0071] Optionally, before obtaining the first task item corresponding to the first task record, a target task item that meets the first fault condition is obtained, and the survival time of the target task item is determined; if the survival time of the target task item is greater than or equal to the first time threshold, the target task item is determined to meet the preset recovery mechanism; if the target task item is less than the first time threshold but greater than or equal to the second time threshold, the number of task records contained in the target task item is determined, and when the number of task records meets the preset number requirement, the target task item is determined to meet the preset recovery mechanism; the target task item that meets the preset recovery mechanism is determined as the first task item corresponding to the first task record.

[0072] In an optional embodiment, each time a task-level state degradation or project-level state recovery is performed, a state change event is emitted to the system event bus. The event payload includes at least: a task record identifier, a resource dimension identifier (taken from the set {prompt word, image, video}), and a recovery source identifier (taken from the set {start recovery - prompt word, start recovery - image, start recovery - video, scheduled inspection - prompt word, scheduled inspection - image, scheduled inspection - video, manual forced reset}). Based on this, the front end can subscribe to the event to refresh the UI in real time; the operation and maintenance system can use the event stream to count the self-healing trigger frequency to assist in monitoring.

[0073] like Figure 2 The diagram shown is an overall architecture diagram of a fault detection and repair system provided in an embodiment of this disclosure. Figure 2Using project records and task records in a pre-defined database (each project contains multiple task records, and each task record corresponds to a smallest independently schedulable unit in the pipeline) as the core data carrier, the following five modules are constructed: a first-layer startup recovery module, a second-layer periodic inspection module, a third-layer lock lease heartbeat module, a status intelligent inference module, and an asset reconciliation module. The first-layer startup recovery module executes when the application process starts, scanning all task records in the "generating" state and project records in the "failed" state to obtain the first task record that meets the first fault condition, and then repairs the residual state of the first task record. The second-layer periodic inspection module scans stuck tasks, failed projects, and expired lock leases at multiple independent timed intervals during the application process's runtime to obtain the second task record that meets the second fault condition, and then repairs the second task record during runtime. The third-layer lock lease heartbeat module controls the holding of database locks with expiration times during task execution, and the holder periodically... For contract renewal, after the heartbeat stops, the second-level inspection takes over the handling; the status intelligent inference module is jointly called by the first and second levels, the input is the project record to be restored, and the output is the restoration target status (i.e., which stage the pipeline should continue running from) obtained by back-inferring the completeness of the "intermediate product field" in all task records under the project; the asset reconciliation module is executed as the first step in the restoration initiated by the first level, which verifies the actual existence and byte size of the local asset file corresponding to the task record in the "completed" state in the database, and selects different degradation paths for inconsistent records according to whether the original URL returned by the upstream AI service is saved.

[0074] The fault detection system disclosed herein consists of three independent, mutually protective detection layers with progressively larger time windows: the first layer scans the entire table at application startup (coarsest granularity, but most comprehensive coverage); the second layer scans "generating" status records at minute intervals during runtime (medium granularity); and the third layer is a database lease lock with an expiration time, renewed by the holder at second intervals (fineest granularity, but only covers lock-holding tasks). These three layers form mutually exclusive detection windows in the time dimension, meaning competitors using only one or two layers cannot cover all fault scenarios.

[0075] Meanwhile, in the recovery process, asset reconciliation is performed first, scanning the "completed" task records in the database to verify the existence and size of local asset files. When a file does not exist or has zero bytes, the status is downgraded to "download failed" (can be downloaded again based on the URL) or "completely failed" (requires re-generating from the upstream AI service) depending on whether the "original URL" field returned by the upstream AI service is retained in the task record. The reconciliation action strictly follows three major security constraints: "only downgrade, no upgrade, standard state machine verification, and reconciliation failure does not block the main process".

[0076] Corresponding to the above method embodiments, this disclosure also provides a fault detection and repair device, such as... Figure 3 As shown, the device includes: The startup recovery module 30 is used to scan each task record in the preset database when the application process starts, obtain the first task record that meets the first fault condition, repair the first task record based on the intermediate product field corresponding to the first task record, and write the repaired first task record into the preset database; wherein, each task record in the preset database corresponds to an independent scheduling unit in the preset pipeline; the intermediate product field is the stage output of the preset pipeline and the field written into the preset database.

[0077] The timed inspection module 31 is used to periodically scan each task record in the preset database during the application's operation, obtain the second task record that meets the second fault condition, repair the second task record based on the intermediate product field corresponding to the second task record, and write the repaired second task record into the preset database.

[0078] The aforementioned fault detection and repair device performs task record fault detection during both the application process startup and operation phases, and automatically recovers fault records based on intermediate product fields already written in the preset database. This avoids the phenomenon of discarding successfully completed stage products and rerunning the process, and also improves the efficiency of task record repair.

[0079] Furthermore, the aforementioned task record includes a prompt word status field, an image status field, and a video status field; based on this, the aforementioned first fault condition and second fault condition include: the prompt word status field is in the generating state but the prompt word generation time exceeds the first preset time, the image status field is in the generating state and the image generation time exceeds the second preset time, the video status field is in the generating state and the video generation time exceeds the third preset time, or the lock expiration time of the task record is less than the current time.

[0080] Furthermore, the second failure condition mentioned above also includes: the task record in the task project meets the first failure condition and the task project meets the preset recovery mechanism; wherein, the task project includes at least one task record.

[0081] Furthermore, the following method is used to determine whether a task item meets the preset recovery mechanism: obtain the target task item that meets the first fault condition and determine the survival time of the target task item; if the survival time of the target task item is greater than or equal to the first time threshold, determine that the target task item meets the preset recovery mechanism; if the target task item is less than the first time threshold but greater than or equal to the second time threshold, determine the number of task records contained in the target task item, and when the number of task records meets the preset number requirement, determine that the target task item meets the preset recovery mechanism.

[0082] Furthermore, the aforementioned timed inspection module 31 is used to: execute a first inspection task once every first time period during the application's operation; wherein, the first inspection task is used to: scan each task record in the preset database and obtain a second task record whose image status field is in the generating state and whose image generation time exceeds a second preset time, and a second task record whose video status field is in the generating state and whose video generation time exceeds a third preset time; execute a second inspection task once every second time period; wherein, the second inspection task is used to: scan each task record in the preset database and obtain a second task record whose prompt word status field is in the generating state but whose prompt word generation time exceeds a first preset time; execute a third inspection task once every third time period; wherein, the third inspection task is used to: obtain a target task record that meets the first fault condition and the preset recovery mechanism; determine the task record contained in the target task record as the second task record; execute a lock expiration inspection task once every fourth time period; wherein, the lock expiration inspection task is used to scan second task records whose lock expiration time is less than the current time.

[0083] Furthermore, the above-mentioned device also includes a lock update module, used to: request a lock from a preset database before the task corresponding to the task record is executed; after the task successfully acquires the lock, start a heartbeat timer and perform a lock expiration time update to the preset database once every heartbeat cycle; wherein, the lock expiration time update includes: extending the lock expiration time of the task record to the current lock expiration time.

[0084] Furthermore, the aforementioned startup recovery module 30 is used to: obtain the first task item corresponding to the first task record, and determine the total number of task records contained in the first task item; if the total number of records is greater than a preset number threshold, count the number of task records in the first task item that contain intermediate product fields; determine the repair stage in the preset pipeline based on the number of records and the total number of records, and start repairing the first task record from the repair stage.

[0085] Furthermore, the aforementioned intermediate product fields include a timestamp field, an image description field, and a visual description field; based on this, the aforementioned startup recovery module 30 is also used to: count the number of first records of task records containing an image description field in the first task project; count the number of second records of task records containing a visual description field in the first task project; count the number of third records of task records containing a timestamp field in the first task project; wherein, a task record containing an image description field indicates that the task record has completed image generation; a task record containing a visual description field indicates that the task record has completed visual description generation; and a task record containing a timestamp field indicates that the task record has completed time extraction.

[0086] Furthermore, the aforementioned startup recovery module 30 is also used to: determine the repair stage in the preset pipeline based on the preset judgment priority, the number of first records, the number of second records, the number of third records, and the total number of records.

[0087] Furthermore, the aforementioned preset judgment priorities include: the judgment priority of the image description field is higher than that of the visual description field, and the judgment priority of the visual description field is higher than that of the timestamp field; the preset pipeline includes the timeline stage, director stage, visual description stage, image generation stage, and video generation stage executed in sequence; based on this, the aforementioned startup recovery module 30 is also used to: determine whether the ratio of the first number of records to the total number of records is greater than a preset ratio threshold; if so, determine the video generation stage as the stage to be repaired; if not, determine whether the ratio of the second number of records to the total number of records is greater than a preset ratio threshold; if so, determine the visual description stage as the stage to be repaired; if not, determine whether the third number of records is equal to the total number of records; if so, determine the director stage as the stage to be repaired; if not, determine the draft stage as the stage to be repaired.

[0088] Furthermore, the aforementioned device also includes a project determination module, configured to: before acquiring the first task project corresponding to the first task record, acquire a target task project that meets the first fault condition and determine the survival time of the target task project; if the survival time of the target task project is greater than or equal to a first time threshold, determine that the target task project meets a preset recovery mechanism; if the target task project is less than the first time threshold but greater than or equal to a second time threshold, determine the number of task records contained in the target task project, and when the number of task records meets a preset quantity requirement, determine that the target task project meets the preset recovery mechanism; and determine the target task project that meets the preset recovery mechanism as the first task project corresponding to the first task record.

[0089] Furthermore, the aforementioned startup recovery module 30 is also used to: scan task records in a preset database that are in a target state; wherein, the target state includes a downloaded completed state and a generated completed state; determine whether the local file corresponding to the task record in the target state exists, and if it exists, determine whether the file size of the local file is greater than zero; if the local file does not exist or the file size is not greater than zero, downgrade the task record in the target state to a download failed state; and determine the task record in the download failed state as the first task record that meets the first fault condition.

[0090] Furthermore, the above-mentioned device also includes a status adjustment module, used to: after repairing the first task record based on the intermediate product field corresponding to the first task record and writing the repaired first task record into a preset database, determine whether the task record in the download failure state has been successfully re-downloaded; if so, upgrade the task record in the download failure state to the download completed state.

[0091] The fault detection and repair device provided in this disclosure has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0092] This embodiment also provides an electronic device, such as... Figure 4 As shown, the electronic device includes a processor and a memory. The memory stores computer-executable instructions that can be executed by the processor. The processor executes the computer-executable instructions to implement the aforementioned fault detection and repair method. This electronic device can be a server or a terminal device.

[0093] Specifically, the above-mentioned fault detection and repair method includes: when the application process starts, scanning each task record in the preset database, obtaining the first task record that meets the first fault condition, repairing the first task record based on the intermediate product field corresponding to the first task record, and writing the repaired first task record into the preset database; wherein, each task record in the preset database corresponds to an independent scheduling unit in the preset pipeline; the intermediate product field is the stage output of the preset pipeline and the field written into the preset database; during the application's operation, periodically scanning each task record in the preset database, obtaining the second task record that meets the second fault condition, repairing the second task record based on the intermediate product field corresponding to the second task record, and writing the repaired second task record into the preset database.

[0094] The above-mentioned fault detection and repair method performs task record fault detection during the application process startup and operation phases, and automatically recovers fault records based on intermediate product fields already written in the preset database. This avoids the phenomenon of discarding successfully completed stage products and rerunning the process, and also improves the efficiency of task record repair.

[0095] In an optional embodiment, the task record includes a prompt word status field, an image status field, and a video status field; based on this, the first fault condition and the second fault condition include: the prompt word status field is in the generating state but the prompt word generation time exceeds a first preset time, the image status field is in the generating state and the image generation time exceeds a second preset time, the video status field is in the generating state and the video generation time exceeds a third preset time, or the lock expiration time of the task record is less than the current time.

[0096] In an optional embodiment, the second fault condition further includes: the task record in the task project meets the first fault condition and the task project meets the preset recovery mechanism; wherein, the task project includes at least one task record.

[0097] In an optional embodiment, the following method is used to determine whether a task item meets the preset recovery mechanism: obtain the target task item that meets the first fault condition and determine the survival time of the target task item; if the survival time of the target task item is greater than or equal to the first time threshold, determine that the target task item meets the preset recovery mechanism; if the target task item is less than the first time threshold and greater than or equal to the second time threshold, determine the number of task records contained in the target task item, and when the number of task records meets the preset number requirement, determine that the target task item meets the preset recovery mechanism.

[0098] In an optional embodiment, the step of periodically scanning each task record in a preset database to obtain a second task record that meets the second fault condition during application operation includes: executing a first inspection task once every first time period during application operation; wherein the first inspection task is used to: scan each task record in the preset database and obtain a second task record whose image status field is in the generating state and whose image generation time exceeds a second preset time, and a second task record whose video status field is in the generating state and whose video generation time exceeds a third preset time; executing a second inspection task once every second time period; wherein the second inspection task is used to: scan each task record in the preset database and obtain a second task record whose prompt word status field is in the generating state but whose prompt word generation time exceeds a first preset time; executing a third inspection task once every third time period; wherein the third inspection task is used to: obtain a target task record that meets the first fault condition and satisfies the preset recovery mechanism; determine the task record contained in the target task record as the second task record; executing a lock expiration inspection task once every fourth time period; wherein the lock expiration inspection task is used to scan second task records whose lock expiration time is less than the current time.

[0099] In an optional embodiment, the method further includes: requesting a lock from a preset database before the task corresponding to the task record is executed; starting a heartbeat timer after the task successfully acquires the lock, and updating the lock expiration time to the preset database once every heartbeat cycle; wherein, updating the lock expiration time includes: extending the lock expiration time of the task record to the current lock expiration duration.

[0100] In an optional embodiment, the step of repairing the first task record based on the intermediate product field corresponding to the first task record includes: obtaining the first task item corresponding to the first task record and determining the total number of task records contained in the first task item; if the total number of records is greater than a preset number threshold, counting the number of task records containing the intermediate product field in the first task item; determining the repair stage in the preset pipeline according to the number of records and the total number of records, and starting the repair of the first task record from the repair stage.

[0101] In an optional embodiment, the intermediate product fields include a timestamp field, an image description field, and a visual description field. Based on this, the step of counting the number of task records containing intermediate product fields in the first task project includes: counting the first number of task records containing an image description field in the first task project; counting the second number of task records containing a visual description field in the first task project; and counting the third number of task records containing a timestamp field in the first task project. Wherein, a task record containing an image description field indicates that the task record has completed image generation; a task record containing a visual description field indicates that the task record has completed visual description generation; and a task record containing a timestamp field indicates that the task record has completed time extraction.

[0102] In an optional embodiment, the step of determining the stage to be repaired in the preset pipeline based on the number of records and the total number of records includes: determining the stage to be repaired in the preset pipeline based on the preset judgment priority, the first number of records, the second number of records, the third number of records, and the total number of records.

[0103] In an optional embodiment, the aforementioned preset judgment priority includes: the judgment priority of the image description field is higher than that of the visual description field, and the judgment priority of the visual description field is higher than that of the timestamp field; the preset pipeline includes a timeline stage, a director stage, a visual description stage, an image generation stage, and a video generation stage executed in sequence; based on this, the step of determining the stage to be repaired in the preset pipeline based on the preset judgment priority, the first number of records, the second number of records, the third number of records, and the total number of records includes: determining whether the ratio of the first number of records to the total number of records is greater than a preset ratio threshold; if so, determining the video generation stage as the stage to be repaired; if not, determining whether the ratio of the second number of records to the total number of records is greater than a preset ratio threshold; if so, determining the visual description stage as the stage to be repaired; if not, determining whether the third number of records is equal to the total number of records; if so, determining the director stage as the stage to be repaired; if not, determining the draft stage as the stage to be repaired.

[0104] In an optional embodiment, before obtaining the first task item corresponding to the first task record, the method further includes: obtaining a target task item that meets the first fault condition and determining the survival time of the target task item; if the survival time of the target task item is greater than or equal to a first time threshold, determining that the target task item meets a preset recovery mechanism; if the target task item is less than the first time threshold but greater than or equal to a second time threshold, determining the number of task records contained in the target task item, and when the number of task records meets a preset number requirement, determining that the target task item meets the preset recovery mechanism; and determining the target task item that meets the preset recovery mechanism as the first task item corresponding to the first task record.

[0105] In an optional embodiment, the step of scanning each task record in a preset database to obtain a first task record that meets the first fault condition includes: scanning task records in the preset database that are in a target state; wherein, the target state includes a downloaded completed state and a generated completed state; determining whether the local file corresponding to the task record in the target state exists, and if it exists, determining whether the file size of the local file is greater than zero; if the local file does not exist or the file size is not greater than zero, downgrading the task record in the target state to a download failed state; and determining the task record in the download failed state as a first task record that meets the first fault condition.

[0106] In an optional embodiment, after repairing the first task record based on the intermediate product field corresponding to the first task record and writing the repaired first task record into a preset database, the above method further includes: determining whether the task record in the download failure state has been successfully re-downloaded; if so, upgrading the task record in the download failure state to the download completed state.

[0107] Furthermore, Figure 4 The electronic device shown also includes a bus 102 and a communication interface 103, with the processor 101, the communication interface 103 and the memory 100 connected via the bus 102.

[0108] The memory 100 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 103 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 102 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0109] Processor 101 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 101 or by instructions in software form. The processor 101 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a readily available storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 100, and processor 101 reads information from memory 100 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0110] This disclosure also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are called and executed by a processor, they cause the processor to implement the above-described fault detection and repair method. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0111] Specifically, the above-mentioned fault detection and repair method includes: when the application process starts, scanning each task record in the preset database, obtaining the first task record that meets the first fault condition, repairing the first task record based on the intermediate product field corresponding to the first task record, and writing the repaired first task record into the preset database; wherein, each task record in the preset database corresponds to an independent scheduling unit in the preset pipeline; the intermediate product field is the stage output of the preset pipeline and the field written into the preset database; during the application's operation, periodically scanning each task record in the preset database, obtaining the second task record that meets the second fault condition, repairing the second task record based on the intermediate product field corresponding to the second task record, and writing the repaired second task record into the preset database.

[0112] The above-mentioned fault detection and repair method performs task record fault detection during the application process startup and operation phases, and automatically recovers fault records based on intermediate product fields already written in the preset database. This avoids the phenomenon of discarding successfully completed stage products and rerunning the process, and also improves the efficiency of task record repair.

[0113] In an optional embodiment, the task record includes a prompt word status field, an image status field, and a video status field; based on this, the first fault condition and the second fault condition include: the prompt word status field is in the generating state but the prompt word generation time exceeds a first preset time, the image status field is in the generating state and the image generation time exceeds a second preset time, the video status field is in the generating state and the video generation time exceeds a third preset time, or the lock expiration time of the task record is less than the current time.

[0114] In an optional embodiment, the second fault condition further includes: the task record in the task project meets the first fault condition and the task project meets the preset recovery mechanism; wherein, the task project includes at least one task record.

[0115] In an optional embodiment, the following method is used to determine whether a task item meets the preset recovery mechanism: obtain the target task item that meets the first fault condition and determine the survival time of the target task item; if the survival time of the target task item is greater than or equal to the first time threshold, determine that the target task item meets the preset recovery mechanism; if the target task item is less than the first time threshold and greater than or equal to the second time threshold, determine the number of task records contained in the target task item, and when the number of task records meets the preset number requirement, determine that the target task item meets the preset recovery mechanism.

[0116] In an optional embodiment, the step of periodically scanning each task record in a preset database to obtain a second task record that meets the second fault condition during application operation includes: executing a first inspection task once every first time period during application operation; wherein the first inspection task is used to: scan each task record in the preset database and obtain a second task record whose image status field is in the generating state and whose image generation time exceeds a second preset time, and a second task record whose video status field is in the generating state and whose video generation time exceeds a third preset time; executing a second inspection task once every second time period; wherein the second inspection task is used to: scan each task record in the preset database and obtain a second task record whose prompt word status field is in the generating state but whose prompt word generation time exceeds a first preset time; executing a third inspection task once every third time period; wherein the third inspection task is used to: obtain a target task record that meets the first fault condition and satisfies the preset recovery mechanism; determine the task record contained in the target task record as the second task record; executing a lock expiration inspection task once every fourth time period; wherein the lock expiration inspection task is used to scan second task records whose lock expiration time is less than the current time.

[0117] In an optional embodiment, the method further includes: requesting a lock from a preset database before the task corresponding to the task record is executed; starting a heartbeat timer after the task successfully acquires the lock, and updating the lock expiration time to the preset database once every heartbeat cycle; wherein, updating the lock expiration time includes: extending the lock expiration time of the task record to the current lock expiration duration.

[0118] In an optional embodiment, the step of repairing the first task record based on the intermediate product field corresponding to the first task record includes: obtaining the first task item corresponding to the first task record and determining the total number of task records contained in the first task item; if the total number of records is greater than a preset number threshold, counting the number of task records containing the intermediate product field in the first task item; determining the repair stage in the preset pipeline according to the number of records and the total number of records, and starting the repair of the first task record from the repair stage.

[0119] In an optional embodiment, the intermediate product fields include a timestamp field, an image description field, and a visual description field. Based on this, the step of counting the number of task records containing intermediate product fields in the first task project includes: counting the first number of task records containing an image description field in the first task project; counting the second number of task records containing a visual description field in the first task project; and counting the third number of task records containing a timestamp field in the first task project. Wherein, a task record containing an image description field indicates that the task record has completed image generation; a task record containing a visual description field indicates that the task record has completed visual description generation; and a task record containing a timestamp field indicates that the task record has completed time extraction.

[0120] In an optional embodiment, the step of determining the stage to be repaired in the preset pipeline based on the number of records and the total number of records includes: determining the stage to be repaired in the preset pipeline based on the preset judgment priority, the first number of records, the second number of records, the third number of records, and the total number of records.

[0121] In an optional embodiment, the aforementioned preset judgment priority includes: the judgment priority of the image description field is higher than that of the visual description field, and the judgment priority of the visual description field is higher than that of the timestamp field; the preset pipeline includes a timeline stage, a director stage, a visual description stage, an image generation stage, and a video generation stage executed in sequence; based on this, the step of determining the stage to be repaired in the preset pipeline based on the preset judgment priority, the first number of records, the second number of records, the third number of records, and the total number of records includes: determining whether the ratio of the first number of records to the total number of records is greater than a preset ratio threshold; if so, determining the video generation stage as the stage to be repaired; if not, determining whether the ratio of the second number of records to the total number of records is greater than a preset ratio threshold; if so, determining the visual description stage as the stage to be repaired; if not, determining whether the third number of records is equal to the total number of records; if so, determining the director stage as the stage to be repaired; if not, determining the draft stage as the stage to be repaired.

[0122] In an optional embodiment, before obtaining the first task item corresponding to the first task record, the method further includes: obtaining a target task item that meets the first fault condition and determining the survival time of the target task item; if the survival time of the target task item is greater than or equal to a first time threshold, determining that the target task item meets a preset recovery mechanism; if the target task item is less than the first time threshold but greater than or equal to a second time threshold, determining the number of task records contained in the target task item, and when the number of task records meets a preset number requirement, determining that the target task item meets the preset recovery mechanism; and determining the target task item that meets the preset recovery mechanism as the first task item corresponding to the first task record.

[0123] In an optional embodiment, the step of scanning each task record in a preset database to obtain a first task record that meets the first fault condition includes: scanning task records in the preset database that are in a target state; wherein, the target state includes a downloaded completed state and a generated completed state; determining whether the local file corresponding to the task record in the target state exists, and if it exists, determining whether the file size of the local file is greater than zero; if the local file does not exist or the file size is not greater than zero, downgrading the task record in the target state to a download failed state; and determining the task record in the download failed state as a first task record that meets the first fault condition.

[0124] In an optional embodiment, after repairing the first task record based on the intermediate product field corresponding to the first task record and writing the repaired first task record into a preset database, the above method further includes: determining whether the task record in the download failure state has been successfully re-downloaded; if so, upgrading the task record in the download failure state to the download completed state.

[0125] The aforementioned functions, when implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of this disclosure, essentially, or the parts that contribute to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal device, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0126] In the description of this disclosure, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this disclosure. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0127] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A fault detection and repair method, characterized in that, The method includes: When the application process starts, it scans each task record in the preset database, obtains the first task record that meets the first fault condition, and repairs the first task record based on the intermediate product field corresponding to the first task record, and writes the repaired first task record into the preset database; wherein, each task record in the preset database corresponds to an independent scheduling unit in the preset pipeline; the intermediate product field is the stage output of the preset pipeline and the field written into the preset database; During application operation, the system periodically scans each task record in the preset database to obtain the second task record that meets the second fault condition, repairs the second task record based on the intermediate product field corresponding to the second task record, and writes the repaired second task record into the preset database.

2. The method according to claim 1, characterized in that, The task record includes a prompt word status field, an image status field, and a video status field. The first fault condition and the second fault condition include: the prompt word status field is in the generating state but the prompt word generation time exceeds the first preset time; the image status field is in the generating state and the image generation time exceeds the second preset time; the video status field is in the generating state and the video generation time exceeds the third preset time; or the lock expiration time of the task record is less than the current time.

3. The method according to claim 2, characterized in that, The second fault condition further includes: the task record in the task project satisfies the first fault condition and the task project satisfies the preset recovery mechanism; wherein, the task project includes at least one task record.

4. The method according to claim 3, characterized in that, The following methods are used to determine whether a task item meets the preset rescue mechanism: Obtain the target task item that meets the first fault condition, and determine the survival time of the target task item; If the survival time of the target task item is greater than or equal to the first time threshold, it is determined that the target task item meets the preset rescue mechanism. If the target task item is less than the first time threshold and greater than or equal to the second time threshold, the number of task records contained in the target task item is determined. When the number of task records meets the preset quantity requirement, the target task item is determined to meet the preset rescue mechanism.

5. The method according to claim 3, characterized in that, The step of periodically scanning each task record in the preset database during application operation to obtain a second task record that meets the second fault condition includes: During the application's operation, a first inspection task is executed once every first time period; wherein, the first inspection task is used to: scan each task record in the preset database, and obtain a second task record where the image status field is in the generating state and the image generation time exceeds a second preset time, and a second task record where the video status field is in the generating state and the video generation time exceeds a third preset time. The second inspection task is executed once every second time period; wherein, the second inspection task is used to: scan each task record in the preset database and obtain the second task record where the prompt word status field is in the generating state but the prompt word generation time exceeds the first preset time. A third inspection task is performed every third time period; wherein, the third inspection task is used to: acquire the target task record that meets the first fault condition and satisfies the preset recovery mechanism; and determine the task record contained in the target task record as the second task record; A lock expiration inspection task is executed every fourth time period; wherein, the lock expiration inspection task is used to scan the second task record where the lock expiration time is less than the current time.

6. The method according to claim 5, characterized in that, The method further includes: Before the task corresponding to the task record is executed, a lock is requested from the preset database; After the task successfully acquires the lock, a heartbeat timer is started, and the lock expiration time is updated in the preset database every heartbeat cycle; wherein, the lock expiration time update includes: extending the lock expiration time recorded by the task to the current lock expiration time.

7. The method according to claim 1, characterized in that, The step of repairing the first task record based on the intermediate product field corresponding to the first task record includes: Obtain the first task item corresponding to the first task record, and determine the total number of task records contained in the first task item; If the total number of records is greater than a preset threshold, count the number of task records in the first task project that contain the intermediate product field; The repair stage in the preset pipeline is determined based on the number of records and the total number of records, and the repair of the first task record begins from the repair stage.

8. The method according to claim 7, characterized in that, The intermediate product fields include a timestamp field, an image description field, and a visual description field; The step of counting the number of task records in the first task project that contain the intermediate product field includes: Count the number of first records in the first task item that contain the image description field; count the number of second records in the first task item that contain the visual description field; count the number of third records in the first task item that contain the timestamp field. The task record includes the image description field to indicate that the task record has completed image generation; the task record includes the visual description field to indicate that the task record has completed visual description generation; and the task record includes the timestamp field to indicate that the task record has completed time extraction.

9. The method according to claim 8, characterized in that, The step of determining the stage to be repaired in the preset pipeline based on the number of records and the total number of records includes: Based on the preset judgment priority, the number of the first record, the number of the second record, the number of the third record, and the total number of records, the repair stage in the preset pipeline is determined.

10. The method according to claim 9, characterized in that, The preset decision priority includes: the decision priority of the image description field is higher than that of the visual description field, and the decision priority of the visual description field is higher than that of the timestamp field; the preset pipeline includes a timeline stage, a director stage, a visual description stage, an image generation stage, and a video generation stage executed in sequence; The step of determining the repair stage in the preset pipeline based on a preset judgment priority, the number of the first record, the number of the second record, the number of the third record, and the total number of records includes: Determine whether the ratio of the first number of records to the total number of records is greater than a preset ratio threshold. If so, determine the video generation stage as the stage to be repaired. If it is not greater than, determine whether the ratio of the second number of records to the total number of records is greater than a preset ratio threshold. If it is, determine the visual description stage as the stage to be repaired. If it is not greater than, determine whether the number of the third record is equal to the total number of records. If it is, determine the director stage as the stage to be repaired. If not equal, the draft stage is designated as the stage to be repaired.

11. The method according to claim 7, characterized in that, Before the step of obtaining the first task item corresponding to the first task record, the method further includes: Obtain the target task item that meets the first fault condition, and determine the survival time of the target task item; If the survival time of the target task item is greater than or equal to the first time threshold, it is determined that the target task item meets the preset rescue mechanism. If the target task item is less than the first time threshold and greater than or equal to the second time threshold, determine the number of task records contained in the target task item. When the number of task records meets the preset number requirement, determine that the target task item meets the preset rescue mechanism. The target task item that satisfies the preset rescue mechanism is determined as the first task item corresponding to the first task record.

12. The method according to claim 1, characterized in that, The step of scanning each task record in the preset database to obtain the first task record that meets the first fault condition includes: Scan the task records in the preset database that are in the target state; wherein, the target state includes the downloaded completed state and the generated completed state; Determine if the local file corresponding to the task record in the target state exists; if it exists, determine if the file size of the local file is greater than zero. If the local file does not exist or its size is not greater than zero, the task record in the target state will be downgraded to a download failure state; The task record that is in the download failure state is identified as the first task record that meets the first fault condition.

13. The method according to claim 12, characterized in that, After the step of repairing the first task record based on the intermediate product field corresponding to the first task record and writing the repaired first task record into the preset database, the method further includes: Determine whether the task record in the download failure state has been successfully re-downloaded. If so, upgrade the task record in the download failure state to the download completed state.

14. A fault detection and repair device, characterized in that, The device includes: The startup recovery module is used to scan each task record in the preset database when the application process starts, obtain the first task record that meets the first fault condition, repair the first task record based on the intermediate product field corresponding to the first task record, and write the repaired first task record into the preset database; wherein, each task record in the preset database corresponds to an independent scheduling unit in the preset pipeline; the intermediate product field is the stage output of the preset pipeline and the field written into the preset database; The timed inspection module is used to periodically scan each task record in the preset database during the application's operation, obtain the second task record that meets the second fault condition, repair the second task record based on the intermediate product field corresponding to the second task record, and write the repaired second task record into the preset database.

15. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the fault detection and repair method according to any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the fault detection and repair method according to any one of claims 1-13.

Images