Data collaborative processing method and system of heterogeneous system under dual-rail architecture

Abstract

This application discloses a data collaborative processing method and system for heterogeneous systems under a dual-track architecture, designed to solve the data silo problem between AI analysis platforms and office automation systems. This application constructs a dual-track parallel architecture including an AI track and an OA track, defining standardized integration interfaces through an API gateway middleware layer; adopting an event-driven publish-subscribe model, with data flow via an event bus; maintaining a state mapping table based on version numbers and business priorities to achieve conflict detection and intelligent resolution; performing incremental synchronization based on a synchronous cursor and a hybrid triggering mechanism; and initiating offline compensation based on a backoff strategy when the target system is unavailable. This application establishes a closed data loop from analysis and decision-making to execution, significantly improving the reliability, real-time performance, and consistency of data collaboration between heterogeneous systems.

Description

Technical Field

[0001] This application relates to the field of data processing and system integration technology, and more specifically, to a data collaborative processing method, system, device, and storage medium for a dual-track parallel architecture between an AI workflow platform and an office automation system. Background Technology

[0002] During enterprise digital transformation, AI workflow platforms (such as Coze and Dify) and office automation systems (such as Lark and DingTalk) operate independently, creating significant data silos. AI platforms excel at handling complex business scenarios requiring multiple steps and logical judgments (such as market research, technology research, and intellectual property analysis), outputting in-depth analysis reports; while OA systems handle daily communication, meetings, tasks, and project management. The lack of an effective data flow mechanism between the two prevents analysis results from being automatically translated into actionable steps.

[0003] However, existing system integration solutions face numerous challenges. First, the data models, interface protocols, and state definitions of AI platforms and OA systems are often inconsistent, requiring the development of separate adaptation code for each integration scenario, resulting in high costs and maintenance difficulties. Second, in a distributed environment, the independent operation of the dual-track systems makes them prone to data state conflicts due to concurrent modifications or network issues, lacking effective detection and intelligent resolution mechanisms. Furthermore, when the target system (such as the OA system) is temporarily unavailable due to maintenance or failure, the synchronization task will fail directly, lacking reliable offline buffering and recovery mechanisms. Finally, full synchronization is inefficient and struggles to meet the real-time requirements of scenarios with high-frequency data changes.

[0004] Therefore, how to provide a reliable and efficient method to solve the problems of inconsistent data states and collaboration between heterogeneous systems has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, this application aims to provide a data collaborative processing method and system for heterogeneous systems under a dual-track architecture, so as to solve the problems of inconsistent data states and low collaborative efficiency between heterogeneous systems in the prior art.

[0006] To achieve the above objectives, according to one aspect of this application, a method for collaborative data processing of heterogeneous systems under a dual-track architecture is provided. The method includes: A dual-track parallel architecture is constructed, which includes an AI track and an OA track, and the integration interface between the AI ​​track and the OA track is defined based on the API gateway middleware layer; An event-driven publish-subscribe model is adopted, and data flows between the AI ​​track and the OA track through an event bus. The event bus routes standardized event messages to registered event handlers based on event type. Maintain a state mapping table based on version number and business rule priority. When a state synchronization conflict between the two tracks is detected, select the corresponding resolution strategy from the preset resolution strategy matrix according to the type of the conflict field and perform the merging. During incremental synchronization, the synchronization progress is recorded based on the synchronization cursor, and the changed data is captured and synchronized through a hybrid mechanism that combines event triggering and timed compensation. When the target system is unavailable, an offline compensation mechanism is activated, storing the data to be synchronized in a local offline queue and retrying based on a backoff strategy until synchronization is successful or the data enters a dead-letter queue.

[0007] Furthermore, the internal module interfaces of the AI ​​track include: a workflow definition interface for defining the workflow topology, a workflow execution interface for triggering workflow execution, a node execution interface for executing specific AI processing tasks, and a result publishing interface for formatting AI output into standardized results; the internal module interfaces of the OA track include: a meeting minutes generation interface for generating structured meeting minutes based on meeting recordings, a task management interface for creating and managing task cards, and a table maintenance interface for updating multidimensional table data.

[0008] Furthermore, the API gateway middleware layer provides the following integrated interfaces: a one-way push interface for pushing analysis results from the AI ​​track to the OA track, a one-way pull interface for the OA track to initiate analysis task requests to the AI ​​track, and a two-way synchronization interface for configuring two-way or one-way synchronization modes; the gateway middleware layer also provides a unified routing interface, an authentication and authorization interface, and a format conversion interface.

[0009] Furthermore, the state mapping table is defined in JSON Schema format and stored in a relational database. Its data structure includes source system, source state, target system, target state, transformation rules, and action configuration. The state transformation rules are implemented based on a priority matrix. There is a bidirectional mapping path and triggering action defined between the workflow state of the AI ​​track and the task state of the OA track.

[0010] Furthermore, conflict detection employs an optimistic locking mechanism, achieved by comparing the version numbers of data records. The priority of the conflict resolution strategy matrix, from highest to lowest, is as follows: manual intervention rule, system source priority rule, timestamp rule, confidence rule, and field-level merging rule. Among these, the system source priority rule determines the priority system based on the field type: for task execution status fields, the OA track takes priority; for analysis conclusion fields, the AI ​​track takes priority; and for content description fields, field-level merging is used.

[0011] Furthermore, in the incremental synchronization mechanism, the data structure of the synchronization cursor includes the channel identifier, the last synchronization timestamp, the last synchronization change sequence number, and the synchronization direction; the hybrid triggering mechanism includes: a real-time event triggering mode based on data change events, and a timed compensation triggering mode executed at fixed time intervals; for deletion operations, a soft deletion method is adopted, and the deletion event is synchronized to the target system through a flag bit.

[0012] Furthermore, in the offline compensation mechanism, the local offline queue is implemented based on an SQLite database. Its data structure includes operation type, entity identifier, serialized synchronous data payload, number of retries, and queue status. The backoff strategy includes fixed interval backoff, linear growth backoff, and exponential backoff. When the number of retries exceeds the maximum retry threshold, the data is moved into the dead letter queue, and an administrator alarm is triggered.

[0013] According to another aspect of this application, a data collaborative processing system for heterogeneous systems under a dual-track architecture is provided. The system includes: The architecture building module is used to build a dual-track parallel architecture, which includes an AI track and an OA track, and defines the integration interface between the AI ​​track and the OA track based on the API gateway middleware layer; The event-driven module is used to transfer data between the AI ​​track and the OA track using an event-driven publish-subscribe pattern and an event bus. The state synchronization module is used to maintain a state mapping table based on version number and business rule priority, and to perform conflict detection and intelligent resolution. The incremental synchronization module is used to perform incremental data synchronization based on a synchronization cursor and a hybrid triggering mechanism; The offline compensation module is used to store data in a local queue and retry based on a backoff strategy when the target system is unavailable.

[0014] According to another aspect of this application, a data collaborative processing device for heterogeneous systems under a dual-track architecture is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the methods described above.

[0015] According to another aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, wherein a processor executes the program to implement any of the methods described above.

[0016] According to the technical solution of this application, the interconnection problem of heterogeneous systems is effectively solved by constructing a standardized dual-track integration interface and event-driven architecture. By introducing an intelligent conflict detection and resolution matrix based on version number and business priority, the consistency of dual-track data states is significantly improved. Through incremental synchronization cursors and a hybrid triggering mechanism, synchronization efficiency is optimized while ensuring real-time performance. Finally, through an offline queue compensation mechanism with a backoff strategy, the robustness of the system and eventual data consistency when the target system is unavailable are greatly enhanced. This application provides a complete, efficient, and reliable data collaboration solution for heterogeneous systems. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the data collaborative processing system for heterogeneous systems under a dual-track architecture according to an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] In the following descriptions of the embodiments, for ease of understanding, specific application scenarios (market research and analysis reports on AI tracks are pushed to the Lark OA system to generate task cards) will be used for illustration. However, it should be understood that this application scenario is merely exemplary, and the technical solutions of this application can be widely applied to other data collaboration scenarios between AI systems and OA systems.

[0020] In addition, in the following description, in order to more clearly illustrate the interaction relationship and data flow between the modules, the two core concepts of "AI track" and "OA track" will be uniformly named: AI track refers to a system or subsystem with artificial intelligence processing capabilities, including but not limited to workflow orchestration, AI model execution, and result output; OA track refers to an office automation system or collaboration platform, including but not limited to meeting minutes management, task management, and spreadsheet data maintenance.

[0021] The following embodiments will describe in detail the complete construction process of the dual-track parallel architecture and the specific implementation of the API gateway middleware layer.

[0022] Reference Figure 1First, we build the AI ​​track. The AI ​​track consists of three core sub-modules: a workflow orchestration module, an AI node execution module, and a result output module. Data is transferred between these sub-modules through standardized interfaces.

[0023] The workflow orchestration module exposes the `WorkflowDefine` interface to define the node topology of the workflow. This interface receives `workflowId` (a unique identifier for the workflow), `nodeList` (a list of nodes, each including `nodeId`, `nodeType`, `inputSchema`, and `outputSchema`), `edgeList` (defining the execution order and data flow between nodes), and `triggerConfig` (trigger configuration, supporting timed triggering, API call triggering, and event triggering modes). It also exposes the `WorkflowExecute` interface to trigger workflow execution, receiving `inputData` (input data conforming to the `inputSchema`) and `executionContext` (including execution ID, caller identifier, and execution priority). A separate `WorkflowStatusQuery` interface is provided to query the workflow execution status.

[0024] The AI ​​node execution module exposes the NodeExecute interface to perform specific AI processing tasks. This interface receives nodeType (e.g., text summarization, sentiment analysis, data comparison, report generation, etc.), inputPayload (input data payload), and modelConfig (model configuration parameters, including model provider, model version number, generation temperature, and maximum number of output tokens). It also exposes the NodeResultCallback asynchronous callback interface. After the AI ​​node completes execution, it sends the results back to the workflow orchestration module, including structured output data and confidence scores (floating-point numbers between 0 and 1). When the confidence score falls below a preset threshold, the workflow orchestration module automatically triggers a retry or manual review process.

[0025] The output module exposes the ResultFormat interface, which converts the raw output of AI nodes into a standardized format, supporting JSON, XML, and Markdown. A separate ResultPublish interface is provided, used to publish the formatted results to a specified target system. This interface includes the target system identifier, the target API endpoint, and retry policy configuration.

[0026] Next, we construct the OA (Office Automation) track. This embodiment will use Lark (Feishu) office collaboration system as an example. The OA track includes the following modules: The meeting minutes generation module exposes the MeetingMinutesGenerate interface, which automatically generates structured meeting minutes based on the meeting recording URL, meeting metadata (meeting ID, participant list, start time, duration), and template ID. It returns the meeting topic, a list of key decisions, a list of to-do items, and timestamped key points. A separate MeetingMinutesReview interface is provided for meeting minutes review.

[0027] The task management module exposes the TaskCreate interface for creating task cards, including task name, description, responsible person, due date, priority, and task source reference (for tracing the task's origin). It also exposes the TaskStatusUpdate interface for updating task status (pending, in progress, completed, canceled) and the TaskProgressQuery interface for querying task execution progress.

[0028] The multidimensional table maintenance module exposes the TableUpdate interface for updating multidimensional table data. It supports three operation types: insert, update, and update if the data exists, insert if it doesn't. A conflict handling strategy can also be specified. A separate TableQuery interface is provided for querying table data.

[0029] To connect the AI ​​and OA tracks, an API gateway middleware layer is deployed as the core hub for dual-track integration. The API gateway provides the following core functions: Unified Routing: Exposes the GatewayRoute interface, which distributes requests to the corresponding backend services based on the target system tags (AI_TRACK, OA_TRACK, STORAGE, NOTIFICATION, etc.) in the request. It supports various load balancing strategies and failover configurations, such as round-robin, minimum number of connections, and IP hash.

[0030] Authentication and Authorization: Exposes the GatewayAuth interface, supports multiple authentication methods such as API key, OAuth2, JWT and SAML, and adopts a role-based access control model to manage request permissions in a fine-grained manner.

[0031] Format conversion: Exposes the GatewayTransform interface, which supports mutual conversion between JSON, XML, Protobuf and form data, and performs conversions through predefined mapping rules and configuration file identifiers.

[0032] Monitoring and Alerts: Expose the GatewayMonitor interface to collect operational metrics such as API call count, response time, and error rate in real time.

[0033] The following core integration interfaces are defined through the API gateway middleware layer: The AiToOaPush interface pushes analysis results from the AI ​​track to the OA track. This interface supports three delivery modes: synchronous mode (waiting for the OA system to process and return a response), asynchronous mode (sending only without waiting for a response), and callback mode (sending asynchronously and receiving the processing result through a pre-registered callback interface). Parameters include the AI ​​track's standardized output, field mapping configuration file ID, and the selected delivery mode.

[0034] The OaToAiPull interface: The OA track sends analysis task requests to the AI ​​track. OA system users can use this interface to describe their analysis needs in natural language, specify the data range (including a list of data sources, time range, filter conditions, etc.), and set the expected completion deadline.

[0035] BidirectionalSync interface: a bidirectional status synchronization interface used to configure the synchronization direction (AI to OA unidirectional, OA to AI unidirectional or bidirectional) and synchronization mode (full synchronization, incremental synchronization or differential synchronization).

[0036] Specific application scenario examples In a specific application scenario, after completing a market research and analysis report, the AI ​​track calls the AiToOaPush interface to push the analysis results to the OA track. Upon receiving the request, the API gateway first performs authentication and authorization verification to confirm that the AI ​​track has permission to call the OA track interface. Then, it calls the GatewayTransform interface to convert the data in the AI ​​track format to the data format required by the OA track (Lark) according to the mapping configuration file. Next, it routes the converted request to the corresponding OA track server endpoint according to the load balancing strategy. After receiving the request, the OA track's task management module creates the corresponding task card and returns a successful creation response. Finally, the API gateway returns the response to the AI ​​track, completing a full dual-track data push.

[0037] The event-driven architecture and format conversion engine in another embodiment will be described in detail below. This embodiment is basically the same as the previous embodiment, except that the event-driven architecture and API gateway format conversion engine are described in more detail.

[0038] In this embodiment, all cross-track data operations are encapsulated into standardized event messages. Each event message includes the following fields: event_id: A globally unique event identifier, in the format of "evt_date_random string", used for event tracking and idempotency verification.

[0039] event_version: Event version number.

[0040] event_type: An enumeration of event types, including analysis_completed, analysis_failed, task_created, task_updated, task_completed, sync_required, conflict_detected, etc.

[0041] event_category: Event categories, divided into four categories: workflow, task, data, and system.

[0042] source_system: Identifier of the system from which the event originated (AI_TRACK or OA_TRACK).

[0043] source_module: Source module.

[0044] target_system: Identifier of the target system.

[0045] target_module: The target module.

[0046] timestamp: The timestamp of the event, accurate to milliseconds.

[0047] trace_id: Distributed tracing identifier, used throughout the entire request chain.

[0048] payload: Event payload, which includes business data, and its structure changes dynamically depending on the event_type.

[0049] metadata: Metadata includes contextual information such as workflow ID, execution ID, model version, and processing time.

[0050] For example, in an event message completed by AI analysis, the payload includes business data such as report ID, title, summary, list of key findings, list of recommendations, priority, and responsible person recommendations.

[0051] The event bus is implemented using a publish / subscribe pattern, specifically including: Topic Division: Events are divided into multiple topics based on their categories, including ai.workflow, oa.task, sync.bidirectional, and alert.system. Each topic can be configured with multiple partitions for horizontal scaling.

[0052] Message persistence: Event messages are synchronously written to the Write-Ahead Log (WAL) upon publication, and a publication success response is only returned after successful writing confirmation. Consumers send an ACK confirmation after processing; messages without an ACK are automatically redelivered after a timeout (default 5 minutes).

[0053] Routing strategy: Routing is based on the target_system and target_module fields in the event message, supporting both content-based and rule-based routing modes.

[0054] Order guarantee: For events that require guaranteed processing order (such as continuous state changes in the same workflow), a partition key mechanism (such as workflow_id) is used to route related events to the same partition, ensuring that the consumer's processing order is consistent with the publishing order.

[0055] Dead-letter queue: When the number of retries for consuming an event message fails to reach the maximum number (default 3), the message is moved into the dead-letter queue and processed by a dedicated dead-letter processor.

[0056] In this embodiment, the event handler adopts a plug-in architecture, supporting dynamic registration and unloading. Registering a handler with the event bus via the RegisterHandler interface requires specifying the event types the handler is interested in, the handler identifier, filtering conditions, and the callback address. Three types of event handlers are supported: synchronous handlers (blocking and waiting for processing to complete, suitable for scenarios requiring real-time response), asynchronous handlers (non-blocking, suitable for time-consuming processing), and batch handlers (accumulating a certain number of events or processing them in batches after a time window, suitable for high-throughput scenarios).

[0057] Furthermore, the core of the format conversion engine is the mapping configuration file, defined in YAML format. A typical AI track to Lark task mapping configuration file includes the following parts: Field mapping list: Each mapping item includes source field path, target field path, conversion function, value mapping table, maximum length limit, etc.

[0058] Data type conversion configuration: Define the fields that need to be converted and their conversion rules.

[0059] Validation rules: Define the validation rules for the converted data and the actions to take when a failure occurs.

[0060] The standard process for a conversion engine to perform a conversion includes the following five steps: Step 1: Schema Validation. Validate the input data according to the source system's schema to ensure that data types, required fields, and value ranges conform to the source system's output specifications. If validation fails, choose to reject the entire conversion request or fill in missing fields with default values, depending on the configuration.

[0061] Step 2: Field Mapping and Value Transformation. Perform the mapping field by field according to the fields list in the mapping configuration. For each field, execute the following in sequence: (1) Extract the field value from the source data according to the source_field path; (2) If a transform function (such as CONCATENATE, SUBSTRING, PREFIX, etc.) is configured, execute the function to perform string-level transformation; (3) If a value_map value mapping table is configured, map the original value to the target value (for example, map "high" to "urgent"); (4) If the field value length exceeds max_length, truncate it to the specified length; (5) Write the transformed value to the target_field position in the target data.

[0062] Step 3: User and Organization Information Conversion. For fields containing user email or department information, the user lookup function (USER_LOOKUP) is called to convert the email address or department name into a user ID or department ID used internally by the OA platform by calling the API provided by the OA platform.

[0063] Step 4: Transformation of Complex Data Structures. For array or object-type data, perform one of the following transformations: JSON_TO_MARKDOWN (converts the array of key findings in JSON format to a list in Markdown format), ARRAY_MAP (calls a mapping function on each element in the array, generating each suggestion as a subtask of the Lark task), JSON_TO_XML, or XML_TO_JSON.

[0064] Step 5: Data Type Conversion and Validation. Perform conversion operations on fields requiring type conversion. Supported conversion types include conversion between strings and numbers, between date / time and timestamps, and between JSON and XML. After conversion, validate the resulting data according to the target system schema to ensure it meets the target platform's interface requirements. Finally, serialize the validated data into the target format and return it.

[0065] The following example demonstrates how to convert AI track output analysis reports into Lark task cards: Priority field: The "priority" value of the AI ​​track is "high", which is converted to "urgent" according to the value_map mapping rules (high→urgent, medium→normal, low→low).

[0066] Title field: The "title" value of the AI ​​track is "Q2 Market Research". By adding a prefix using the CONCATENATE function, it is converted to "[AI Analysis] Q2 Market Research".

[0067] The responsible person field: The "assignee_suggestion" value of the AI ​​track is "zs@company.com". It is converted into the Lark internal user ID "ou_12345678" by calling the Lark API through the USER_LOOKUP function.

[0068] Key discovery field: The "key_findings" field of the AI ​​track is a JSON array, which is converted into a Markdown list using the JSON_TO_MARKDOWN function and then populated into the description field of the Lark task card.

[0069] Suggestion field: The "recommendations" field of the AI ​​track is an array of suggestions. The ARRAY_MAP function calls the create_subtask function for each suggestion to generate a list of subtasks for the Lark task.

[0070] The following section will describe in detail the implementation of the state mapping table and the complete execution flow of the conflict detection and intelligent resolution strategy.

[0071] The state mapping table is stored in the state_mapping table of the relational database. The table structure includes the following fields: mapping_id (primary key), source_system (source system identifier), source_state (source state code), target_system (target system identifier), target_state (target state code), transition_rule (transition rule definition in JSON format, supporting conditional expressions), action_config (trigger action configuration in JSON format), priority (rule priority, the smaller the value, the higher the priority), enable_flag (enable flag), and a timestamp field.

[0072] The conditional expression stored in the `transition_rule` field is as follows: json { "condition": "source.confidence >= 0.85 AND source.priority == 'high'", "transform": { "target.priority": "urgent", "target.due_date": "DATE_ADD(NOW(), INTERVAL 1 DAY)" }, "validation": { "required_fields": ["report_id", "title", "key_findings"], "field_types": { "report_id": "string:uuid", "priority": "enum:high,medium,low" } } } The mapping rules from AI track workflow status to OA track task status are as follows: The mapping rules for reverse synchronization (OA task status to AI workflow status) are similar.

[0073] According to embodiments of this application, an optimistic locking mechanism is used to implement conflict detection. Each synchronized data record maintains the following version control fields: version: Integer type, initial value is 1, and it is automatically incremented by 1 after each successful update.

[0074] last_modified_time: Last modified timestamp, with millisecond precision.

[0075] last_modified_by: The identifier of the last modifier, which adopts a combination format of "system identifier + user identifier".

[0076] change_sequence: Change sequence number, assigned by the central sequence generation service, and globally monotonically increasing.

[0077] edit_history: Edit the history array, retaining the last 50 changes.

[0078] Version number updates utilize atomic database operations: UPDATE sync_data SET version = version + 1, data = ?, last_modified_time = NOW() WHERE data_id = ? AND version = ? If the update operation affects 0 rows, it means the data has been modified by other operations after being read, and is considered a conflict; if it affects 1 row, the update is successful.

[0079] According to embodiments of this application, the resolution strategies, ranked from highest to lowest priority, are as follows: P0 - Manual Intervention Rule: Fields marked as "Locked for Manual Processing" are not applicable to any automatic resolution strategies.

[0080] P1 - System Source Priority Rule: Defines which system's modifications have priority based on the field type.

[0081] P2 - Timestamp Rules (Last-Write-Wins): Retrieves the modification with the later timestamp.

[0082] P3 - Confidence Rule: For AI output results, take the version with higher confidence.

[0083] P4 - Field-level merge rule: Merge non-conflicting fields, retaining the modifications made by each side.

[0084] Depending on the field type, the system source priority rules are applied as follows: Task execution status fields (such as status, progress): OA track priority (actual employee execution status takes precedence over AI prediction).

[0085] Analysis conclusion fields (such as analysis_result, confidence_score): AI track priority (AI analysis results overwrite old conclusions).

[0086] Time-related fields (such as due_date, remove_time): OA track takes priority (manually adjusted time schedules take priority).

[0087] Content description fields (such as description and comments): Field-level merging (the content of the two systems is usually complementary).

[0088] Priority fields (such as priority): Business rule judgment (combining AI suggestions and historical adjustment records).

[0089] When a conflict is detected, the system enters the resolution process: Step 1: When the AI ​​track or OA track submits a data update, compare the version in the current database with the version in the submitted data. If the versions do not match, a conflict is determined.

[0090] Step 2: Extract the list of diff fields between the two versions (diff_fields). For each diff field, query the field type mapping table to determine its type and mark whether manual locking exists.

[0091] Step 3: Based on the type of the difference field, find the corresponding resolution strategy from the resolution strategy selection matrix. If multiple difference fields in the same record require different resolution strategies, apply the corresponding strategies one by one for each field.

[0092] Step 4: Execute the selected resolution strategy to generate the merged data. During the execution process, record the resolution decision log (resolution_log) for each field, including the field name, the strategy used, the two values ​​before merging, the value after merging, and the reason for the selection.

[0093] Step 5: After the conflict resolution is complete, store the merged results in the database, incrementing the version number by 1. Simultaneously, send conflict resolution notifications to the administrators of both the AI ​​and OA tracks, including the complete resolution_log for manual review. If any field triggers the "manual processing" policy, the entire record's status is marked as pending_manual_review, pausing automatic synchronization until manual intervention.

[0094] The following describes in detail the data structure, update mechanism, and hybrid triggering mode of the incremental synchronous cursor with reference to an embodiment.

[0095] Synchronous cursor records are stored in the sync_cursor table, which includes the following fields: cursor_id: Primary key.

[0096] channel_id: Synchronization channel identifier (e.g., ai_to_oa_workflow), used to distinguish different synchronization tasks.

[0097] last_sync_time: The timestamp of the last synchronization, in milliseconds.

[0098] last_change_sequence: The last synchronized change sequence number, used for precise incremental fetching based on the sequence number.

[0099] last_data_id: The record ID of the last piece of data that was last synchronized.

[0100] sync_direction: Synchronization direction (ai_to_oa, oa_to_ai, or bidirectional).

[0101] entity_type: Synchronization entity type (workflow_task, report_data, status_update, etc.).

[0102] sync_count: The total number of times this channel has been synchronized.

[0103] Updates to the synchronized cursor employ atomic operations to ensure concurrency safety. After each incremental synchronization is complete, the cursor position is updated with the latest synchronized `change_sequence` and `sync_time`.

[0104] Incremental synchronization employs a hybrid triggering mechanism, primarily event-driven and supplemented by timed compensation. Event-triggered mode (real-time synchronization): When data changes occur in the AI ​​or OA track, the change operation immediately sends a change_event message to the synchronization service after completing the local transaction commit. Upon receiving the event, the synchronization service uses the change_sequence corresponding to the event as the synchronization endpoint and starts from the current cursor position to retrieve all change records within that interval for synchronization. This mode is suitable for high-priority scenarios requiring real-time synchronization (such as task status changes and emergency alarm pushes).

[0105] Scheduled Trigger Mode (Compensation Synchronization): As a fallback mechanism for event-triggered synchronization, the synchronization service performs compensation synchronization at fixed time intervals. Configuration parameters for scheduled synchronization include: `sync_interval` (synchronization interval, default 60 seconds, configurable per channel), `batch_size` (maximum number of records synchronized per batch, default 100 records), and `timeout_threshold` (synchronization timeout threshold, default 30 seconds). This mechanism ensures that even if event messages are lost due to network anomalies or system failures, data can eventually be synchronized via scheduled tasks.

[0106] Manual trigger mode: Administrators can manually trigger the synchronization operation of a specified channel through the management backend, supporting data backtracking synchronization within a specified time range or change_sequence range.

[0107] To avoid data unrecoverability and synchronization issues caused by direct physical deletion, a soft deletion mechanism is used. A `is_deleted` flag (0 = normal, 1 = deleted) and a `deleted_at` timestamp are added to the data record. During incremental synchronization, records with `is_deleted=1` are synchronized to the target system as deletion events, and the target system performs the corresponding deletion operation based on the deletion event. Soft-deleted records are retained in the database for 90 days by default before being physically cleaned up by a scheduled task, during which time data recovery is supported.

[0108] In one embodiment, the synchronization trigger employs a hybrid mode of Webhook + timed polling. Webhook push mode: When the state changes, the system proactively sends an HTTP POST request to the pre-registered Webhook URL. The request body includes complete state data before and after the change. The Webhook endpoint must return a 200 status code within 5 seconds; otherwise, a retry mechanism is triggered (maximum of 3 retries, with intervals of 1 second, 5 seconds, and 30 seconds). Timed polling mode serves as a fallback mechanism, polling the state change log table (state_change_log) at fixed time intervals (default 30 seconds), comparing it with the last synchronization position recorded by last_sync_cursor, and capturing all state changes that occurred during that period.

[0109] The offline compensation mechanism when the target system is unavailable will be described in detail below with reference to embodiments.

[0110] When the target system is unavailable, the data to be synchronized is temporarily stored in a local queue. The local queue uses an SQLite database for storage, ensuring that data is not lost after a system restart. The structure of the offline queue list `offline_sync_queue` includes the following fields: queue_id: an auto-incrementing primary key.

[0111] channel_id: Synchronization channel identifier.

[0112] operation_type: Operation type (INSERT / UPDATE / DELETE / UPSERT).

[0113] entity_type: Entity type.

[0114] entity_id: Entity identifier.

[0115] payload: Serialized synchronous data (JSON format).

[0116] payload_checksum: Data checksum (SHA-256) used for integrity verification.

[0117] retry_count: The number of retries, initially set to 0.

[0118] max_retry: Maximum number of retries, defaults to 5.

[0119] next_retry_time: The time for the next retry.

[0120] retry_backoff: Backoff strategy code (1=fixed interval, 2=linear growth, 3=exponential backoff).

[0121] status: Queue status (pending, processing, failed, dead_letter).

[0122] error_log: Error log.

[0123] The queue operation interfaces include: Enqueue (enqueue, calculate checksum and set retry parameters), Dequeue (retrieve data to be processed in FIFO order), UpdateStatus (update status, automatically increment retry_count and calculate next_retry_time on failure), and PeekFailed (query failure records).

[0124] The retry mechanism employs a backoff strategy to avoid request surges during the initial recovery phase of the target system. The system supports three backoff strategies: Fixed-interval backoff: The retry interval is fixed at base_interval seconds, which is suitable for scenarios with brief network jitter.

[0125] Linear growth backoff: The interval for the nth retry is base_interval × n seconds, and the interval increases with each retry.

[0126] Exponential backoff: The interval between the nth retry is Seconds are suitable for scenarios where the target system may be unavailable for an extended period of time.

[0127] The default retry strategy is exponential backoff, with the following time intervals: 1 second for the first retry, 2 seconds for the second, 4 seconds for the third, 8 seconds for the fourth, and 16 seconds for the fifth, for a total wait time of approximately 31 seconds. For critical business channels (such as emergency alarm synchronization), `max_retry` can be increased to 10 times, and `base_interval` can be reduced to 500 milliseconds; for non-critical business channels (such as statistical report synchronization), `max_retry` can be reduced to 3 times, and `base_interval` can be increased to 5 seconds.

[0128] The synchronization service maintains a health status cache for the target system. The detection mechanisms include: Active Probe: Send a health check request (Ping / Health endpoint) to the target system every 30 seconds and record the response time and status code.

[0129] Passive Probe: Count the number of consecutive failures in normal synchronization requests. When the number of consecutive failures reaches 3, mark the target system as unavailable.

[0130] Recovery detection: When the target system is marked as unavailable, the detection interval is shortened to 5 seconds. After 3 consecutive successful detections, it is marked as available, and the compensation synchronization of the offline queue is immediately triggered.

[0131] If the number of retries for synchronizing data exceeds max_retry and still fails, the status of the data is updated to dead_letter, and the following processing flow is triggered: The data is moved into the dead letter queue to save the original payload, complete error logs, and retry history.

[0132] Send alarm notifications to the system administrator via Lark messages, emails, and SMS, depending on the alarm level configured for the channel.

[0133] Administrators can view the dead letter queue through the management backend and choose from the following processing methods: manual retry (re-trigger synchronization after manually confirming that the target system has recovered), edit and retry (re-synchronize after correcting data problems), ignore (confirm that the synchronization does not need to be executed), or export (export the data to a file for offline processing).

[0134] All parts not covered in this application are the same as or can be implemented using existing technology. Although embodiments of this application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A data collaborative processing method for heterogeneous systems under a dual-track architecture, characterized in that, The method includes: A dual-track parallel architecture is constructed, which includes an AI track and an OA track, and the integration interface between the AI ​​track and the OA track is defined based on the API gateway middleware layer; An event-driven publish-subscribe model is adopted, and data flows between the AI ​​track and the OA track through an event bus. The event bus routes standardized event messages to registered event handlers based on event type. Maintain a state mapping table based on version number and business rule priority. When a state synchronization conflict between the two tracks is detected, select the corresponding resolution strategy from the preset resolution strategy matrix according to the type of the conflict field and perform the merging. During incremental synchronization, the synchronization progress is recorded based on the synchronization cursor, and the changed data is captured and synchronized through a hybrid mechanism that combines event triggering and timed compensation. When the target system is unavailable, an offline compensation mechanism is activated, storing the data to be synchronized in a local offline queue and retrying based on a backoff strategy until synchronization is successful or the data enters a dead-letter queue.

2. The method according to claim 1, characterized in that, The internal module interfaces of the AI ​​track include: a workflow definition interface for defining the workflow topology, a workflow execution interface for triggering workflow execution, a node execution interface for executing specific AI processing tasks, and a result publishing interface for formatting AI output into standardized results; the internal module interfaces of the OA track include: a meeting minutes generation interface for generating structured meeting minutes based on meeting recordings, a task management interface for creating and managing task cards, and a table maintenance interface for updating multidimensional table data.

3. The method according to claim 2, characterized in that, The integrated interfaces provided by the API gateway middleware layer include: a one-way push AiToOaPush interface for pushing analysis results from the AI ​​track to the OA track, a one-way pull OaToAiPull interface for the OA track to initiate analysis task requests to the AI ​​track, and a two-way synchronization BidirectionalSync interface for configuring synchronization direction and mode; the API gateway middleware layer also provides a unified routing interface, an authentication and authorization interface, and a format conversion interface.

4. The method according to claim 1, characterized in that, The state mapping table is defined in JSON Schema format and stored in a relational database. Its data structure includes source system, source state, target system, target state, transformation rules, and action configuration. The state transformation rules are implemented based on a priority matrix. A bidirectional mapping path and triggering actions are defined between the workflow state of the AI ​​track and the task state of the OA track.

5. The method according to claim 1, characterized in that, The conflict detection adopts an optimistic locking mechanism, which is achieved by comparing the version numbers of data records; the priority of the resolution strategy matrix from high to low is as follows: manual intervention rule, system source priority rule, timestamp rule, confidence rule, and field-level merging rule; The system source priority rule determines the priority system based on the field type: for task execution status fields, the OA track takes priority; for analysis conclusion fields, the AI ​​track takes priority; and for content description fields, field-level merging is used.

6. The method according to claim 1, characterized in that, In the incremental synchronization mechanism, the data structure of the synchronization cursor includes channel identifier, last synchronization timestamp, last synchronization change sequence number, and synchronization direction; The hybrid triggering mechanism includes: a real-time event triggering mode based on data change events, and a timed compensation triggering mode executed at fixed time intervals; for deletion operations, a soft deletion method is adopted, and the deletion event is synchronized to the target system through a flag bit.

7. The method according to claim 1, characterized in that, In the offline compensation mechanism, the local offline queue is implemented based on an SQLite database, and its data structure includes operation type, entity identifier, serialized synchronous data payload, number of retries and queue status. The backoff strategies include fixed-interval backoff, linear growth backoff, and exponential backoff; when the number of retries exceeds the maximum retry threshold, the data is moved into the dead-letter queue and an administrator alarm is triggered.

8. The method according to claim 1, characterized in that, The internal module interfaces of the AI ​​track include: a workflow definition interface for defining the workflow topology, a workflow execution interface for triggering workflow execution, a node execution interface for executing AI processing tasks, and a result publishing interface for formatting AI output into standardized results. The internal module interfaces of the OA track include: a meeting minutes generation interface for generating structured meeting minutes based on meeting recordings, a task management interface for creating and managing task cards, and a table maintenance interface for updating multidimensional table data.

9. A data collaborative processing system for heterogeneous systems under a dual-track architecture, characterized in that, The system includes: An architecture building module is used to build a dual-track parallel architecture, which includes an AI track and an OA track, and defines the integration interface between the AI ​​track and the OA track based on the API gateway middleware layer; The event-driven module is used to transfer data between the AI ​​track and the OA track using an event-driven publish-subscribe pattern. The state synchronization module is used to maintain a state mapping table based on version number and business rule priority, and to perform conflict detection and intelligent resolution. The incremental synchronization module is used to perform incremental data synchronization based on a synchronization cursor and a hybrid triggering mechanism. The offline compensation module is used to store data in a local queue and retry based on a backoff strategy when the target system is unavailable.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method of any one of claims 1 to 7.

Images